Margins of Error

By | July 14, 2011

Vintage 7UP advertisement (1960s?) showing a baby drinking from a bottle of 7UP. Because it's WHOLESOME.

Shannon from Fierce Fatties tweeted a link at me earlier this week, to a report from the Robert Wood Johnson Foundation (RWJF) about the obesity scourge.  The RWJF, in case you’re unfamiliar with it, is a massive “philanthropic” organization specializing in public health. The organization’s titular founder is, yes, the guy who also brought us Johnson & Johnson, the bazillion-dollar international megacorporation that makes everything from prosthetic joints and blood glucose monitoring systems, to that orange bar of Neutrogena facial soap melting away in your shower, to the tube of KY Jelly you used before you knew there were better lubes out there, to the “No More Tears” baby shampoo your mom washed your little noggin with when you were a wee bairn. You want a list of subsidiary holdings and consumer brands? Wikipedia’s got you covered.

It is not overstating the matter to say that Johnson & Johnson looms over all things healthcare like a towering and inescapable monolith. So it’s little wonder that the Robert Wood Johnson Foundation should rank as the ninth wealthiest charitable organization in the world, outmatched by such notable competition as the J. Paul Getty Trust, the Ford Foundation, the Howard Hughes Medical Institute, and coming in first, the Bill and Melinda Gates Foundation. These are all organizations that deal in billions and billions of dollars, and many of them do quite a bit of good.

This, of course, is not one of those cases.

Why am I am telling you about the background and fiscal resources of this foundation? To make clear that this is one of the most well-funded and highly-respected organizations on the planet. What the RWJF publishes matters. So when they put out an annual report entitled “F as in Fat” on the state of national rampant fattery, people pay attention. The media that shapes our culture and our discourse pay attention. This is the kind of thing that influences news coverage of obesity, which influences how we talk about obesity, which influences what we think we know about fatness and fat people, which ultimately impacts all of us — fat, thin, and bell-curved.

I’m not going to pick apart the whole report, as that would take ages, and frankly, it’s unnecessary — there is so much doublespeak in there that doing so would rapidly become redundant. It frets over rates of diabetes and hypertension seemingly without accounting for the effects of improved diagnosis of such ailments, plus an aging boomer population reaching the stage of life where such problems are hardly unusual. It does usefully identify differences in obesity rates based on gender, race, and class, but doesn’t do much with this information aside from make obvious platitudes about encouraging poor families to eat more salad. You can download the whole thing, for free, as a PDF file to read at your leisure.

For my part, I am just going to draw your attention to a couple of points of interest, to illustrate why we should all be a lot more critical of the panicked rhetoric around obesity paranoia.

For one, the RWJF has a particular interest in the currently trendy social terror of childhood obesity. This is explicitly named as one of their seven program areas. Their report is riddled with dire predictions about the future consequences of so many fat kids, and like most of the media discussing this topic, uses the cultural fear of out-of-control weight gain amongst children to bolster the need for action. And yet, on page 7:

Recent studies have shown that the number of obese children and adolescents may have leveled off since 1999, except among the very heaviest boys ages 6-19, but the rates remain startlingly high.

Here’s a personal memory: in 1999 I was a graduate student in developmental psychology, specializing in the development of media literacy for kids age 6 to 14, particularly as it impacted the body image and self esteem of girls. I spent a truly absurd amount of time analyzing and decoding both teen and women’s magazines, as well as television, film, and news media. I have no recollection of there being half as much “childhood obesity” panic at that time, and as a fat activist, if there was even a whisper of the level of modern day news coverage on this topic, I can assure you I would have picked up on it. And analyzed and written about it.

So. Why weren’t we panicking back in 1999, especially if the rates were “startlingly high”? Why haven’t we seen a decade of ominous warnings? If rates of childhood obesity (except amongst the fattest boychildrens, which represent between 4% and 7% of the total boy population) have been flat for the past twelve years, why are we all talking like the kids are suddenly exploding, when the hard evidence indicates they are not?

Maybe we were too busy demonizing smokers. I have no idea. Honestly. I have used the scapegoat analogy before; part of me thinks we just need to fix on some public ill as the source of our problems, to name it as the blight on our future. But the reality, even according to the Robert Wood Johnson Foundation, would seem to be that the childhood obesity bombshell is a big dud.

The second point to which I would like to draw your attention is a chart. Well, part of a chart. I’ve cropped the interesting bit below; clicking the image links to a full-size image of the whole chart.

This chart tracks the current obesity rates for all fifty states from 2008-2010, ranks them, and then shows the percentage difference — in other words, the rate of change — compared to rates from 2007-2009. I want to point out the rate of change, but also the margin of error, displayed in parentheses next to the current obesity rate in the first column. I even circled them for you.

Cropped image of a chart from the RWJF report, showing obesity rates and rates of change for all fifty states.

The margin of error is the amount by which the results may be off, owing to random sampling error. You find it on all sorts of polls, as it influences the quality of the results — a big margin of error means the poll was poorly designed or administered. Usually there is a direct connection between the size of the sample and the margin of error: a sample consisting of 10,000 participants will have a lower margin of error than one only using a sample of 100.

All of the margins of error above are pretty low. But when we compare them to the rate of change, there’s an interesting convergence. Alabama, the second fattest state, shows an obesity rate increase of 0.7% between the 2007-2009 results and the 2008-2010 results. However, the margin of error is plus or minus 1%. If we apply this margin of error to the rate of change, both adding it and subtracting it, we get a different picture. It is statistically feasible that obesity rates in Alabama have not changed at all, or have even decreased by 0.3%. It is also possible that the increase was as much as 1.7%, but even then — it’s a hypothetical increase of less than two percent. Hardly an epidemic.

If this was only true of Alabama, then maybe our hysterical wailing could be forgiven, but no: scroll down the list and you’ll find that nearly all the rates of change are within or near the margins of error.

So where is our missing epidemic? Is it possible it could just be in our heads, some strange mass psychosis brought on by sheer repetition? An acquaintance recently commented to me, upon my mentioning that people my size only comprise roughly 5% of the population: “But it sure seems like people are getting bigger.” It seems that way because that’s what we’ve been told. Even studies that do claim to show a clear increase in fattery over the past few years are illustrating a mere handful of pounds, spread across the entire population, something which can, again, be at least partly explained by my parents’ generation getting older and putting on a bit of weight, as older people often do.

You can disagree with my read on this. That’s fine. I’m just asking you to be critical of the anti-obesity handwringing; do not trust that it’s true simply because you read it on CNN. Read the results and draw your own conclusions. They don’t have to mirror mine, but you should be capable and even inclined to seek out this information for yourself. Because how we think about our bodies and our health affects our whole lives.



JupiterPluvius on July 14, 2011 at 1:42 pm.

You know what? Americans, as a group, are significantly taller than Americans of a couple generations ago. BUT WHAT ARE WE DOING ABOUT THE HEIGHT EPIDEMIC I ASK YOU


Ruth of All Evil on July 14, 2011 at 9:43 pm.

BAHAHAHAHA. Do you mind if I steal this?


JupiterPluvius on July 15, 2011 at 11:22 pm.

It’s for everyone! ENJOY!


fatvegancommie on July 15, 2011 at 7:30 pm.

i am totally tweeting that


notblueatall on July 14, 2011 at 1:49 pm.

Thank you for putting this so plainly. Man, I wish people were taught early on to question these things and dig just a bit deeper than the surface. Rock on, Lesley!


Tom Brokaw on July 14, 2011 at 1:57 pm.

A large margin of error does not indicate a poorly designed or administered poll, just as the lower comparative quality of a prius to a maybach doesn’t mean a prius is a poorly designed car or that it is not well executed for its intended purpose.

It’s just the margin of error the pollsters found acceptable/affordable.

A poorly designed or administered poll would have an inaccurate margin of error.


Lesley on July 14, 2011 at 2:00 pm.

Hm. I guess. I tend to think the things that make for a big margin of error, especially with regard to sample size, signal poor design, or at least poor funding. But I will defer to you on that.

ETA: Also, if results are released with a big margin of error, which the researchers found acceptable, I kind of doubt the good intentions of those researchers. Good research is accurate research.


Daniel on July 15, 2011 at 12:36 am.

I don’t know whether those would be considered a large or small error in polling, but the important point is that the change between the two periods is within the 95% confidence interval. This means that by the logic of inferential statistics, we cannot reject the possibility that the obesity proportion did not change at all, and in a scientific article this would be interpreted by all readers as essentially zero evidence for an increase in obesity (as the magic incantation goes, p is not < .05) Even the states like Texas where the change falls outside of the 95% confidence interval don't constitute valid evidence, because if you have so many of these in a big table some are definitely going to fall outside the confidence interval by chance. Anyway that's just the perspective from the "rules" of scientific arguing – your reasoning in this piece is absolutely sound, that it's the size of the difference that is important, especially relative to the amount of error in the measurement. Love your debunking!


Amy on July 14, 2011 at 2:00 pm.

I think you make some good points, but I’m not convinced that we weren’t starting to panic in 1999. I kind of feel like the genesis of the whole “childhood obesity epidemic” was in the late 90’s. My first child was born in 1999, and I remember hearing inklings of problems then, and certainly for at least the last 5 years. I remember so clearly around 1999 or 2000 when the news did a little blurb on how researchers say that the show Teletubbies is bad for children, because it teaches them that it is OK to be fat. I was so incensed by that presentation, yet it was done so matter-of-factly.

C. Everett Koop surely played a part with the Shape Up America thing in the mid 90’s, making sure people saw obesity as a health issue, against which a war should be waged. And then it was carried on with Carmona and his comments in 1996, about how obesity is a bigger problem than terrorism. Childhood obesity was in the forefront at that point.


Lesley on July 14, 2011 at 2:03 pm.

We were definitely talking about it! But given that rates STOPPED increasing in 1999 and yet the panic has yet to reach its apex twelve years later… I think that’s strange.


erylin on July 14, 2011 at 2:04 pm.

the problem is no one takes the time to read anything, or even question teh OMG OBESITY blurb that lasts all of 2 min on tv…and then they segway into their guest, writer of the book eat this not that.


Bonnie on July 14, 2011 at 4:31 pm.

Also important to note that thresholds for overweight BMI were lowered in 1998:) This is never noted in the media and they have been referring to data from 1995 as of late.


Lesley on July 14, 2011 at 4:34 pm.

They actually do note this in the report, although the statistics are not adjusted in any way to account for it. I was impressed they acknowledged it at all.


Alexie on July 14, 2011 at 6:33 pm.

One thing is never clear to me in the discussions about the ‘obesity epidemic’. Is it that more people are getting bigger/fatter, or is it that people who were already big are getting bigger? e.g. is it more people, or is it the same people who were always big, but they’re bigger? And how dynamic is the situation? My mother, for example, was borderline obese until recently. Thanks to age-related medical conditions, she’s now underweight. So is she part of the obesity epidemic or not?


JupiterPluvius on July 15, 2011 at 11:27 pm.

Actually, the numbers of people (in the US, at least) who fall into the “obese” category have been pretty much stable for the last twelve years. Obesity rates among children have been absolutely stable over that period.

The rate of “overweight” that is not characterized as “obese” is absolutely flat over something like 30 years.



vesta44 on July 14, 2011 at 7:26 pm.

“But it sure seems like people are getting bigger.”

Ever notice that when you buy a different car, you start to notice more of those cars on the road, where you hadn’t really noticed them before? I don’t think it’s that people are getting that much bigger, it’s that it’s been brought to our attention, pounded into us, that people are getting fatter, and we’re seeing images of people with BMIs over 40 in every article about the OMGOBESITYEPIPANICeleventyone! so of course we’re going to notice more fatter people everywhere we look, even though those fat people have been there all along. Only now, it’s not “normal” to see those people, it’s a catastrophe and “something needs to be done about it”. Yeah, well, I’m not holding my breath waiting for people my size (BMI over 40, waaaaaay over 40) to get thin and satisfy the haters among us.


Atchka! on July 14, 2011 at 10:25 pm.

Hey, I’ve got a question… how do you fit all your outrage into such a succinct, readable little package? I’m toying with revisiting this piece of crap already because of the sheer magnitude of dishonesty behind it.

I hope that, like the Republican party, the overreach will continue to be so egregious that society can’t help but miss the thoroughly rotten state of Denmark.



G on July 14, 2011 at 11:43 pm.

Oh man! In climatology, if we tried to publish stats like that as a “trend”, people would HOWL.

Also, I love how they slap a big ol’ “95% Confidence Interval” on the header and completely disregard what that means.


cim on July 15, 2011 at 5:38 am.

I’m not sure if they account for this or not, but the margin for not being significant is actually larger than that table implies. That table gives the margin for error of the 2008-2010 measurement, and the change from the 2007-2009 measurement. But that’s presumably a midpoint-to-midpoint change, so the sampling error size in the 2007-2009 study also needs to be accounted for.

(So, for instance, if 2007-2009 and 2008-2010 both had 0.7% sampling error at 95%, then it’s possible that 2007-2009 was measured low, and 2008-2010 was measured high, so the difference needs to be somewhat larger than 0.7% to be significant)

Also, they’re using a 95% confidence interval which means “only a 1 in 20 chance that this is a coincidence”, but making 50 separate measurements. So even if there’s no change at all, you would expect 1 in 20 states – 2 or 3 – to show “statistically significant” differences just from random noise.


Stephanie on July 17, 2011 at 8:11 am.

Could I just point out that the advertisement that you’ve used at the top actually suggests that mothers should mix 7up into babies’ milk to get them to drink it! Does that sound disgusting to anybody else? I’ve heard of mixing a little corn-syrup into formula to get babies to drink it, but 7up? Wouldn’t the milk curdle?


jerri on July 19, 2011 at 5:25 pm.

No worse than Pepsi and milk 😀


Paintmonkey on July 21, 2011 at 6:43 am.


Not enough! I have just this minute read that being a tall woman apparently increases your chances of getting 16 different types of cancer. 16! Not 15 – but 16! Tall women are costing the nation millions simply be selfishly existing and it is not a laughing matter.
This is a bummer for me standing at a colossal skyscraping 5ft 7 inches, as apparently the average woman is 5ft 3. Help me, I’m doomed and a burden on society. Thankfully tall men are unaffected and may go about business as normal.


Leave Your Comment

Your email will not be published or shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>