Burden of Proof


I’m going to tell you something that might shock you: Science can’t prove anything. I know, I sounded like a creationist there for a second, but hear me out. The best way to support a theory is to submit it to a test where, if you don’t get the results you expected, the theory cannot possibly be true. For instance, if there were no overlap between the DNA of humans and the DNA of other primates, evolution would have to be overturned. When a test like that fails, it’s a pretty damn good indicator that the theory, or at least that part of the theory, is wrong. But if you get the expected results, it absolutely does NOT prove that the theory is right. It just proves that it could be. And since most theories are awfully complex, it’s pretty hard to prove them wrong either; you can’t always come up with a plausible risky test that would defeat the whole thing, since you’re often looking at multiple factors and sometimes at the wrong ones. Studies contribute to a greater dialogue; they increase the vocabulary we have to interpret the world. But they don’t “prove” things. They don’t “lay questions to rest.” This is generally how they’re reported, but it’s just not what they do.

Take the study in today’s New England Journal of Medicine which is being touted as “proving” that gastric bypasses are better for you than being fat. (The link goes to Reuters, but if you want a more balanced approach, check out the WaPo treatment.) Certainly there are things reporters won’t tell you – like the fact that suicide rates were three times higher in the surgery group (the AP report expressed confusion over the higher non-disease death rate among surgery patients – “my goodness, whyever could that be?” – but it’s not too mysterious when you look at the numbers). Or the fact that even in the “severely obese,” who could of course drop dead any minute if popular opinion is to be believed, there were only 321 deaths out of 7795 over the study period (compared to 213 in the bypass group – just over 4 percent versus just under 3 percent). Or the fact that many of the gastric bypass patients were probably still fat after surgery (to be fair, the study didn’t look at this either; another study in the same NEJM saw weight losses of 14 to 25 percent, but many of those were not gastric bypass surgeries).

But picking over the reporting is Kate’s territory, and she does it well; I’m not gonna step on her toes. I’m looking at the actual study numbers right now, though, and I have to tell you: they’re statistically significant and the methodology is fairly sound. The BMI cutoff for “severely obese” was 35, which I can quibble with – that’s 204 pounds on a 5’4” woman, hardly bedridden – but the average BMI was in fact in the range that has been associated with genuine health risks. Brian worried that deaths due to surgery had been quietly “corrected for,” and it’s possible – I don’t see any clear evidence either way – but it would be deaths on the operating table only, since they take time of surgery as the baseline. (They do mention, though, that post-surgical deaths might have been low — about the same rate as the controls — because the procedure was performed by excellent, experienced surgeons.) They basically did their homework. Their results should be heeded.

But they haven’t proven that obesity increases mortality, or that gastric bypass surgery is good for you, or that losing weight will save your life. That’s not how science works. They’ve contributed to the conversation, and their contribution is sound, but it will be interpreted much differently in the current context – surrounded by studies funded by the diet and weight-loss industry, couched in moral panic about the dangers of fat – than it would be in the context I would consider fair, where studies focus on habits and not mass, and scientists question whether weight itself can hurt you or whether perhaps unhealthy behavior is unhealthy across the board. And, of course, where dieting is revealed as one of the unhealthiest of unhealthy behaviors.

For instance, looking at these numbers, I’m inclined to say it looks pretty likely that for a (small) subset of the population, drastic measures for weight loss are, in the long term, less risky than continuing on a frankly self-destructive path. Within the popular paradigm, this is easily interpreted: fat kills, so less fat kills less, even when the weight loss comes at the expense of digestive integrity. But here’s my HAES interpretation: people who get WLS are generally required or forced to adopt good habits, verging on the orthorexic. You physically cannot overeat (or even eat normally), and you can’t keep down “bad” foods. Weight loss surgery itself doesn’t make you exercise, but exercise is generally a precondition – take care of yourself or you can’t get your stomach amputated! And some very fat people who get surgery will find themselves more mobile and therefore able to exercise for the first time in a long while. Plus, they may begin to value themselves for the first time in their life, and thus take more care of their health. Of course, some will feel despair, accounting for the astronomically higher suicide rate. But we can assume that many will lose weight for at least the first few years, and that they will consequently achieve a higher status in a fatphobic culture, both as No Longer Fat and as Properly Self-Flagellating (nothing says “I know I’m a disgusting pig and I repent” than having your guts cut out). They will be treated better by their doctors. They will treat themselves better. They will inch towards Health at Any Somewhat Smaller Size. It seems reasonable that this could make the modest difference between a 2.7 percent and a 4.1 percent death rate. (And for anyone inclined to say Occam’s Razor – if fatter people die more, death must be caused by fat! – I ask you to look me in the eye and tell me the higher rate for almost all diseases among African Americans is due to skin color.)

The study doesn’t “prove” my interpretation either, and it doesn’t disprove it. It adds a few words to the dialogue, that’s all. In the conversation science is having right now, my version isn’t even audible (in fact, science is sticking its fingers in its ears and going “la la la”). In the conversation that it could be having, this perfectly satisfactory study’s perfectly valid conclusions fit in just as nicely, if not better.

Because we diverge from conventional wisdom, we’re going to get accused – Kate certainly has been accused – of ignoring all evidence that doesn’t fit our preconceived notions. I can’t really be offended by this, because it’s such a common and almost inevitable human behavior that I understand why people are quick to diagnose it. (Well okay, I’m a little offended, but only because critical thinking is my job, so I irrationally expect people to assume better of me.) But you know, science isn’t immune to this behavior any more than its consumers are. Experiments are performed, carried out, and interpreted within a paradigm. They contribute to that paradigm, or they contribute to something new brewing on the horizon, or sometimes they contribute to both. What they don’t do is PROVE a damn thing.

55 thoughts on “Burden of Proof

  1. Brian worried that deaths due to surgery had been quietly “corrected for,” and it’s possible

    there were only 321 deaths out of 7795 over the study period (compared to 213 in the bypass group – just over 4 percent versus just under 3 percent).

    Awooga. A quick google gets me periop mortality rates of around 0.5%-2.5%, with the median quoted figure around 1%. If these were “quietly corrected for”, this could make a pretty dramatic difference to the actual results.

    Who funded the study?

  2. I was surprised this morning to hear no less a hype-er of the “obesity epidemic” than Sanjay Gupta feeling compelled to point out in his report on this study that part of the reason for the higher death rate among patients who didn’t have surgery might be that they were sicker to begin with.

    Apparently, the protocol for having WLS “weeds out” people whose underlying health issues make them poor surgical risks. So, the folks who had the surgery may have been healthier to begin with (and therefore less likely to die) than those who didn’t have it.

    Not having read the study, I can’t say whether this issue was accounted for or not. But if the researchers didn’t account for variation in baseline health, I think it would be interesting is to compare the morbidity rates of people whose health status was roughly equivalent to begin with, some of whom had WLS and some of whom didn’t. That might get us closer to the truth about the impact of WLS.

  3. NIH, or I woulda said something. :>

    If periop deaths are corrected for, it’s done silently — I can’t find an indication in the article that this happened. But like I say, they do acknowledge that there might have been an unusually low number of deaths directly related to surgery (vs. the deaths of malnutrition that undoubtedly make up some of the long-term mortality rate) because of the quality of the surgeons used. A sample that pulled from various hospitals would be more representative.

  4. JMars, my university login just stopped working, so I can’t quote directly from the article, but they did acknowledge the difference in medical care. Can’t remember what they said about it, but I’ll let you know if I can ever get to the article again!

    The point is, though, that the basic findings are probably more or less accurate. The problem is that nobody is asking these interpretive questions because that’s not the conversation they want to be having. A study concludes that x number of people with x BMI had died x years later. It’s the interpretive community that concludes “gastric bypass is awesome.” Kate does a fantastic job shredding media reporting, and we need that like whoa. But not every study (that isn’t getting shady funding) is totally unreliable. They just don’t necessarily mean what they’re being taken to mean.

    Hm… I also want to go back to the article, though, and see if they pushed the WLS clinic. They talk about the low number of deaths immediately following surgery, and my recollection is that this didn’t come off as an advertisement, but…

  5. Aha, it’s back! Here’s what they say:

    “Surgeons in this study denied surgery to less than 1% of subjects because of preexisting health conditions, suggesting that a bias with respect to health status was limited to very few patients.”

    But of course, proportionally the mortality rate is a “very few” patients as well. Still, like I said, they did their homework. It’s just that this very modest difference in death rates could be the result of many factors. There is nothing about the findings of this study that says “fat people should get stomach amputations” or “starvation is on the whole good for you” or “fat will give you cancer and gastric bypass will cure it” or any of the other ways it will inevitably be interpreted. Those have to do with the context, and how the results are viewed through that lens. It’s context that determines whether you look at this study and say “slice open the fatties” or “yeah big surprise, being restricted to eating two tablespoons of food at a time will reduce your blood sugar.”

    Here’s the sentence that makes me think that they did NOT control out the surgery deaths, but that they might be subtly shilling the surgery center:

    “Despite surgery-specific complications leading to death, the combined risk of death associated with both surgery and obesity did not differ from the rate of death for subjects in the control group. A possible contribution to this low rate of death during the first year is the fact that the center where subjects underwent surgery has performed more than 12,000 gastric bypass operations since 1979.”

  6. I wish I could articulate what I want to say, but out of all of this, the suicide rate strikes me as one of the saddest things I’ve ever seen. Insurance will pay for this tremendously expensive surgery and surgeons will spend hours doing the pre-op labs and evaluations and performing the actual surgery and doing the follow-ups… and apparently no one is taking two minutes to notice that some of these people, however successful they may be at losing weight, are really, really not okay. That just seems very sad to me. As if one’s body truly is one’s entire identity.

    Nice piece. Very well stated, though it ought to have as much effect on fat-haters as it does on creationists. :)

  7. OK, good to have an answer. I didn’t mean to start study bashing; I was just VERY surprised that Dr. Sanjay would make an even remotely critical comment about a study like this, rather than just shrieking “Slice up the fatties” while brandishing a scalpel (he is a surgeon, albeit a neurosurgeon, after all).

  8. secretly, no kidding. Although I should point out that the suicide rate, while much much higher among post-surgical patients, was still very low (15 of 7795, versus 5 for intact fatties). Still, considering that you often have to undergo psychological screening before surgery — although the article doesn’t say whether these subjects did — it seems really significant. In a different paradigm, it would warrant further research.

  9. Am i reading this right? so the bmi cutoff of 35 means that they would take no one ABOVE A 35? I think the media would have us believing that even a bmi of 27 poses “health risks”. So if they used BMIs around 35ish, no wonder they had successful surgeries- they probably operated on fat but fit folks (fit people always being better candidates for more successful surgery).

  10. Nono, 35 was what the designated as the “severely obese” cutoff. I quibble with it because it’s below what’s generally considered “morbidly obese.” The average BMI was higher for non-surgical subjects but in both cases was over 40.

  11. I don’t think the psychological screening is as good as it should be, though. My best friend was a multiple personality (from severe abuse by family when she was a child) and she passed the psych tests for WLS, even though she had been hospitalized several times for mental problems and was still cutting herself (I think, even though she was fat, that this was a permitted way for her to disfigure herself and she was smart enough to pass the tests). She eventually had to have her WLS done a second time, and she died within a year of myocarditis. I don’t know if her surgery contributed to it, but with all the complications she had, I’m just not sure. Her surgery was done at the University of MN by a doctor who teaches this surgery, he also did mine, which failed, but I don’t think either of us would be statistics that would be allowed in a study like this, we would be one of those quiet corrections.

  12. If anyone comes after this BMI 36 fat bitch with a knife, I’ll show them how “frigging” immobile I am. I’ll run so fast, it’ll blow their pomaded hair back.

  13. Ah, thanks for the clarification. My interpretation did seem too outrageous to be true.

    And Vesta, my god, i’m so sorry for your loss. I think that as insurance companies and doctors like ole Sanjay push this surgery to be the end all be all for fat people, we’re going to see a rise in experiences like your friend’s. They will overlook any underlying psychoses in order to just do the surgery. It’s truly tragic. Again, i’m so sorry.

  14. You point out that the risk of suicide was still quite low, but so was the risk of death without the surgery. So why is the slight decrease there something we need to be accept as significant while a much higher proportional increase in suicides is dismissed as merely “puzzling”. Even omitting surgery deaths on the table would represent a considerable distortion since lauredhel points out that even the median rate of 1% would erase nearly all of the benefit that the study claims are found with the surgery. I don’t know if they corrected for it, but I don’t know that they didn’t either and its an extremely valid question to raise given past shenanigans in fat statistical studies (remember the attacks on Flegal were based on her not selective controlling to advantage thinness).

    Also note that the authors conclude that doctors rejecting 1% of patients as too unhealthy meant that this wasn’t a significant factor yet then conclude that the 1% difference in mortality rates is significant. Why is 1% proof of insignificance one place yet proof of significance in another. If rejecting 1% of patients means very few were affected, why does that 1% become so important when they analyzed the results.

    While the media reporting does often sensationalize the results of these sorts of studies, that doesn’t mean that their authors were naive about this eventuality and it doesn’t mean they aren’t contributing to that outcome in their own analysis. The data may be accurate, but that doesn’t mean the conclusions drawn by the study are necessarily justified or ultimately meaningful. Remember, the media is largely turning to scientists, doctors, and academics for perspective on this study and they are loudly proclaiming is universal conclusiveness which is a reach to say the least. The media and the scientific community very much share blame on the sensationlistic reporting about fatness.

  15. I can’t access the actual study, but here are my questions
    Did the control group (the ones who didn’t get surgery) get the same medical care? WLS requires intensive medical follow up. Did the other group see the doctors as often? And, as you mentioned, doctors treat those who lose weight better. Did the non surgery group get the same old don’t come back until you have lost 50lbs treatment?

    Did they control for social class? Weight loss surgery requires either health insurance (and the probably of a more stable job) or lots of money. Did they make sure those in the 2 groups were of the same socio economic status?

    Were any of those in the non surgery group rejected for the operation? Because they were in poorer health to begin with? Had conditions such as alcoholism? “Bad” patients, those who didn’t take medications and were considered a poor risk?

    Did they take into account the placebo effect? We all know that believing something will work affects some people. That is why drug trials include control groups with sugar pills. Was this accounted for?

    Were any “creative” statistics used?

    Having read many of these studies myself for the past 20 years, I have found very few that have really been methodologically sound and really complete. When this much money is involved, usually the studies find whatever the sponsors want them to. And btw, who did sponsor this one?

  16. Oh for heaven’s sake. I did get at least the abstract from NEJM. In the U.S. study, they compared 9949 patients who had undergone WLS to 9628 “severly obese patients who applied for drivers licenses!!!” While it was only the abstract, chances are they did not know how often the “control group” saw the doctor. The abstract said they controled for age, sex and body mass. So it doesn’t look like they controlled for socio economic status. If all they did was just that and then look at cause of death, it wasn’t a very good study at all. Couse it would be better to read the full article but one has to be a member.

    http://content.nejm.org/cgi/content/short/357/8/753

    The second study was from Sweden where access to health insurance was not an issue. They said 129 deaths were in the non surgery group, vs 101 in the surgery group. The surgery group had a lower rate of DEATH from cancer, heart disease, but it did not report the differences in people actually getting the diseases. So did they have more episodes of disease, or just more death from heart disease? How much doctor bias was involved in their treatments?

    I don’t know why I asked questions in my previous post, cause I can’t read anyone elses post to get the answer (browser challenged!) Sorry if I said what has already been said.

  17. That is a really interesting study and it is great that they followed patients for a long time after surgery. I would tend to agree with the “landmark” characterization in the media articles because so few of these studies seem to be extensive or long-term enough to demonstrate anything.

    This is not the fault of the study authors, because there is only so much data. But what I really want to know is the average life expectancy of a WLS patient vs. the general population. I would be shocked if it is not eventually found that removing part of your intestine and completely changing how your digestive system operates, to the point that your hair falls out, etc. unless you are mainlining vitamins, does not cause more long-term harm than good. (e.g. if fat supposedly takes x years off your life, I would be really surprised to learn that the surgery doesn’t take something >x, especially since x is surprisingly few to people who think anyone “overweight” is going to be dead by 60).

    But this is a totally unfounded opinion and I hope I am wrong because the surgery is so widespread these days and I wouldn’t want to see anyone’s life shortened because of it. So I hope the results of this study continue to hold.

  18. BStu:

    So why is the slight decrease there something we need to be accept as significant while a much higher proportional increase in suicides is dismissed as merely “puzzling”.

    I think they’re both significant.

    Sherie:

    But that’s exactly my point. No one study can accommodate everything. That’s why each study is part of a larger conversation, and why no study can be said to “prove” anything. They point to things. They raise questions. They give us ideas for further research, which is always, ALWAYS needed (you’ll see that line in pretty much every reliably report). They suggest avenues of inquiry. But they do not and should not be said to PROVE anything. I think to even say “the study shows x” is misleading — “the study suggests x” or “the study shows x within the context of the study” would be more accurate. NO study can stand alone. The fact that we know enough about this particular issue to point out the study’s failings doesn’t mean it’s any more or less self-sufficient than any other scientific endeavor.

  19. Oh, and spacedcowgirl’s comment is an awesome example of what another part of the conversation might look like — in a less fatphobic paradigm. I doubt anyone will do that study, or if it does get done and it shows what we both suspect, I doubt it will be widely reported. But that’s what I’m saying here — that this study is part of a scientific tradition, that no study can stand alone or be said to “prove” anything in a vacuum, and that context (including possible avenues for future research) affects how we interpret results. Especially how the media reports them — but again, that’s Kate’s domain.

  20. Celeste, without revealing too much, busting on spin is my JOB. Part of being alert to deception means not discounting a study because you don’t agree with its conclusions. For the study these folks did, they controlled it well and acknowledged its limitations. They went overboard on their interpretation, and the media even more so, but much more important than tearing apart this particular study is looking critically at the idea that any study can “prove” anything (beyond its specific results) in the absence of other studies. The ground needs to be covered exhaustively before anyone walks on it.

  21. There’s something about the idea of having your own stomach amputated that makes the hair on my arms stand up on end.

    I think the number of people who might be helped by this is pretty miniscule, if any. I’m not sure if I agree that this surgery is the less self-destructive path than complusive overeating. If you did get fat by overeating, imagine the emotional agony of spending the rest of your life barely able to eat anything solid again without excrutiating pain! Even if this did lead to a longer life span, they might be some pretty unhappy and uncomfortable years. I mean, if you’re very promiscious, should you get surgery in which your genitals mutilated in such a way that you’ll never be able to enjoy sex again? You’ll certainly be less likely to get AIDS or other sexually transmitted diseases. I don’t think anyone would recommend such measures to treat promiscuity, so why do this to people who are fat? If you’re a complusive overeater, therapy would be a much saner solution to exploring the problem and working on it in a positive manner.

    Anway, whatever my disagreements, I found your post interesting and well thought out. Welcome fillyjonk, I look forward to reading more from you!

  22. Oh, and I meant to respond to Vesta: I’m so sorry about your friend, and stories like hers are exactly why I am angry (which I am, in case it’s not clear) about the way studies like this one are being overinterpreted and irresponsibly applied. For what it’s worth, they would have counted her in the study — whether patients lost weight or not wasn’t even tracked. They certainly didn’t conclude that people don’t die of gastric bypass; just that it’s rarer than people dying of obesity-related diseases, in a comparable population. In my world, the next studies would involve how many people’s deaths were directly attributable to the surgery (they didn’t break that out — just “disease) and whether it made a difference if you took them from a random sampling of hospitals and how class affected it and what’s up with that suicide statistic and whether quality of life was comparable and whether you’d even still get a 30% lower mortality rate if you looked at a different population of subjects. After you do all those, you can start to THINK about drawing a conclusion. (In this world, I don’t think most of them will get done at all.)

  23. And Rose, we don’t disagree at all. I have serious problems with weight loss surgery. It’s just that it would be unfair to let my problems with weight loss surgery influence my reaction to this study. What does influence my reaction to this study is knowing how irresponsible it is to draw conclusions from any one study standing alone, and knowing how preconceptions affect how studies are done and how they’re interpreted even in context. The study doesn’t have to be poorly-done for us to take the conclusions with a grain of salt.

  24. I think they’re both significant.

    The issue, though, is that the researchers didn’t. They valued the data differently to suit their conclusions. Even if the data is technically accurate, that kind of interpretive judgment is what is being challenged. We should feel the need to proclaim the validity of data to raise questions about the conclusions and priorities on display. That’s not what is being questioned. Part of that is going to be doing what I did and what Sherie did here and list the reasons the conclusions drawn aren’t justified. That’s not an attack on the data, but it does acknowledge that the scientists and study authors play a role in the hysteria over fatness. We should acknowledge that role. Its scientists who say that this study proves that fat is awful and we must lose weight at all costs. The media goes further, certainly, but we mustn’t forget that it is the scientific community’s own biases which started us down this path and there are a lot of deeply troubling things to note in how this data is being manipulated (as well as whether contradictory evidence by these researchers or others simply gets suppressed). They are the ones who are announcing that these studies do PROVE something. They aren’t talking about adding to the discussion or dispassionately offering mere data for consumption. They call this PROOF and we need to push back.

  25. Brian, I’m a little confused… you know you’re not disagreeing with me, right? And that what you wrote is exactly what I wrote?

    FJ: What they don’t do is PROVE a damn thing.

    Brian: They call this PROOF and we need to push back.

  26. Thanks, fillyjonk. I understand what you’re saying about this. I also agree that you shouldn’t reject these kind of findings off hand, but rather look at them carefully. The point isn’t to simply reject such studies as detrimental to the dialogue, but to figure out how to include it in the larger dialogue. Is there a way for the scientific community and the fat acceptance community to have a give and take in a decent and non-judegemental manner? Right now I’d say the answer is no, but by taking this seriously, you’re helping to create the conditions where it could possibly happen and I truly hope one day it does.

  27. Nancy, it’s not the only study that’s ever been published in this field that is, in and of itself, basically well-executed. But they’re far from having undertaken a careful study of fat, or of weight loss surgery. This is just one tiny aspect of that study, and most of the investigation isn’t being carried out carefully at all.

  28. Rose, much better put than I could have managed!

    Now my long day of checking spin and encouraging critical thinking (it really is my job) has ended in 1) forgetting to eat lunch and 2) alcohol. I’m going to drag my sorry drunk ass over to the farmer’s market before all the good stuff is gone.

  29. That was my point fillyjonk. A number of your posts now have seemed to position your concerns in contrast with what frankly seems like a strawman: this presumed reflexive or emotional rejection of the study. I don’t suggest this is a strawman of your creation. I gather you may have been trying to be proactive in defense of such charges, but doing so does give it weight. No one is saying that the researchers made up data. Insisting that you aren’t doing that, though, would imply that others are. No one should be made to feel that they are “study bashing” for raising valid concerns which question the conclusions drawn by the authors, publishers, and media and I’m troubled that some people have felt that way. Though I don’t believe that was your purpose or intent, I think we should recognize that asking these questions isn’t something we need to justify or apologize for. It isn’t reflexive or emotional and we should outright reject efforts to imply that it is. Those are charges we shouldn’t have to defend ourselves from and we should imply that people are doing that. We don’t need to laud the execution of the study to recognize that its own interpretations and especially the conclusions of others are dangerously misrepresenting its findings.

  30. the combined risk of death associated with both surgery and obesity did not differ from the rate of death for subjects in the control group.

    OK, so in fact the surgery did NOT improve a person’s odds of staying alive? (Once you factored in death from the surgery as well as death after the surgery)

    And they’re using this as proof that WLS works?

    I’ve always thought about WLS maybe some of my health problems would improve, but I would likely develop different health problems. Given that the gastrointestinal system is one of organ systems in my body that actually works pretty well, I’ll keep it, thank you.

  31. Someone on one of the HAES lists I read pointed out that the study followed the subjects for only 7.1 years, and that many of the problems with nutritional deficiencies and such after gastric bypass really only start showing up in notable numbers at 10 years. I’d like to see the numbers on this study after another 5-7 years, I suspect there’d be rather a difference.

    I’m concerned that the quality of the control group wasn’t good, too. They picked people who reported as fat on their drivers’ licenses[*]. (How do they know the figures were accurate?) But what else did they control for besides BMI? As commenters above have also asked.

    Paul Ernsberger and Paul Campos, in the Washington Post article, point out that the study actually shows that mortality for non-surgically altered “morbidly obese” is far, far lower than most obesity researchers and surgery cheerleaders would like us to believe.

    And the big question: what’s the quality of life like for the surgical group compared to the control group? What good is a “30-40% decrease in risk of death” if you’re living with a lot of unpleasant side effects?

    I am eager to see Sandy Szwarc’s take on the study.


    * I am so glad that Australian states do not require you to put your weight on your drivers’ license or passport! (For a multitude of reasons, not me being personally ashamed of my size.)

  32. Excellent start fillyjonk! I’m impressed with this blog. My eyes were blaring at the African American comment* too.

    *I’m mixed (Puerto-Rican, African American, and Algonquin). I wasn’t offended or anything (Not at all), but I LOVE LOVE LOVE it when racism (In this case: African Americans as an example) and fat hatred are compared for the use of an attack point. Though some of the weight bigots will contend they aren’t the same, I’m always one to come in and say “Oh really? You’re going to say my opinion for me? What are you going to do next?” :P

  33. And the big question: what’s the quality of life like for the surgical group compared to the control group? What good is a “30-40% decrease in risk of death” if you’re living with a lot of unpleasant side effects?

    DING DING DING DING. Was it Mark Twain who said something to the effect of, if you give up all your so-called vices, will you actually live longer, or will it just seem that way?

    Anyway, I don’t think of my stomach as a “vice.” But I’d sure feel like it was in a vise if they put a giant rubberband around it. Ow, ow, ow.

  34. A number of your posts now have seemed to position your concerns in contrast with what frankly seems like a strawman: this presumed reflexive or emotional rejection of the study.

    Brian, this is Fillyjonk’s first post here, so you must be referring to her previous comments. Just to be clear.

    Though I don’t believe that was your purpose or intent, I think we should recognize that asking these questions isn’t something we need to justify or apologize for.

    And I think Fillyjonk does recognize that. Not to mention, I wrote about exactly that last week. Fillyjonk was the first commenter, agreeing with me (albeit indirectly).

  35. Stef, sorry, that quote’s not clear; it refers to the first year of the study, when death rates were comparable (.53% and .52% respectively).

    Jon B., yeah, I know people (rightly) have a problem with that comparison; in this case I’m only using it as an example of why the apparently simplest explanation can, in a complex situation, still be the wrong one.

    Brian, I think you’re misreading my post, and accusing me of a logical fallacy like straw man is a pretty serious charge.

    La Di Da, 7.1 was the average number of years that they followed people. Though yeah, a longer-term study would be great! I bet they’d have very different findings. And as I mentioned above, in my ideal world, a quality of life study would be another part of this conversation. To answer your other questions, they did a regression on the driver’s license figures based on usual misreported weight, although when they matched subjects (which is where most of the data came from) I believe they used the reported weights. They matched for average age, average BMI, and sex, by the way. Class wasn’t included and that would be another good study.

    I’m sorry, guys, but this study was pretty okay. It’s trying to treat it as a conclusion rather than a study — which the people who conducted it do, as I’ve said, and which the media will do more egregiously — that’s the problem. It’s not an answer; it’s a data point. All studies are data points, some good, some bad. The problem is that most of the country (world?) is willing to just go ahead and slot those data points, uncritically, into a framework that makes sense to them. This is typical for how science works in society, but it’s also the reason we need an overhaul.

  36. Oh cool! With the new format I can read the comments! Yea! Fillyjonk, I completely agree with your position that no study can cover everything, and no study proves everything. As a person with a background in sociology, I have had excellent research training. I even teach my own students how to study a study. I do feel that while they can’t control for everything, they should at least control for things that are highly significant. Social class would be significant, as richer people die earlier that poorer people. Certainly access to medical care would also be important. If they did what I think they did only from reading the abstract, they knew very little about the control group other than what they died of. Comparing two groups that get equal medical attention is necessary to form any kind of valid conclusion, and not that difficult. Also, how many people tell the truth about the weights on their driver’s licenses? I am speaking about the study I linked above. Usually when studies involve numbers that large, they tend to know little about participants. Data is either collected thru surveys or medical records. People are shocked studies could have that many holes in them, but they don’t realize many times that is deliberate to get the results they want!

  37. I even teach my own students how to study a study.

    Awesome, I’m glad someone is. I’m involved in critical thinking education too and sometimes it feels like there aren’t very many of us.

    Comparing two groups that get equal medical attention is necessary to form any kind of valid conclusion, and not that difficult.

    No, you’re absolutely right here. There’s a lot of medical care involved in getting WLS, and presumably they couldn’t know that their control subjects had the same level of care. They did at least control pretty strictly for location, if I recall correctly, but how much that has to do with class varies by the location. Again, though, the problem is as much with looking at that study, brushing off your hands, and saying “well, we’re done here!” Which is what the researchers are apparently inclined to do, undoubtedly because there’s a push for sound-biteable conclusions. If they were looking at science as a process, this study would suggest some interesting new directions to go in — another study that matched for class, or that looked solely at class and how it interacts with weight and habits to affect mortality (we’ve not had nearly enough of these).

    Also, how many people tell the truth about the weights on their driver’s licenses?

    They actually took this into account, but until I can look at the article (which I can’t from home) I’m not able to say with authority how it was done.

  38. “the combined risk of death associated with both surgery and obesity did not differ from the rate of death for subjects in the control group.”

    OK, so in fact the surgery did NOT improve a person’s odds of staying alive? (Once you factored in death from the surgery as well as death after the surgery)

    And they’re using this as proof that WLS works?

    What Stef said. We need actual numbers, I think. Because right now, it looks like the death rates are actually equal. If I’m interpreting this correctly, if you’re in the Going To Die group, WLS either kills you RIGHT NOW or via grinding unrelenting postop misery eventually resulting in suicide, whereas no-WLS kills you from natural causes sometime down the track.

  39. Well I wasn’t offended by it fillyjonk (Just putting it out there, because it is sometimes a hot button when spoken). Like I said: Great comparison for an attack point. These generalizations about a person’s size annoy me way more than the use of racism as a comparison to fat hatred. :P

  40. I missed the boat on all this discussion, but I wanted to chime in and say this is an excellent post, and the broader point about science as a conversation and not a set of definitive proofs is an incredibly useful one. U R SMRT

  41. http://junkfoodscience.blogspot.com/2007/08/was-this-really-proof-that-bariatric.html

    Well, Sandy Szwarc wrote about the Swedish study and she doesn’t think much of it either. Surprise, surprise, it was funded by a grant from Hoffman -LaRoche and AstraZeneca. The head researcher has ties to a whose who of pharmaceuticals. For those readers who wonder why some of us automatically greet any weight loss study with cynicism, it is because the finger almost ALWAYS points back to BigDiet/Pharma. I have been researching this issue for over 20 years and it is always the same story! Yet, the media spins these studies to perfection! It is little more than marketing!

    BTW, she did read the actual study and points out that the control (non surgical) group was slightly older and had more heart disease and diabetes to begin with.

  42. All right, I get the picture; nobody wants to talk history and philosophy of science. I guess I knew that even when I majored in it.

    Laura, thanks, but I think you’re in the minority! The reframing doesn’t seem to have gone over well. Personally, though, I’m always going to think that if you discredit a particular study, you’re left with just a deflated study; if you analyze and rethink the way that studies work together and contribute to/are influenced by public opinion, you can actually get somewhere.

  43. U R SMRT

    IAWTC.

    And Fillyjonk, I think both are important. It’s absolutely crucial to hold these scientists’ feet to the fire when their methodology is questionable and their conclusions are blatantly biased. It’s equally crucial to keep any given study in perspective.

  44. “if you want a more balanced approach, check out the WaPo treatment…”
    “[the researchers'] contributions are sound….They basically did their homework. Their results should be heeded.”

    Now, I know the fat community has gone insane.

    This study was criminal in its manipulations designed to reach the sponsor’s conclusions. This is about scientific integrity and the fat community should be screaming murder.

    Instead, it’s giving “kudos” for spending more for bariatrics.

  45. Fillyjonk, I critique studies because that is what I was trained to do as a sociologist. We spend an entire semester in research methodology picking out flaws in research design. It is also the purpose of peer review and one of the reasons studies are published in the first place. That does not mean critiques in study design and the paradigms behind the studies are mutually exclusive. I agree with Kate, both are important. Since studies have degenerated today into nothing more than marketing tools, I do think it is important for the public to know that.

    Equally important is the way media spins the studies. I can’t link WaPo, but remember that stupid fat is contagious study? The MacNeil Leher Newshour spun it like it was perfect. Not one valid criticism, all the questions asked provided the health reporter the opportunity to make it dazzle! Have you ever heard of a VNR (video news release)? It is a segment in the news, with a “real” newscaster that is actually scripted by the sponsor. It is a bought and paid for ad made to look like news. Not just in the area of fat, but all kinds of things. With this going on we all should be on guard.

    Of course you are right that we need to challenge the basic assumptions of fat hatred, and the limited “fat will kill you” mentality of modern medicine. Why not challenge on all fronts and leave each to do what they feel most called to?

  46. As I’ve said upthread, my job involves questioning distortion and spin. But the most important skill that someone in my line of work can have is not logic or knowledge — it’s the ability to tell when you need to let something go. In order to be nonpartisan and thus reliable and convincing, you have to know the difference between disagreeing with something and knowing it to be false. We’ve all got convictions here, so sometimes we’ll be chomping at the bit to do a story because someone said something we find outrageous, and the boss has to say “we’ve got nothing solid here.” Often I think something is being spun when really it’s just an interpretation I disagree with. And I can argue with that person, but I can’t, in an official capacity, call them a liar.

    Some studies have flawed methodology; we can critique those. But when a study is flawed not in its methodology but in its philosophy, then we need to look at that philosophy. I think that’s the case with this study, and I think it’s the case with most studies that are carried out within reasonable scientific parameters. Are their results false? No; when they examined these subjects, they found 100 deaths vs. 300 deaths. Are their interpretations biased? Are they operating in a paradigm that’s based on flawed premises? Did they design the study based on flawed premises? These are useful questions.

  47. Pingback: boystowners.com » Things you can tell just by looking at...

Comments are closed.