Aug 21 2014

Researching Magic

You are currently browsing comments. If you would like to return to the full story, you can read the full entry here: “Researching Magic”.

Share

57 responses so far

57 Responses to “Researching Magic”

  1. BillyJoe7on 21 Aug 2014 at 9:14 am

    “Cherry picking” is by far the favourite game played by CAM practitioners and supporters.

    I wonder if you should have added a point about “cherry picking” to head off the inevitable backlash of CAM practitioners and supporters linking to individual studies that seem to show an effect of various CAM modalities. About one in twenty will be positive even if there is no effect.

  2. Kieselguhr Kidon 21 Aug 2014 at 12:26 pm

    I hear you, but I think the kind of rule you’re suggesting would have miserable unintended consequences. A strong attack line that the psuedoscience community uses is to suggests some massive conspiracy by establishment scientists to cover up the awesomeness of (say) homeopathy, and this kind of blanket ban on funding will surely exercise people still more. That’s hardly guaranteeing an outcome of greater public belief in nonsense, but it’s worth considering.

    In general I’m a bit surprised how much the public admires science and scientists but distrusts us too. I remain amazed how much currency the idea that global warming is a conspiracy of scientists — that scientists are entirely devoted to their own predicatable and self-reinforcing grant interests and are devoid of public spirit and the hope to help make things better for everyone — has. That idea shows up in GMO topics too; setting aside the scientific issues there I remain amazed how pervasive the assumption is that the scientists are, ultimately, up to no good, just because they’re scientists. That’s a really powerful delusion because at one blow it shatters much of the good folks like Dr. Novella might try to do: you can talk about the scientific method, and evidence, and experiment, but of course if you don’t trust the experimenter then all that is so much nonsense.

    There has been a movement — and I know this movement creates rifts in the skeptical community (of which I’m not a participant, though I imagine I see many things the same way) — you see that a lot in a very aggressive New Atheism that has gotten a lot of discussion in other venues. Personally I’m worried about that underlying distrust of scientists and I think it’s potentially a lot more harmful than the promotion of a few quack cures — but I’m not so sure, and other folks are welcome to think differently of course. But I worry about the increasing trend toward, say, harshly dismissive and mocking responses to folks who raise some wacky, ludicrous concern about (say) GMOs, and I try — probably badly — to be as accommodating and helpful as I can be and to recognize the questioner’s fears and concerns with a “well, we’re concerned about that too, and it’s a legitimate worry, but here’s the precatuons we take and are taking, and why” attitude.

    I think blanket bans will get a lot of peoples’ backs up, and undermine that sort of approach, thus becoming counterproductive.

  3. The Other John Mcon 21 Aug 2014 at 12:34 pm

    KK — i’m not understanding or seeing the blanket ban to which you are referring

  4. Steven Novellaon 21 Aug 2014 at 1:03 pm

    BJ – The 1 in 20 figure is likely an underestimate.It assumes perfect research technique and only accounts for statistical effects. But researcher bias and publication bias can increase the figure significantly.

    KK – No one is talking about a ban. Let any researcher who wishes public funding make a case that their research is worthwhile and ethical. The NCCAM is creating a double standard in favor of researching nonsense. I just want to get rid of the double standard.

  5. jsterritton 21 Aug 2014 at 2:01 pm

    “…a completely bogus therapy may enjoy 20 years of false legitimacy before definitive trials show that it is completely useless.”

    A defining characteristic of alternative modalities is that they have very little prior plausibility and virtually no evidence supporting efficacy. When there is some evidence, it is (famously) a dangerous thing. People selling or cheerleading for CAM modalities for whatever reason can cherry-pick some data from some studies — no matter how preliminary or flawed or contravened by better studies — or even, as we have seen, from studies whose authors claim false conclusions not supported by their own data. Then, as we often see in comments in these pages, supporters of CAM will fall back on shifting the burden of proof with the well-worn “well, show me the studies proving [insert CAM modality] doesn’t work.”

    The cherry-picking and presumption of equivalence (with better studies and better conclusions) allows sCAMmers to claim scientific legitimacy for their sCAMs. Without a standard of proof for safety and efficacy, CAM magical thinking and echo-chambers allow modalities to get a toehold and snowball into wide use, which of course lends an ever greater (though less-deserved) air of legitimacy to these practices.

    To me, it seems like CAM modalities should either be studied exhaustively and in a hurry to scientifically quash sCAM modalities before they can do much harm or CAM modalities must be held to standards of rigor and proof that they currently aren’t. Of course, this is a vast oversimplification of the problem. The former approach squanders precious resources on hocus-pocus, the latter requires a wholesale change in regulatory practices and sidesteps science with politics and all the slings and arrows that mess is heir to.

    Maybe an information-based system could be developed to assign an index value to treatment modalities based on current scientific understanding. A basic algorithm, weighted in favor of factors like number of studies, journal impact factor, etc (and against factors like no known or likely mechanism of action) could be applied to treatment modalities and provided to the curious on anything from a website to product packaging. For instance, under “H” for “headache” aspirin might have a score of 9, while reiki would score a 1 and acupuncture a solid 3 (3 representing the best placebo response produced by any modality). I know it sounds like a pipe dream, but imagine any consumer being able to see modalities ranked by known, scientifically established efficacy!

  6. jsterritton 21 Aug 2014 at 4:45 pm

    Dr Novella…

    The journal article is excellent. Nice job!

    “These trials degrade the scientific basis of medicine by treating modalities where the basis rests in prescientific thinking as though they were well-supported science- and evidence-based modalities, while clinical investigators labor under a seemingly reasonable delusion that negative RCT results will lead to the abandonment of CAM and IM modalities that fail to perform above placebo in RCTs.”

    I never really thought about RTCs as being a bad thing, because I am overwhelmingly guilty of the kind of thinking the above statement addresses (the “seemingly reasonable delusion” that the truth will out). Is it really beyond hope to ask that sellers, developers, and practitioners of CAM demonstrate preclinical evidence in support of their modalities? I’m not being rhetorical — I’d love to know if it is something that could actually happen.

  7. Ekkoon 21 Aug 2014 at 5:15 pm

    @jsterritt,
    ” Is it really beyond hope to ask that sellers, developers, and practitioners of CAM demonstrate preclinical evidence in support of their modalities?”

    They are supposed to have better than just pre-clinical evidence for claims made. If you’re talking about the US specifically, the FDA allows “structure-function” claims to be made for dietary supplements. Under DSHEA, when making this type of claim you have to notify the FDA of what the claim is within 30 days of it being brought to market. You also have to put the FDA disclaimer on the product label (that they haven’t evaluated the claim, yadda yadda). And, you have to have “substantiation” of the claim being made. DSEHA originally did not define what “substantiation” meant at all so the FDA later released guidance documents: http://www.fda.gov/food/guidanceregulation/guidancedocumentsregulatoryinformation/dietarysupplements/ucm073200.htm

    Basically the manufacturers have to have clinical evidence supporting their claims in case the FDA ever asks them for it, but that’s basically it as there is no market pre-approval in the US. In other countries, there is pre-approval of the evidence beforehand.

  8. BillyJoe7on 21 Aug 2014 at 5:54 pm

    Kieselguhr Kid,

    You are labouring under the delusion that accommodating alternative views works better than confronting them.

    I challenge you to provide evidence that this is true. In fact history shows that this is not true. Think of all the great equality movements in history (blacks, women, gays). None of them could have been achieved their goals in changing societal attitudes without confrontation (ie Martin Luther King).

    And what exactly are you referring to when you say “aggressive New Atheism”?

    This is a term applied by religious apologists – and, unfortunately taken up by some atheists themselves – who cannot offer counter-aguments to the issues raised by atheistic ideas. There is nothing “new” in today’s atheism (ie Robert Ingersoll), and the word “aggression” simply means views strongly held and expressed. Some would prefer to have certain ideas silenced, and take attacks on their own ideas as personal attacks. It’s just easy to dismiss it all as “aggressive New Atheism” than to answer questions about their evidence-free and illogical their beliefs.

  9. jsterritton 21 Aug 2014 at 6:35 pm

    Of course, many “alternative” modalities are not really alternatives at all, because the modality is being used in the absence of what I would call a valid diagnosis. In other words: there is no evidence-based alternative because the problem itself is being manufactured and peddled to the credulous. I’m talking about dangerous “alternative” therapies like long-term antibiotic treatment for “chronic Lyme disease” and nasty quack autism “cures.” In the case of CLD, the treatment is ostensibly to combat bacterial infection, but no evidence of infection can be found. Not a trace (“diagnosis” is made based on subjective complaints comprising a rainbow of symptoms). The popularity of the diagnosis and its “alternative” treatment has resulted in a number of high-quality RCTs just to see if clinical observation can be found in support of the therapy’s effect on symptoms (none found). The resources brought to bear on the scientifically unsupported diagnosis of CLD, coupled with the negative results, should have spelled an end to such “treatment,” if not the dubious diagnosis itself. After all, the treatment poses significant risk to the patient (e.g., months- and years-long IV antibiotics requiring multiple surgeries for port installation) and risk to community health (e.g., antibiotic resistance). And yet, as Drs Gorski and Novella point out, these RCTs, despite their negative results, provide a patina of scientific legitimacy for the quacks to trade on. It also creates a literature to point to, cherry-pick, criticize, and otherwise use to make it seem like there is a rich scientific tradition surrounding a modality, perhaps even a controversy (paydirt for sCAMmers).

    I guess what I’m saying is that the sCAMmers are as dangerous as the sCAM — maybe more so. Can we label them?

  10. grabulaon 21 Aug 2014 at 8:50 pm

    My gut reaction is to say we should save the money for real research, especially when we know the concepts involved would have to break the known laws of the universe BUT…I think what it does is help lend credibility to educating people.

    Just saying, ‘well homeopathy can’t be real because it breaks the laws of physics’ isn’t as substantial as saying ‘after hundreds of studies into homeopathy, science has found there to be no plausible working mechanism’. This is why I think the research needs to be done, if nothing more than to hopefully move towards a more rational society.

    We’ll never hit 100% but as an example, look at the increase in secular society over religious society as our world has become more rational. We’re gaining ground, and we won’t gain everyone but overtime we can work towards marginalizing magical thinking.

  11. Steve Crosson 21 Aug 2014 at 9:33 pm

    @Kieselguhr Kid,

    I challenge you to name even one outspoken atheist who can legitimately be characterized as being anywhere near as aggressive as the likes of The Westboro Babtist Church or ISIS or dozens of other radical religious groups.

    There are also hundreds of mainstream religious spokesmen who regularly, very aggressively denounce atheists or any other religious group they happen to disagree with. It is very rare for any atheist to match even their somewhat lower level of intensity, let alone exceed it.

    You are falling for the “big lie”. The religious right has been making a concerted effort to play the “victim card” by attempting to claim that atheists are aggressively trying to deprive them of their “rights”.

    In actual fact, atheists have only been asking for rights equal to those which Christians and other religious majorities have always enjoyed. It is ludicrous to claim that the white, Christian majority is the victim — they have been persecuting atheists and religious minorities for two thousand years.

    BillyJoe7 is absolutely right. The actual underdogs will never get respect until we demand it.

  12. Steve Crosson 21 Aug 2014 at 9:51 pm

    grabula on 21 Aug 2014 at 8:50 pm

    Just saying, ‘well homeopathy can’t be real because it breaks the laws of physics’ isn’t as substantial as saying ‘after hundreds of studies into homeopathy, science has found there to be no plausible working mechanism’. This is why I think the research needs to be done, if nothing more than to hopefully move towards a more rational society.

    I’m inclined to agree with this. Sadly, common sense isn’t very common, and simply saying that homeopathy defies common sense isn’t very convincing to people who willingly fall for the “3 out of 4 doctors” trope.

    We need to have at least some legitimate studies to be able to compare and contrast, and most importantly, be able to convincingly say that the “results are no better than placebo”. Otherwise, it is too easy for people to be given false hope by a few anecdotes.

    I hate to waste research money as much as anyone, but as the SGU “What’s the harm?” segments often point out, there is real potential for significant harm and I think that a more active approach is justified.

  13. grabulaon 21 Aug 2014 at 10:25 pm

    “You are falling for the “big lie”. The religious right has been making a concerted effort to play the “victim card” by attempting to claim that atheists are aggressively trying to deprive them of their “rights”.”

    A few days ago on that cesspit facebook a buddy of mine posted something about share if you like some meme about letting school kids say the pledge of allegiance with ‘under god’ in it. I replied no thanks. Later his dad came in and wanted to know why I was trying to deny him his religious freedom. I had to point out that not only wasn’t I trying to deny his religious freedom – he can practice his religion freely at church, at home, or wherever else it might be appropriate but that in actuality he was trying to force his religious beliefs on those who may not share them. Religion and woo have a way of blinding people to the rational and it’s important we have the tools to help educate over time.

  14. Steve Crosson 21 Aug 2014 at 11:19 pm

    grabula on 21 Aug 2014 at 10:25 pm

    A few days ago on that cesspit facebook a buddy of mine posted something about share if you like some meme about letting school kids say the pledge of allegiance with ‘under god’ in it. I replied no thanks. Later his dad came in and wanted to know why I was trying to deny him his religious freedom.

    One of the reasons I don’t do Facebook (well, that and their really sleazy policies combined with zero respect for their users).

    I’m stuck living in a deep red state and if my neighbors ever realized I was an atheist or a “goddam librul”, I would be ostracized or worse.

    Fortunately, the chances of any of them ever willingly reading Neurologica is less than that of finding a molecule of active ingredient in a homeopathic vaccine.

  15. Charonon 21 Aug 2014 at 11:34 pm

    Just saying, ‘well homeopathy can’t be real because it breaks the laws of physics’ isn’t as substantial as saying ‘after hundreds of studies into homeopathy, science has found there to be no plausible working mechanism’. This is why I think the research needs to be done, if nothing more than to hopefully move towards a more rational society.

    I’m going to object to this… I would, of course, because I’m a physicist. But still. As David and Steven say in their article, the physics conclusion is way more secure than RCTs. Medical research is hard because it works with incredibly complex systems (humans) with tons of confounding factors. Physics works with very simple systems. Also, we’ve been rigorously testing physics for four centuries, so we’ve got a few centuries head start on medicine.

    Saying something is physically impossible is the strongest scientific statement we can make. I don’t care how awesome your RCT is, you’re not going to match that.

  16. Charonon 21 Aug 2014 at 11:43 pm

    BTW, you might object that Reiki or homeopathy or whatever is taking advantage of effects we don’t have in our physics, so we can’t get a meaningful prior. To someone who’s never studied physics, this is a perfectly reasonable objection – science discovers new things all the time, right? But you have to appreciate that we know all the relevant physics. We really do. Newton wasn’t overturned by Einstein – Einstein broadened the domain of applicability for our theory of gravity, but Newton is a great approximation in most circumstances. The same will be true when we finally get a quantum theory of gravity (maybe string theory) – it won’t prove Einstein wrong, it will just extend the domain of applicability.

    But the domain of applicability of our current theories of physics is everything humans will ever directly experience. Seriously. These laws fail only under utterly extreme conditions like the centers of black holes, the Big Bang, and if we’re lucky, tiny failures in the most energetic collisions in world’s most advanced particle accelerations.

    We understand the physics underlying everyday life.

    No, seriously.

    Seriously, people.

  17. grabulaon 22 Aug 2014 at 12:02 am

    “I’m stuck living in a deep red state and if my neighbors ever realized I was an atheist or a “goddam librul”, I would be ostracized or worse”

    I don’t know, I was in the army and now I’m in the guard surrounded by cops and the like. I’ve always had ‘liberal’ leanings though I don’t officially identify with any political movement personally. I’ve never caught as much trouble as you would think for being surrounded by a lot of heavy conservatives. There are some topics I meet eye to eye with but more importantly I find most people aren’t really as much of something as they think. After talking to them I generally find we’re not as different as some would like to believe. It’s a generalization but it’s held up pretty well.

    I do facebook to keep in touch with friends and family more easily but I’m pretty outspoken about my own beliefs. Occasionally I get removed from someones friends list for it. I find that sort of behavior a little childish.

  18. jsterritton 22 Aug 2014 at 12:14 am

    @Steve Cross

    “We need to have at least some legitimate studies to be able to compare and contrast, and most importantly, be able to convincingly say that the “results are no better than placebo”. Otherwise, it is too easy for people to be given false hope by a few anecdotes.”

    There are any number of studies — good ones — showing overwhelmingly negative results for CAM modalities. Again and again and again. They do not appear to carry weight, though, since the spurious modalities are still peddled. Then there are the bad studies: clinical trials looking for any evidence of efficacy with no real plausible hypothesis being tested — just “we want to believe this modality works and we can muddy the waters of science with our confirmation bias.” What Drs Gorski and Novella are saying is that the very idea of skipping ahead to clinical trials in the absence of plausibility and/or a possible mechanism of action is a dodge used by CAM proponents to sidestep the unreality of their various health claims. Moreover, as Gorski and Novella point out:

    “In RCTs testing modalities with low pre-test probability (i.e., low plausibility), confounding effects are vastly magnified, easily producing false-positives.”

    My takeaway from their journal article is that since RCTs absent prior (good) probability/plausibility are likely to yield lousy results, it should be incumbent on scientists to demonstrate plausibility via the usual methods of rigorous science (basic science followed by in vitro and animal studies; not magical thinking and case studies (anecdotes) followed by leapfrogging to human trials) before undertaking large-scale human trials. CAM proponents are able to use RCTs to obscure and not illuminate. And those hoping to de-bunk CAM with RCTs, no matter their good intentions, cannot avoid these pitfalls either. Good scientists use cautious language and look at all the data; sCAMmers will mislead, move goalposts, and cherry-pick. Further, since this research necessarily draws resources from other, worthier endeavors, it is important to prioritize good science over science that is getting us nowhere.

    Good science and bad has been demonstrating for too long that negative results from RCTs are not helping to take the wind out of any CAM sails. On the contrary, CAM proponents are abusing the scientific method itself and twisting it to suit their mystical and implausible beliefs. We see it here in these pages all the time: goalposts move, the burden of proof is shifted, data is endlessly sifted, mined, and cherry-picked. Regardless of their quality and conclusions, these RCTs are generating noise that is used to drown out the fundamental truth lying at the heart of CAMs: they don’t work. And in many cases the “why don’t they work?” is as simple as “how in the world could they possibly work?”

  19. BillyJoe7on 22 Aug 2014 at 12:50 am

    I side with jsterrit on this.
    Because of zero plausibility, there never was any justification for doing clinical trials on homoeopathy, and there certainly is no justification for doing any now. If all the negative trials to date haven’t convinced everyone, then nothing will.
    They’ll just continue to cherry pick the few shoddy trials that show some positive effect or that show an effect by pure chance.

  20. grabulaon 22 Aug 2014 at 1:01 am

    ” If all the negative trials to date haven’t convinced everyone, then nothing will.”

    I’m not advocating ongoing and continuous study. A handful of studies to confirm what we already know would be fine.

    In an ideal world it would be nice if we could require CAM supporters perform good scientific studies on tier own to show efficacy but we all have seen how that goes.

    Ultimately, I know the rare moment I’ve made someone think or reconsider a stance it’s almost always because I have solid evidence to support my claim. The more accessible that evidence is, the more readily people will accept it.

  21. jsterritton 22 Aug 2014 at 1:54 am

    Grabula…

    “I’m not advocating ongoing and continuous study. A handful of studies to confirm what we already know would be fine.”

    As Gorski and Novella explain, these types of studies should not be done absent any pre-test probability or plausibility. Reverting to my “chronic Lyme” example, RCTs tested the efficacy of the popular, controversial treatment (long-term antibiotics). However, since there was really nothing to test for — there was no bacterial infection present (hence no mechanism of action) — they tested to see if patients’ subjective and far-flung symptoms (e.g., mood issues, “brain fog,” headache, sleep difficulties, dizziness, anxiety, depression, etc) resolved with treatment. As you might expect, the results were mostly negative (the same as placebo) with some groups reporting improvement of certain symptoms at certain points in the trials. Reports of improvement came from both treatment groups and control groups, and there was no discernable pattern. In other words: the trial produced exactly what could be expected: noise. This should have been a killing stroke ending the treatment practice entirely. But the cautious language of the good scientists in their conclusions (“further study might…,” “cannot be ruled out entirely…” and so on) along with individual data points (3 people claimed to suffer fewer headaches at week 3) allowed CLD proponents to rightly criticize the studies for only producing noise while cherry-picking the noise that suited them and calling it signal. CAM jiu-jitsu is mighty (and devious).

    No good is coming from these RCTs. Science can never prove a negative and CAM proponents will accept nothing less. I would like to see funding for RCTs absent a sound, plausible hypothesis dry up to nothing. No more leapfrogging to the head of the scientific line of inquiry. No more cherry-picking the types of studies that suit your BS along with the cherry-picking of everything else.

  22. grabulaon 22 Aug 2014 at 2:12 am

    @jsterritt

    “As Gorski and Novella explain, these types of studies should not be done absent any pre-test probability or plausibility.”

    I understand their point, and I sympathize with the fact that they in their fields have to compete for money with CAM modalities we all are pretty confident don’t work. I still disagree that they serve no purpose, though as I mentioned before within reasonable limits.

    “But the cautious language of the good scientists in their conclusions”

    I tend to think of this as scientific language, because science doesn’t deal with absolutes. It’s certainly possible to use stronger language, and sites like this blog take that data and give it more substantial meaning. That’s where I feel the battle can be won.

    “CAM jiu-jitsu is mighty (and devious).”

    Certainly, but ignoring it won’t make it go away. Unfortunately human beings are credulous and generally ignorant until they’re exposed to information. Everytime Dr. Novella blogs about issues I know people around me aren’t clear on, I post on FB. I do this with articles from SBM and a few other sites to show that as science is done, we move closer to a consensus on something and the science is where the truth lies, not in the practitioners of magic.

    “No good is coming from these RCTs. Science can never prove a negative and CAM proponents will accept nothing less.”

    I disagree. I definitely believe we’re making headway but it’s not an easy fight and woo practitioners are constantly adjusting their tactics (look at the reverse skeptic tactics that are becoming more common here for example). I’m not concerned with what CAM proponents want, they’re not the target audience. Our target audience is their target audience and those are the people we’re working to educate.

    We can’t stop them from cherry picking or performing their own studies. We can criticize their methods but what do you think will happen if say we were to stop our scientific inquiries on these things and they continued, building up mounds of studies (good quality or not) to ‘support’ their views?

  23. Kawarthajonon 22 Aug 2014 at 9:08 am

    # BillyJoe7: “Because of zero plausibility, there never was any justification for doing clinical trials on homoeopathy, and there certainly is no justification for doing any now. If all the negative trials to date haven’t convinced everyone, then nothing will.”

    I respectfully disagree. In an ideal world, all the research money would go to legitimate scientists who are researching higher plausibility questions. The reality is, however, that there is always money for bogus CAM research and we need properly funded, properly designed research to counter the nonsense claims and to expose the shoddy research. This is particularly true of life or death issues, such as vaccination and homeopathic “treatments” for severe or potentially fatal diseases. We’ll never convince everyone, but we need to steadily chip away at the nonsense claims.

  24. Steve Crosson 22 Aug 2014 at 9:15 am

    We are talking about the difference between things that “make sense” scientifically, and the things that “make sense” emotionally. Sadly, the later is almost an oxymoron, but emotions are still something that humans have to deal with.

    @Charon, you point out the contradiction yourself when you commented on how people without a physics background make statements like “we discover new things all the time”. That is exactly the “reasoning” that we must combat.

    I’m not advocating more (or even very many) RCTs for obvious woo, but we should at least be able to address the varieties of new woo as it crops up.

    OF COURSE, our knowledge of physics should be able to trump virtually all nonsense before it gains a foothold — but it doesn’t because most people allow emotions to rule instead of logic.

    Very few people understand even basic physics, let alone the more esoteric stuff, and it is unrealistic to expect that to change much. However, almost anyone should be able to comprehend the virtues of RCTs if properly explained, i.e. “We know that this works more often than that does because we actually counted the number of times!!!”

    Failure to perform at least a limited number of RCTs will simply allow the woomeisters to make the tired old “they’re not open to new discoveries” claim.

    The plural of anecdotes is certainly not data, but MORE anecdotes tend to outweigh LESS anecdotes. We need to do a much better job of controlling the conversation (and our legislators). We probably can’t improve science literacy in the foreseeable future, but we should at least try to help people make better decisions based on what knowledge they actually have.

    Most people want to make good decisions, but they tend to make choices based on what they can actually understand. I know that I sound elitist, but in my experience, this approach works much better in the real world.

    You can’t teach physics in an hour, but you can explain RCTs. Do it often enough and it starts to sink in.

  25. sonicon 22 Aug 2014 at 1:21 pm

    Researching the effects of things the population is doing would be a valid interest to a government of the people, for the people, and by the people. It really isn’t a matter of science as much as politics.
    And politicians often deal with emotion rather than rationality.

    If you are being attacked for being an atheist, you might want to point to this study to defuse the situation-

    http://www.science20.com/writer_on_the_edge/blog/scientists_discover_that_atheists_might_not_exist_and_thats_not_a_joke-139982

    To quote Lord Kelvin-
    “There is nothing new to be discovered in physics now, All that remains is more and more precise measurement.”
    I think he said that about 1890 or so…

  26. jsterritton 22 Aug 2014 at 1:23 pm

    @Steve Cross

    “I’m not advocating more (or even very many) RCTs for obvious woo, but we should at least be able to address the varieties of new woo as it crops up.”

    CAM proponents are making extraordinary claims, so it should fall to CAM proponents to provide extraordinary evidence. When scientists say “there is no plausibility for [insert wacky health claim],” it is a challenge to the CAM proponents to show evidence — put up or shut up. By conducting RCTs trying to debunk woo, scientists play right into the woomeisters’ hands by helping create a scientific literature on an unscientific subject. It is a near-certainty that a RCT will produce at least one spurious data point. It is that data point that will be used to sell CAM. This is how pseudoscience works: it co-opts the legitimacy and authority of science to trick people into buying pre- or non-scientific nonsense (sometimes very dangerous nonsense).

    I think that “real science doesn’t do pointless studies about magic” makes a more compelling statement than “real science caved into arguments from popularity, so here are some RCTs investigating magic — of course there’s no 100 percent certainty and one subject did report feeling like a frog for 5 minutes — make of them what you will.”

  27. jsterritton 22 Aug 2014 at 2:44 pm

    @sonic

    “Researching the effects of things the population is doing would be a valid interest to a government of the people, for the people, and by the people. It really isn’t a matter of science as much as politics.”

    In the case of RCT of CAM modalities, what exactly is being researched? We have very good data on what the population is doing (according to NCCAM/NIH 38% of Americans use CAM). Understanding the “why” questions is very important, but a different topic. As to “how” does CAM work, that’s easy: it doesn’t (not beyond placebo effect). If it did work, it wouldn’t be called CAM…just M.

    I would be very interested in seeing research of the effects of CAM in dollars spent and life hours lost. A quick check shows: 34 billion dollars out-of-pocket in the US. Across the world, pseudoscience and mysticism cost countless lives in preventable deaths (AIDS, starvation/nutrition, vaccine-preventable illness to name a few).

    Speaking of vaccine-preventable diseases, the anti-vaccinationists famously abuse clinical trial data to dismiss the overwhelming scientific consensus on vaccine safety. All while clamboring for the mythical “one study to lead them all” — i.e., an unethical, impossible RCT. The argument is always the same: if science cannot absolutely prove that vaccines are safe, then vaccines must be absolutely unsafe.

    This is how RCTs aid and abet CAM. Vaccine safety is not strictly analogous, because there are endless good studies testing valid hypotheses (not to mention there is a world of preclinical data from in vitro and animal studies and Phase I/II trials). But the magical thinking remains the same: cherry-pick satisfying little bits of data that suit your beliefs while ignoring the actual scientific conclusions.

  28. BillyJoe7on 22 Aug 2014 at 5:49 pm

    sonic,

    “If you are being attacked for being an atheist, you might want to point to this study to defuse the situation”

    You might do that for a laugh and as an object lesson in how headlines can completely misrepresent the content. I’m not going to say much because all I’ll get in return is a one line retort, so I’ll just leave you with the headline of the article and the statistic they quote in support and hope that you will at least try to read critically in the future:

    The headline: “Scientists discover that atheists might not exist, and that’s not a joke”

    The statistic: “In the United States, 38% of people who identified themselves as atheist or agnostic went on to claim to believe in a God or a Higher Power”

    I hope you can at least agree that 38% is a just little short of 100%.

    ————————

    “To quote Lord Kelvin-
    “There is nothing new to be discovered in physics now, All that remains is more and more precise measurement.”
    I think he said that about 1890 or so…”

    You might have noticed that it is now 2014.

  29. Steve Crosson 22 Aug 2014 at 6:21 pm

    @jsterritt

    Scientifically, I agree with you completely. But the fact remains that most people think emotionally, and simply ignoring a problem will not make it go away.

    No matter what we do, the CAM proponents will try to counter it, and they already have a well-established (and successful) playbook to counter claims of scientific implausibility. They will simply point to one or two “plucky underdogs” who fought against the evil scientific establishment until they were finally able to overcome conventional wisdom.

    You and I both know that the chance of any maverick scientist actually achieving that grows less with every single year and that almost every single new discovery merely fills in the gaps in existing knowledge. But look around you — popular culture paints a completely different picture.

    Movies, TV and books are literally filled with underdog stories, including scientific “breakthroughs”. Very few people stop to think that the entire reason for such disparate representation is that “ordinary life” is usually uninteresting precisely because it is NOT extraordinary.

    Consequently, too many people are under the mistaken impression that highly improbable events are actually quite common. Which makes it all too easy for people to be mislead by anecdotal success stories.

    In a perfect world, no one should be allowed to make any medical claim without it being scientifically proven, but even if we could replace ALL of the ignorant and sometimes corrupt politicians and then fix all of the stupid laws that exist, it still would not solve the problem. Desperate people do desperate things.

    Try this simple test: next time you try to convince someone that their particular flavor of woo is nonsense, start with the scientific implausibility gambit — either their eyes will glaze over and they will ignore you or they may counter with the “plucky underdog” defense, but in either case it is extremely rare that you will change their mind.

    But then, and this is the important part, commiserate with them and agree that yes, scientists don’t know everything and some people get well and scientists don’t know why. And then tell them about this interesting study you read where half the people got acupuncture (or homeopathy or whatever you happen to be talking about) and the other half got sugar pills, and guess what — the sugar pill people got well just as often as the flavor of woo you are discussing. At this stage, I usually try to counter the “what’s the harm/just in case” objections (if they arise) by equating the woo with wasting money — something most people dislike.

    Admittedly, this is hardly foolproof and it doesn’t work nearly as often as I wish it did, but it is, by far, the most successful method for countering woo that I’ve discovered in over 50 years of experiment.

    I fully agree that the CAM adherents will do their best to coopt and misrepresent everything they possibly can, but I feel that some data is better than no data, and that the good is greater than the harm.

  30. jsterritton 22 Aug 2014 at 8:08 pm

    Steve Cross…

    We’re clearly on the same page, but…

    I am not suggesting anyone ignore anything. Quite the contrary, as Gorski and Novella point out, this is a “less-is-more” situation. There is no dearth of evidence contradicting the spurious claims of CAM. We need look no further than a basic understanding of biology, chemistry, and physics. In CAM RCTs, there is essentially NO hypothesis to test (only “I think it works, let’s test people to see if it does”). In the absence of plausibility or any possible mechanism of action, there is no hope for positive results — the very notion of using RCT to do this kind of “Texas sharpshooter” testing is antithetical to science. And it doesn’t matter who does it, the “good” guys or the CAM guys. The results will usually be negative, but occasionally they will be false positive for no other reason than because testing was done. RCTs with low/no prior plausibility are just a good way to engineer noise and invite bias and trouble. That’s why science should insist on sticking with the scientific method: no leapfrogging to RCTs without doing basic science first. A Orac says:

    “Basic science alone can’t demonstrate efficacy. However, it can show that a proposed treatment is so implausible based on its purported mechanism of action as to be utterly not worth testing in RCTs.”

    I say: Amen.

  31. mumadaddon 22 Aug 2014 at 8:14 pm

    BJ7,

    Thanks. Seriously…

  32. Bronze Dogon 23 Aug 2014 at 11:53 am

    I recall a few conversations that made the distinction between “evidence-based medicine” and “science-based medicine.” I didn’t directly participate, but what I took away from the conversation was that prior plausibility is the crux of the matter.

    It’s easy to cherrypick false positives from a large body of RCTs and use them to claim a treatment is “evidence-based medicine.” If your first experiment turns out positive because of a ~5% statistical fluke, just mindlessly following that evidence without paying attention to prior plausibility or the broader picture would likely lead to a tangle of superstitious ad hocking to explain subsequent failures. Researchers are human, and even sincere ones might be reluctant to accept that their lauded “breakthrough” or exciting preliminary results happened because of dumb luck.

    It’s not so easy to build a rationale on top of the existing scientific consensus and produce repeatable positive results. We want to have a framework for understanding why X might work combined with an awareness of the big picture. If something has no known mechanism for it to work or relies on non-consensus hypotheses, it’ll need to meet higher standards of evidence to convince us it’s true. If such a thing does turn out to be true, it would mean we need to take a serious look at the conflicting consensus theories and figure out how to resolve the issue. The consensus theories are the consensus theories because they’ve survived countless challenges and proven their worth. If someone wants to overturn a hard won consensus, they need to show us that we aren’t going to be wasting our time chasing statistical illusions or sloppy protocols by humoring them.

    Regarding doing RCTs on magic: I would very much prefer if we had better government policies that avoid funding such studies with tax dollars. Even if we got that, they’re still going to happen and different people respond to different arguments, so we might as well discuss them if we think it’ll help get our points across. We should still emphasize prior plausibility, the importance of independent replication to counter popular notions of ‘silver bullet’ studies, the difference between preliminary and conclusive research, and the importance of research quality.

  33. Charonon 23 Aug 2014 at 12:52 pm

    Steve Cross:

    You can’t teach physics in an hour, but you can explain RCTs. Do it often enough and it starts to sink in.

    I call BS on that. You absolutely can explain the relevant physics in an hour. For homeopathy, all you need to explain is that things are made of atoms, what Avagadro’s number is, how dilution works, and that liquids don’t have memory. Done. Reiki, you need to explain how we found the forces that act on particles, and why there can’t be any new forces or energy fields undiscovered that could noticeably affect humans. You don’t need to integrate some quantum field theory Lagrangian density over spacetime to do this, any more than you need [insert advanced biochemistry thing there that only a biochem PhD would know] to talk about testing some medical treatment.

    There are pitfalls in all of this. Explain an RCT to someone, and as we well know, they might end up claiming the placebo effect is some wonderful new treatment!

    Plus, as people have pointed out, we’ve done RCTs of CAM. Numerous treatments have been shown not to work. This is not affecting public perception in the way you claim it would. We need a different way to combat CAM use, but that wasn’t the question addressed in the original article – the original article was about best use of research time and dollars.

  34. Charonon 23 Aug 2014 at 12:57 pm

    @Steve Cross, also it’s not clear to me you’re appreciating the argument about priors. The point is that RCTs are not an appropriate test of extremely implausible treatments. You will get a lot of junk for results, even with good RCTs.

  35. sonicon 23 Aug 2014 at 1:28 pm

    jsterritt-
    In the case of CAM modalities, what exactly is being researched?
    Answer- the effects of the treatment.

    You might research the area you discuss- how much time and money is being spent and what are the actual results and outcomes. You say you would be very interested in such a study. If the government is paying for it, then it involves politics and politics involves people’s interests.

    I’m not saying this is the best use of the money, I just pointing to a counter-argument that involves the way the government makes these kind of decisions.

    Private for profit companies have a strong interest in using dollars in the most effective manner to produce profits.
    Politicians have to keep their constituents happy. And that can involve all sorts of waste.

    By bringing in a different example (vaccines), you make a strong case to stop all research on any topic. Knowledge and information can be used for nefarious purpose.
    I’m not sure the fact that knowledge can be abused is a reason to stop acquiring it.
    That’s not the argument you were trying to make though- is it?

    BillyJoe7-
    I didn’t say the article was correct- I said you might want to use it to defuse a situation.

    You really did just cherry pick the one line from the article that makes you feel comfortable and ignored the actual content of the argument made, though, huh?
    ROFLMAO

  36. Steve Crosson 23 Aug 2014 at 2:36 pm

    @Charon,

    I should have been more clear. Of course you can teach physics in an hour, IF the student is open-minded and logical. But that is not generally applicable to believers in woo. From their perspective, quantum magic is just as likely as peer-reviewed physics.

    Too many people are uncomfortable or afraid of anything that they do not understand. Consequently, they wind up believing whatever is the most comforting.

    And yes, I DO realize that RCTs for BS are a bad idea — scientifically. All I can say is that in my personal experience, when people “see it with their own eyes” (even via a second or third hand account), they tend to be more convinced than they would be by relying on some “esoteric” theory that they don’t understand.

    Face it. The marketplace of ideas has been distorted/polluted by our weak educational system and the popular entertainment industry. Improbable things happen all the time and magic is real — or so too many people believe.

    We all know that the plural of anecdote is not data, but the majority of the world seems to base their opinions on anecdotes, so we may as well take advantage of that fact. As far as I can tell, the primary failing of the current system is one thing and one thing only — we don’t make as much noise as the other side.

    We need to shout from the rooftops. Let the world know that for every “study” that proves a particular flavor of woo works, there are ten or one hundred or a thousand that prove it doesn’t.

    Employ some psychology along with the science. As I said, there seems to be no more convincing paradigm that “I saw it with my own eyes”. And for some bizarre reason, anecdotal evidence seems to be almost as convincing to most people. Hell, every flavor of woo depends on “testimonials” — and it works.

    We just need to provide more and better quality.

  37. jsterritton 23 Aug 2014 at 2:44 pm

    Sonic…

    As @Charon notes above, if a hypothesis violates basic physics, explain the physics, don’t presume to do human testing of a fatally flawed hypothesis (example of a fatally flawed hypothesis: human trials of homeopathy despite Avogadro’s constant). RCTs are being used to test magic and wishes (or to debunk same), but shouldn’t be. Without a plausible hypothesis to test, “the only real net effect of CAM research is to offer a patina of legitimacy to nonsense in order to better exploit the public with snake oil.”*


    *Dr Novella, this post

  38. jsterritton 23 Aug 2014 at 2:49 pm

    @Steve Cross

    “We need to shout from the rooftops. Let the world know that for every “study” that proves a particular flavor of woo works, there are ten or one hundred or a thousand that prove it doesn’t.”

    Hallelujah! The end of your comment got me really worked up and I enjoyed reading it immensely. Science communication remains a HUGE priority.

  39. BillyJoe7on 23 Aug 2014 at 6:32 pm

    sonic,

    “I didn’t say the article was correct- I said you might want to use it to defuse a situation”

    How disengenuous can you get.

    “You really did just cherry pick the one line from the article that makes you feel comfortable and ignored the actual content of the argument made, though, huh?”

    And, as I said, you really must try to actually read what is written.
    For example, I said very clearly that “I’m not going to say much because all I’ll get in return is a one line retort, so I’ll just leave you with the headline of the article and the statistic they quote in support and hope that you will at least try to read critically in the future”.
    I was not a cherry pick, but an initial response that I do not intend to follow up unless something sensible is said in reply. I’ll not hold my breath.

    “ROFLMAO”

    Telling me.
    That article is full of hyperbole, misrepresentation, and equivocation (which is the reason I never use the word “spiritual” for what I feel about the astounding facts revealed by science, or emotional feelings about music, art or beauty – it gives idiots like this author the excuse to call you “religious”)

    Anyway, I’ve already spent too much time responding to this drivel.

  40. BillyJoe7on 23 Aug 2014 at 6:39 pm

    sonic,

    “I’m not sure the fact that knowledge can be abused is a reason to stop acquiring it”

    See, you’ve missed the whole point.
    There is no knowledge to be obtained in the first place by researching what has no plausibility to start with.

  41. jsterritton 24 Aug 2014 at 3:45 pm

    I’m going to make a bold, probably overreaching statement: experimental research testing vanishingly low/no plausibility hypotheses — regardless of how well designed, well intentioned, and concrete the conclusions — is a kind of junk science. Why? Because the results will always be negative or false positive. This begs a “what’s the harm” inquiry, as in: what’s the harm in conducting good studies “proving” CAM modalities are no more effective than placebo? The harm is twofold: it throws good money at bad ideas and lends a false legitimacy to CAM modalities while they are being studied. Simply put: we don’t need RCTs demonstrating that long-term antibiotic treatment won’t have a therapeutic effect in the absence of a bacterial infection (that’s not to say we shouldn’t investigate whether antibiotics may have other effects, but that’s a different story). We don’t need RCTs to disprove rumors about water having memories or that some chemicals will harm us at low doses but not high ones. Too many studies on the evidence-based side of science are being undertaken in response to popularity or to debunk CAM myths. This is not the direction in which science should go. Science is supposed to follow the evidence. Studies with low/no plausibility are working the wrong way round.

    None of this is to say that proponents of CAM shouldn’t be conducting research. Quite the opposite: those claiming the health benefits of a treatment modality should absolutely have to provide evidence in support of those claims. However, RCTs, which are considered the gold standard for demonstrating efficacy among a human cohort, are not where this research should begin — or end. Rather, it behooves anyone making a claim for the effectiveness of a treatment modality to provide compelling evidence in support of that claim along lines of plausibility and basic scientific understanding. RCTs — or any human testing — should only occur in the presence of good prior probability, good science, and plausibility of the hypothesis being tested. Without lab work, animal models, and well-designed earlier-phase human trials, leapfrogging to RCTs is a way of cheating at science. Claiming efficacy before conducting thorough research, or based on only preliminary research, is putting the cart before the horse. It is also a stalling tactic and a misuse of the scientific process, which presumes neither “guilt” nor “innocence,” but merely demands good evidence. CAM proponents would have us believe that the absence of evidence against their claims somehow demonstrates safety and efficacy. This is nothing more than shifting the burden of proof, and it is troubling how often evidence-based (i.e., real) science will stoop to pick it up. Dr Novella addresses this directly:

    “It may take 10-20 years of pre-clinical and clinical research to properly sort out if a treatment hypothesis is valid. This problem is exacerbated and more wasteful the less plausible the original hypothesis. … Combine with this the tendency of preliminary studies to be false positives, and a completely bogus therapy may enjoy 20 years of false legitimacy before definitive trials show that it is completely useless.”

    In the meantime, magical thinking and predatory practitioners are costing consumers billions of dollars out-of-pocket each year and who knows how many life hours lost to substituting bogus modalities for real medicine. As Steve Cross points out (above), evidence-based science is failing to communicate — and resonate — with consumers. This may be a failure of oversight agencies and regulatory bodies or it may be a failure of consumer protection. Scientists aren’t really in the business of worrying about such things, so maybe the fault lies there. Maybe science can never do more than yell into the wind of human nature.

    It’s a real melon-scratcher.

    One place to begin fixing the problem, though, is in restoring the scientific method to science practices concerning CAM modalities. There is surely a place for evidence-based science to hold bad CAM practices to standards of legitimacy and proof (and probably a role for some RCTs of some CAM modalities). At least evidence-based science, as opposed to CAM, has its head in the right place, its priorities straight, and our best interests foremost in mind. If the sCAMmers can’t win on the level playing field of basic science and proper scientific protocol, if they can’t provide preclinical evidence in support of their claims, then they have lost the battle and lost the right to advance to human trials. I am almost certainly being naive, but Drs Gorski and Novella have inspired me to see that something is broken, and something must be done.

  42. BillyJoe7on 24 Aug 2014 at 5:47 pm

    jsterritt,

    You are absolutely correct.
    The present trend in accommodating CAM has lent it an air of legitimacy and resulted in the steady and progressive infiltration of CAM into medical practices, hospitals, medical education, and academia. This approach has clearly failed. It is definitely time for confrontation. The authors of this article deserve the widest support.

  43. grabulaon 24 Aug 2014 at 9:13 pm

    jsterritt said: ” Science communication remains a HUGE priority.”

    Absolutely. If it were done properly then I think it would go a long way towards making CAM even more fringe.

    I think the ultimate problem we’re battling is lazyness. In my own experience when someone begins talking about using CAM or some such woo, they heard it worked for a friend or relative so why not. They haven’t bothered, and most people won’t, to do the research to actually check to see if it works.

  44. Bruceon 25 Aug 2014 at 6:09 am

    This is a fascinating original post and the discussion afterwards has been brilliant, it has had me vacillating quite a bit.

    jsterritt summed it up with:

    “It’s a real melon-scratcher. ”

    A big part of me agrees fully with Charon; if someone comes to me and says that wearing their Anti-Grav Bracelet (TM) will make me defy the laws of gravity, I will understandably tell that person where they can put their bracelet (I can assure you it is up somewhere but will need a little more than anti-gravity to get in there). This was initially going to be my point in response to Steve Cross, in that I don’t need an RCT on an Anti-Grav Bracelet (TM) because it breaks the laws of physics quite blatantly.

    However…

    The more I read the more I realised I would probably, just for sh*ts and giggles, ask for a demonstration… and this is where it gets fuzzy because the person wanting to sell me this bracelet will most likely have anticipated this. Their single demonstration (read: anecdote, badly controlled study, sleight of hand or a combination of the three) will very much be biased towards their bracelet working and as I suspect the demonstration shows that the bracelet appears to work (in some fashion). So, I am still 99.99% sure that their bracelet does not work as described and something hinkey is going on here. What do I do? I either study up on the laws of gravity and the mechanism for how they claim it works (quantum pushing my dingledodies away from my shazbut therefore negating the force of gravity or some such) and I come to the conclusion that they are indeed talking bollocks and that their bracelet is just clever marketing and smoke and mirrors.

    Clever me! I know it doesn’t work and the physics tells me it doesn’t work… but they are selling like hotcakes and before you know it people are jumping off buildings plumetting to their deaths thinking they can fly. One, two, three lives lost, the manufacturer says that those deaths were simply because the bracelets were not being used properly… look at their demonstration, the numbers clearly show it works! Just don’t use it to actually defy the laws of gravity!

    So… I decide to spend some time and money and buy some of these bracelets and do some actual studies on them before more people die. They all come back negative, they are just very stylish cheap rubber bands with a bit of Lapis Lazuli embedded in it. I publish the study… the makers come back and say that this is only one study and that it was conducted by Big Cobblers who are obviously going to say it doesn’t work because we want to keep selling you shoes and with this bracelet your shoes will wear out much more slowly.

    Where do I go from here? Do I spend more money on studies? Do I spend time chasing down the stories of those who get harmed from the bracelet? Do I spend time writing a blog about it? I am only one person (or even a group of people), I have to make a living, I have to feed a family… how much do I sarcifice to stop all these people dying from this?

    It sure is a headscratcher… and I think it really depends on the skeptic and the issue at hand. We are limited in our resources and I think there is always going to be a balancing act between gathering evidence and informing. A RCT is of no use if no one tells people about it, and even then it is of no use if no one understands the impact of it. Even the laws of physics don’t mean a lot to someone who doesn’t even want to know them. The key is not endless studies, but it is not no studies either… the trick is finding that sweet spot of a body of evidence that can then be used to communicate the reality instead of relying solely on the evidence provided by those trying to sell the bracelets. And that, for me is where the problem lies… as long as the woo has its own “evidence”, we are always going to have to counter it with real evidence, and whatever form that evidence takes (laws of physics, a few RCTs) it needs to be communicated effectively.

  45. sonicon 26 Aug 2014 at 10:56 am

    jsterritt-
    You think it is silly to run certain experiments- a waste, even harmful.
    I have no disagreement with that.
    I’m thinking this isn’t really an issue for the for profit companies– they put the money in the places that give highest expected return.
    We are talking about government expenditures here.

    My point is that if ‘the people’ are using a treatment x, then the government would have an interest in seeing scientific experiments regarding that treatment.
    It’s true that gathering this information might do more harm than good and be a waste of money, but the elected officials have other concerns that they will consider legitimate.

    As I understand it, the two biggest concerns are getting money from donors and getting votes. I’m assuming this is not a controversial statement. Perhaps that is where I have over assumed. Is that a controversial statement?

    Either way– the calculation based on money and votes isn’t always going to be the same as the one based on the factors you might think should be rationally considered. But when dealing with government it is often useful to determine the money and votes aspect of the situation when attempting to figure out what they should (or will) do.
    My guess is there is more money and votes in saying ‘we will study what our people are doing using the finest scientific methods and the gold standard of experimental research’ than it would be to say ‘why waste the money on a bunch of self-deluded monkeys?’

    That’s what I’m thinking anyway.

    BillyJoe7-
    Sorry.

  46. grabulaon 26 Aug 2014 at 9:10 pm

    For once I think sonic makes a good point:

    “My point is that if ‘the people’ are using a treatment x, then the government would have an interest in seeing scientific experiments regarding that treatment.”

    I think this is an important point – if the government has to step in to regulate something, for example, the claims that homeopaths can make, you’ll need science to back up the reasons why they are being regulated. We’re seeing this battle right now with CAM and homeopathy here in the US.

  47. BillyJoe7on 27 Aug 2014 at 7:50 am

    sonic,

    BJ: “I’m not going to say much because all I’ll get in return is a one line retort”
    sonic: “Sorry”

    You’ve exceeded expectations. (:

  48. jsterritton 27 Aug 2014 at 10:23 pm

    Sonic…

    “We are talking about government expenditures here.”

    Maybe you are, but I’m not. I think you are trying to change the topic and make this discussion about the government. Scientific research and “the government” are not the same thing. Government funding policy and how grants are budgeted and awarded is its own discussion — and that discussion does not factor in here.

    “My point is that if ‘the people’ are using a treatment x, then the government would have an interest in seeing scientific experiments regarding that treatment.”

    Again, I don’t see why “the government” would have any interest that humanity wouldn’t have all on its own. We all have an interest in knowing the science about treatment modalities. My point — and the point of the article — is that scientific experiments and knowledge are two very different things. Experiments testing crappy hypotheses yield crappy results. In the case of RCTs of CAM modalities, those crappy results come in the form of noise, noise, more noise, and spurious data points (also noise). For no/low plausibility hypotheses, there are better kinds of experiments that should be conducted — starting with thought experiments and fact-checking against known laws of science, then moving onto more resource-intensive methods like in vitro and animal testing. RCTs are late-stage, large-scale human tests performed to assess safety and efficacy after preclinical and earlier-phase human trials establish whether a modality works at all. RTCs of CAM modalities skip ahead to large-cohort “Texas sharpshooter” studies (wherein a large number of people are “tested” and any positive-looking result is circled and called a bullseye) without preclinical evidence of efficacy or even any reality-based proposed mechanism of action. Late-phase human testing is supposed to test therapeutic doses (established in earlier studies by “dose ranging”) — something that cannot even be imagined in the case of modalities like acupuncture, let alone homeopathy.

    I do not see what glaring errors and missteps in science practices have to do with the government. Too many of these modalities are unregulated or under-regulated thanks to off-label quackery and a regulatory system defanged by popularly and PR (e.g., the Dietary Supplement Health and Education Act of 1994). If anyone wants to know about the efficacy of CAM modalities, use the right tools for the job. And that includes the government.

  49. JJ Borgmanon 28 Aug 2014 at 7:15 am

    jsterritt…one of the finest response posts I’ve ever read…thank-you for that.

  50. sonicon 29 Aug 2014 at 12:21 am

    jsterritt-
    Saying what public money should be spent on is a political statement.

    There is a difference between a scientific argument and a public policy argument.
    Perhaps I’m confounding the two, but it seems the argument made in the post involves expenditures- yes I believe it has to do with ‘The question is…” in the fourth paragraph.

    Perhaps I should have said, “I am talking about government expenditures.”
    Then I would say the basis for those expenditures includes campaign contributions and votes and those are considered rational considerations by the people who are deciding what to spend the money on.

    So one could argue that there isn’t a rational scientific reason to run a certain test, but that’s not the only consideration taken into account when one is talking about public money expenditures.

    If you are saying that scientists shouldn’t run experiments that they have some interest in and are getting paid to run, well that’s another matter, isn’t it?

    Once you start to discuss how money should be allocated, you enter the world of politics which has its own rationale for why things are done.

    So if the discussion is about the most effective way to communicate science, that’s one thing- but if it is about how to allocate funds a different logic needs to be applied or at least acknowledged.

    I will step away from the soapbox now.

    BillyJoe7-
    I really did mean the article could be useful to defuse a situation.
    You really don’t want me to defend the musings of a cognitive scientist about how people formulate their deeply held beliefs, do you?

    Don’t you ever make fun of yourself to defuse a situation?

  51. BillyJoe7on 29 Aug 2014 at 1:05 am

    sonic,

    “I really did mean the article could be useful to defuse a situation”
    Clearly you were wrong.

    “You really don’t want me to defend the musings of a cognitive scientist about how people formulate their deeply held beliefs, do you?”
    Not sure what you mean.

    “Don’t you ever make fun of yourself to defuse a situation?”
    There’s never any need, and it’s much more fun calling out the BS.

  52. dudeon 29 Aug 2014 at 1:14 am

    Sonic,

    Maybe I am alone on this but I would be thrilled if politics was informed by science. If a test is a waste of money because it could have no meaningful results then I would hope that information would be more important to informing the government on what course to take when the populous wants to run experiments rather than doing what the populous demands.

    A waste of money is a waste of money. If the public demands to know if homeopathy works then the solution is science education not a study that is agree to be a waste of money.

    What different logic would you employ that would say that something that can produce no meaningful results is a worthwhile use of time/money? Politics should be discussion on points that we can not know, however when a position can be shown to be wrong through logic and reason it has no place in a political debate. You can have opinion on anything you want but if you opinion is contrary to facts then I am sorry but your opinion in this instance should be ignored and everyone should move on.

  53. jsterritton 29 Aug 2014 at 4:07 pm

    @Sonic

    “jsterritt-
    Saying what public money should be spent on is a political statement.”

    I’m not saying what public money should be spent on. In fact, I’m not talking about money at all. Mine is a “right tool for the job” argument and nothing more. If you must reduce that argument to money, then I’m telling everybody how not to spend money. That includes entirely privately-funded research, industry research, and the much-romanticised “pure research” of bygone days. Will everyone save money if they stop skipping basic science in a mad rush to conduct pointless and expensive RCTs? Yes. So in a way I’m like Robin Hood or a communist or one of those late-night TV people who tells you how to live like a king by always buying in bulk, never going on vacation, and never, ever wasting money.

    But let me be perfectly clear: I don’t care about money. I care about science practices as applied to CAM modalities. I’d rather see money burned in the commission of good studies than money saved by doing bad ones.

  54. sonicon 30 Aug 2014 at 12:39 pm

    dude-
    The goal of the politician is to get elected.
    He will consider things that forward that goal to be desirable or good and will consider things that hinder that goal to be undesirable or bad.

    The goal of politics in general is to aid in a smoothly functioning society.

    These are not the same goals as science in general, so I would expect the politicain to take into account what a scientist might say about a situation, but I would expect that the politician would weigh other factors.

    For example- If money is going to his district, or his voting public wants something done, the politician will do what gets him elected as a matter of course- this also keeps the society functioning smoothly- people getting paid to do things they have an interest in and so forth.

    Logic depends on premises– the politicians premises are not the same as the scientists, so the conclusions need not match.

    Here’s the UN approach to the situation to give you some idea of the differences–
    http://apps.who.int/iris/bitstream/10665/92455/1/9789241506090_eng.pdf?ua=1

    jsterritt-
    I’m not sure how not caring about money leads one to conclude it shouldn’t be wasted– or how money could be wasted in a situation where one doesn’t care about it.
    Anyway–
    Do you think the RCT’s being done now are being done by researchers who think the thing they are studying has no chance of working?

  55. jsterritton 30 Aug 2014 at 4:06 pm

    Sonic…

    “Do you think the RCT’s being done now are being done by researchers who think the thing they are studying has no chance of working?”

    What’s your point? Why are you asking me? What RCTs? What researchers?

  56. grabulaon 30 Aug 2014 at 9:25 pm

    I was listening to the latest SGU on the way to work today and Dr. Novella brings this subject up. Bob sort of hit upon a point that sort of goes a ways towards explaining where I stand on this issue. I can’t remember his exact quote but the point was about regulation based on the fact that no matter what the science shows on CAM, CAM practitioners ignore it or blatantly spin it to work for them.

    Drug companies can’t do this. Any serious medicine is held to higher standards. Bayer can’t claim that their aspirin cures cancer as an example without science backing up those claims. Part of the current issue with CAM is they’re basically running amock. There are no standards they’re being held to like real medicine.

    I realize the argument could be said that this might legitimize CAM in some people’s eyes but the ultimate point would be that if science doesn’t support the claims then there should be some standard somewhere that requires they 1 – attempt to support their claims with science and 2 – if their claim fails the scientific process, they cannot continue to make that claim.

  57. sonicon 03 Sep 2014 at 7:18 pm

    jsterritt-
    You were saying you thought that RCT’s were being done inappropriately.
    I’m asking for specifics.

    grabula-
    Is the problem with the regulations or the way they are being applied?
    http://business.ftc.gov/advertising-and-marketing/health-claims

    And it looks like the FDA disagrees with you about the drug companies selling things without evidence of positive effect–
    http://finance.yahoo.com/news/fda-little-evidence-support-testosterone-163231024.html

Trackback URI | Comments RSS

Leave a Reply

You must be logged in to post a comment.