It’s true that funnel plots only go so far, but they’re fairly unproblematic for most part in practice, especially since you can generally take a plot that ‘fits’ as indicative of non-biasing. Since study population heterogeneity and design heterogeneity are primary competing reasons for off-kilter plots, it would also make sense in this case that those could be ruled out, since they were carried out by the same investigator.

It’s hard to blame design heterogeneity and population heterogeneity if the same person at the same institution is taking subjects from the same source population.

david: I suspect that was Dr. N’s intent. I wouldn’t know how you would get a numeric value for “study quality” after all.

“You can make a graph of all published studies with the quality of the study on the vertical axis and the result (positive or negative) along the horizontal axis.”

Why would this form a funnel shape? I can’t see any reason why there should be a relationship between variance in effect size and study quality. Don’t you mean ‘study size’ on the vertical axis?

I’m not sure how to address this problem. Indeed — we do need to have some metric of productivity for researchers, and publishing results is one of them. However — if a researcher spends 100K and 12 months of work testing a hypothesis that turns out to be wrong — should he or she not get future funding or a promotion? The temptation is to keep beating the data to a pulp until some publication can be eked out of questionable statistical-wrangling while playing down the 12 months of what would be perceived as failure.

I have actually witnessed older, established, scientists (with boatloads of funding) beating a dead horse of disproven hypotheses, flawed animal models, ambiguous or negative results, — publishing questionable data; because their reputations depend on being right all the time and “productive.” Younger scientists need to attach their names to the older in order to seem like they are part of a “productive” team, in order to promote their careers and to secure funding.

In my opinion, the problem of publication bias has 3 parts: 1. The “publish or perish” culture in academic research, 2. The capitalistic means of funding science in the US, and 3. The hierarchical culture of academia (new investigators, new ideas don’t get fair play). These problems are interconnected and each affects the other.

http://www.cochrane-net.org/openlearning/html/mod15-3.htm
Publication Bias Interpreting funnel plots
*From these examples, we can see that a funnel plot is not a very reliable method of investigating publication bias, although it does give us some idea of whether our study results are scattered symmetrically around a central, more precise effect. Funnel plot asymmetry may be due to publication bias, but it may also result from clinical heterogeneity between studies (for example different control event rates) or methodological heterogeneity between studies (for example failure to conceal allocation).*

http://en.wikipedia.org/wiki/Likelihood_function
*Attempting to interpret the likelihood of a hypothesis given observed evidence as the probability of the hypothesis is a common error, with potentially disastrous real-world consequences in medicine, engineering or jurisprudence. See prosecutor’s fallacy for an example of this.*

http://lesswrong.com/lw/1ib/parapsychology_the_control_group_for_science/
*Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored – that they are unfairly being held to higher standards than everyone else. I’m willing to believe that. It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter.
— Eliezer Yudkowsky, Frequentist Statistics are Frequently Subjective*

Further, from my dictionary: “Bias is a predisposition either for or against something; one can have a bias against police officers or a bias for French food and wines”
So in research and publication of results, bias should be determined relative to the researcher’s purposes. If you have a new proposition in your field and want to demonstrate that your ideas have value, are you biased? And if so, is such bias necessarily inappropriate? Hardly.

For example in the thread on ECT, I just realized that their entire conclusion is wrong, but because they put their data there it can be understood that it is wrong.

We spend a lot of time finding errors in research, and the same sorts of errors come up time and again. This effectively makes the research useless or, at least, less than useful. Shouldn’t we be coming up with ways to prevent this tragic waste of time and effort?

This could be done by educating researchers and limiting research funds to those who pass muster in knowledge about how to conduct a methodologically sound and unbiased clinical trial. The file drawer effect can be prevented by registering trials and disallowing publication of unregistered trials.

Now that the factors behind faulty trial design are well known, isn’t it time to take preventative action?

Educate yourself first, at least in the basic premise of intelligent discourse, and then we can continue.

Though your use of the term “evolutionist” belies the fact that that will almost certainly never happen.

Toodles!