Science Offline: What is medicine's 5 sigma?

This is an editorial by Richard Horton, who is the editor of Lancet, a prestigious medical journal. He writes about scientific research in general, though it is mostly his observations of medical science practices.

http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(15)60696-1/fulltext?rss=yes

“A lot of what is published is incorrect.” I'm not allowed to say who made this remark because we were asked to observe Chatham House rules. We were also asked not to take photographs of slides. Those who worked for government agencies pleaded that their comments especially remain unquoted, since the forthcoming UK election meant they were living in “purdah”—a chilling state where severe restrictions on freedom of speech are placed on anyone on the government's payroll. Why the paranoid concern for secrecy and non-attribution? Because this symposium—on the reproducibility and reliability of biomedical research, held at the Wellcome Trust in London last week—touched on one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations.

The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”. The Academy of Medical Sciences, Medical Research Council, and Biotechnology and Biological Sciences Research Council have now put their reputational weight behind an investigation into these questionable research practices. The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data. Journal editors deserve their fair share of criticism too. We aid and abet the worst behaviours. Our acquiescence to the impact factor fuels an unhealthy competition to win a place in a select few journals. Our love of “significance” pollutes the literature with many a statistical fairy-tale. We reject important confirmations. Journals are not the only miscreants. Universities are in a perpetual struggle for money and talent, endpoints that foster reductive metrics, such as high-impact publication. National assessment procedures, such as the Research Excellence Framework, incentivise bad practices. And individual scientists, including their most senior leaders, do little to alter a research culture that occasionally veers close to misconduct.

Can bad scientific practices be fixed? Part of the problem is that no-one is incentivised to be right. Instead, scientists are incentivised to be productive and innovative. Would a Hippocratic Oath for science help? Certainly don't add more layers of research red-tape. Instead of changing incentives, perhaps one could remove incentives altogether. Or insist on replicability statements in grant applications and research papers. Or emphasise collaboration, not competition. Or insist on preregistration of protocols. Or reward better pre and post publication peer review. Or improve research training and mentorship. Or implement the recommendations from our Series on increasing research value, published last year. One of the most convincing proposals came from outside the biomedical community. Tony Weidberg is a Professor of Particle Physics at Oxford. Following several high-profile errors, the particle physics community now invests great effort into intensive checking and re-checking of data prior to publication. By filtering results through independent working groups, physicists are encouraged to criticise. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. Weidberg worried we set the bar for results in biomedicine far too low. In particle physics, significance is set at 5 sigma—a p value of 3 × 10–7 or 1 in 3·5 million (if the result is not true, this is the probability that the data would have been as extreme as they are). The conclusion of the symposium was that something must be done. Indeed, all seemed to agree that it was within our power to do that something. But as to precisely what to do or how to do it, there were no firm answers. Those who have the power to act seem to think somebody else should act first. And every positive action (eg, funding well-powered replications) has a counterargument (science will become less creative). The good news is that science is beginning to take some of its worst failings very seriously. The bad news is that nobody is ready to take the first step to clean up the system.

 

Jason Fung is a widely respected kidney specialist in Toronto. He specializes in obesity and diabetes. He has dozens of lectures on YouTube, if you care about the subject matter.

A government researcher got huge funding and contrived a study to disprove the relatively new theory that it is carbs in our diet causing higher insulin levels which causes people to gain weight. As a doctor who treats diabetes, Fung (and every other diabetes doctor) knows the relationship between insulin and weight control. Type 1 diabetics have too little insulin and no matter what they eat, they get horrifically thin. When the doctor prescribes insulin or drugs that stimulates insulin production, the patients get fat.

The existing dogma is that if you eat more calories than you expend, you gain weight. And that dietary fat leads to heart disease and stroke. We have a worldwide obesity epidemic that has grown exponentially since our foods were altered to be low fat and high carb. With all the "eat less, exercise more" being preached, the epidemic has only worsened.

The insulin-carb theory fits the real world observations.

So Fung takes the government researcher to task.

@Further

https://intensivedietarymanagement.com/heres-5-kevin-hall-go-buy-clue/

Kevin Hall, the senior NIH researcher recently published a paper in AJCN that has received alot of media attention. This study, he claims, completely refutes the insulin hypothesis so completely that it is now ‘dead’. That’s interesting, I thought, as I sat down to read the article.

It was a little surprising, therefore to read this paper and realize that Hall’s conclusions were entirely his own opinion. He suffers so badly from confirmation bias that he may as well have written “My mind is already made up regarding the insulin hypothesis. Please do not confuse me with facts”. Confirmation bias is a well-known psychological phenomenon whereby facts that agree with your pre-formed opinion are accepted as true and those that are not, ignored. All facts become filtered through this bias to confirm your previously held opinion. It’s also known as a closed mind.

This happens far more frequently than is often realized or admitted. A similar situation exists in the research behind the weight losing effect of breakfast. Researchers used this question of the proposed effect of breakfast on obesity (PEBO) to look at how researchers consistently interpret results depending upon their beliefs. If you believed breakfast made you lose weight, then studies were interpreted to support this view even if it did not.

Two specific tactics are used – Research lacking Probative Value (RLPV)(association data that can’t prove anything) and biased research reporting (BRR). As more and more people believe something to be true, the effect becomes worse since all research is now interpreted to fit the preconceived facts.

Starting with the preconceived notion of ‘A’, all research is interpreted to support this belief (B), and all negative studies are ignored (C). This only reinforces the belief (D) which then leads to biased research reporting. This, of course, is a vicious cycle. The same effect is obvious in the Calories In, Calories Out (CICO) diehards.

 

Global warming.

 

The science isn't settled. There clearly has been warming for the past 10,000 years.

However, that's not the point of this thread.

It's about how organized science operates and continually reinforces incorrect information and how society acting on that information is harmed greatly.

 

The age of scientific discovery has been tarnished by greed and the seeking of power...the results of one scientist to another vary depending on who is paying for the research

 

I agree, @Further, that how you describe it is how it should work. And used to.

Big time dietary scientist:

It was a little surprising, therefore to read this paper and realize that Hall’s conclusions were entirely his own opinion. He suffers so badly from confirmation bias that he may as well have written “My mind is already made up regarding the insulin hypothesis. Please do not confuse me with facts”. Confirmation bias is a well-known psychological phenomenon whereby facts that agree with your pre-formed opinion are accepted as true and those that are not, ignored. All facts become filtered through this bias to confirm your previously held opinion. It’s also known as a closed mind.