Why ineffective medicines appear to work: online medical reviews suggest widespread over-reporting of positive outcomes

Abstract

Background. People often hold unduly positive expectations about the outcomes of medical treatment. One explanation for this phenomena is that those who have a positive outcome tend to tell more people about their disease/treatment than people with poor or average outcomes. Akin to the file drawer problem in science, this systematically distorts the information available to others and results in the development of inaccurate beliefs and expectations. Method. If people with good outcomes are more inclined to tell other people, then they should also be more inclined to write online medical product reviews. Therefore average outcomes in these reviews should be more positive than those found in RCTs. I analysed 1,675 Amazon reviews of cholesterol reduction (Benecol, CholestOff) and weight loss (Orlistat) treatments. Data on duration of treatment and outcome (i.e., weight / cholesterol change) was extracted and compared to RCT data for the same treatments using ANOVA. Findings. In three independent tests, average outcomes reported in reviews were substantially more positive that outcomes reported in the medical literature (eta squared: .01 to .06; p: .04 to .001). For example, average cholesterol change following use of Benecol is -14mg/dl in RCTs and -45mg/dl in online reviews. Discussion. People with good outcomes are more inclined to share information and this distorts the information available to others. Explanations for erroneous medical beliefs usually focus on flawed reasoning; these results suggest that patterns of communication may be equally important.