Too much paperwork,” wrote Bob Dylan. The world of research is also often wrong, according to science and business journalist David H. Freedman. In Wrong: Why Experts Keep Failing Us—And How to Know When Not to Trust Them, Freedman describes how and why researchers reach flawed conclusions. Consequently Wrong could spells trouble who people rely on research to develop public policies.
Is there a way to spot bad research? Yes, according Freedman, and he lays out some rules for doing so. Be wary of research offering experts’ advice if it:
1. is simplistic, universal, and definitive;
2. rests on a single study or many small or less careful ones;
3. claims to be groundbreaking;
4. is pushed by people or groups that stand if the public accepts the findings; and
5. offers to prevent, for all time, a failure or crisis from happening again.
Be cautious of research:
1. that sounds right or fits too well with your personal world view,
2. upsets the apple cart (i.e., eating fatty foods is good for you),
3. that grabs a lot of good attention,
4. that is widely embraced by other experts,
5. that shows up in a prestigious journal,
6. that supported by only one big, rigorous study, and
7. where experts backing the study have impressive credentials.
Lastly, pay attention to studies that:
1. do not set off other alarms described above,
2. do not confirm interesting or useful hypotheses,
3. qualify its findings and conclusions,
4. incorporate conflicting evidence,
5. provide context for the research,
6. explain how the findings affect the reader, and
7. include the researchers’ doubts and skepticism.
