How an Anomaly Focus Eliminates Bias
The current widespread focus on the problems of replication in scientific studies—especially those of the social and biomedical sciences—has been to invest in ever more sophisticated analytical tools and techniques: more data, more sophisticated tools of analysis, faster computing, etc. All of these efforts have missed an alternative solution: causality. Most research in biomedical and social science begins with data analysis. But data analysis alone is not capable of yielding a genuine causal model. Essentially, all analysis is subject to the problem of induction—roughly, correlation is not causation.
Our theory-building model is founded on building a causal model and testing that model in a variety of situations. The most obvious risk for such a process is that having built a model, people tend to unwittingly find corroborating evidence as they apply their model—confirmation bias. However, in our model, a theory is not tested in other environments to confirm it, but to find anomalies for the theory. Anomalies help to refine the causal model both by bounding the model and by working out the conditions of its application. Gravity looks different under different conditions—on the moon vs. the earth vs. open space—but is still the same mechanism. Sometimes we can explain phenomena without needing to refer to gravity—such as explaining superconduction. It’s not that the law of graviton is suspended; it’s just that the law has nothing to add to our understanding of lots and lots of phenomena.
While an anomaly focus is crucial for improving a causal theory, correlational modes have trouble with “outliers.” Most often outliers are dropped from consideration. But whether to do so relies on extra-statistical thinking. A causal theory must account for any apparent anomaly—resulting in clarification, rejection or bounding of the original theory. Confirmation bias simply cannot enter into such reasoning, since corroborating evidence is interesting, but does not lead to better theory.
Put simply, an anomalies-first focus in theory building either forces biases out or prevents biases from entering into a researcher’s reasoning.
How an Anomaly Focus Eliminates Bias is built on the principles we teach in our live, online Product Science Bootcamp.
Stuck on a business problem? Don’t have time to attend our bootcamp?
Bring it with you to our Unstuck Assist session and Get Unstuck.