Intuitive Appeal and False Positives

Image source: paper boats in puddles

Yesterday, I was lucky enough to sit in on Leif Nelson's journal club at UC Berkeley's Haas School of Business. The article for the week was a forthcoming article in the Journal of Experimental Psychology: General by Andrew Meyer and Shane Frederick (with a bunch of others). In the paper, the authors thoroughly and concisely demonstrate that the effect of disfluency (as manipulated through font) on analytical reasoning is a false positive. After pooling the results of sixteen studies, the authors "find no evidence of a disfluent font benefit under any conditions." To review, the original finding suggests that reducing the clarity of font for a question or task can lead to more analytical or deliberate thought. In turn, this improves performance on counter-intuitive math problems (from the Cognitive Reflection Task or CRT). What I find especially interesting about the determination that this finding is nothing more than a false positive, is that the original finding has so much intuitive appeal and was so easy to manipulate (and potentially apply). For this reason, anyone, psychologist or not, would probably agree with the postulation that making something more difficult to read should make people think more carefully about the problem at hand. This poses an interesting problem, as Leif noted in his discussion: will the original paper and findings still act as a placeholder for an idea that makes intuitive sense and which people think should work? More generally, how do people respond to a false positive finding when the original finding is intuitively appealing versus counter-intuitive?

The Meyer, Frederick et al. paper also brings up an important lesson. Ultimately, the authors found that the original disfluency finding was the result of just one of the three CRT questions (the widget problem, for those who are interested), and that it was a movement in the control group rather than the treatment group. This highlights the need to confirm that an effect is happening where it should be happening (in response to the treatment rather than movement in the control) (this point was made nicely by Uri Simonsohn in the journal club meeting). Hopefully, researchers will move away from the use of font as a fluency manipulation in the future and this idea, no matter how intuitively appealing, will be appropriately discounted.