Jump to ContentJump to Main Navigation
The Phantom Pattern Problem – The Mirage of Big Data - Oxford Scholarship Online
Users without a subscription are not able to see the full content.

The Phantom Pattern Problem: The Mirage of Big Data

Gary Smith and Jay Cordes


Pattern recognition prowess served our ancestors well. However, today we are confronted by a deluge of data that are far more abstract, complicated, and difficult to interpret than were annual seasons and the sounds of predators. The number of possible patterns that can be identified relative to the number that are genuinely useful has grown exponentially—which means that the chances that a discovered pattern is useful is rapidly approaching zero. Coincidental streaks, clusters, and correlations are the norm—not the exception. Our challenge is to overcome our inherited inclination to think tha ... More

Keywords: pattern recognition, big data, self-selection bias, randomized controlled trial, backtesting

Bibliographic Information

Print publication date: 2020 Print ISBN-13: 9780198864165
Published to Oxford Scholarship Online: October 2020 DOI:10.1093/oso/9780198864165.001.0001


Affiliations are at time of print publication.

Gary Smith, author
Pomona College, Fletcher Jones Professor of Economics

Jay Cordes, author
Data Scientist