The Streetlight Effect in Data-Driven Exploration

We consider settings such as innovation-oriented R&D and entrepreneurship where agents must explore across different projects with varying but uncertain payoffs. How does providing partial data on project payoffs affect individual performance and social welfare? While data can typically reduce uncertainty and improve welfare, we present a simple theoretical framework where data provision can decrease group and individual payoffs. We predict that when data shines a light on sufficiently attractive (but not optimal) projects, it can crowd-out exploration activity, lowering individual and group payoffs as compared to when no data is provided. We test our theory in an online lab experiment and show that data provision on the true value of one project can hurt individual payoffs by 11% and reduce the group’s likelihood of discovering the optimal outcome by 52%. Our results provide a theoretical and empirical examination of the streetlight effect, outlining the conditions under which data leads agents to look under the lamppost rather than engage in socially beneficial exploration.

---