AUG 04, 2022
Stop creating self-fulfilling prophecies: How to apply AI to small data problems
DEC 27, 2021
ver the past decade or so, the digital revolution has given us a surplus of data. This is exciting for a number of reasons, but mostly in terms of how AI will be able to further revolutionize the enterprise.
However, in the world of B2B — the industry I’m deeply involved in — we are still experiencing a shortage of data, largely because the number of transactions is vastly lower compared to B2C. So, in order for AI to deliver on its promise of revolutionizing the enterprise, it must be able to solve these small data problems as well. Thankfully, it can.
The problem is that many data scientists turn to bad practices, creating self-fulfilling prophecies, which reduces the effectiveness of AI in small data scenarios — and ultimately hinders AI’s influence in advancing the enterprise.
The term “self-fulfilling prophecy” is used in psychology, investing and elsewhere, but in the world of data science, it can simply be described as “predicting the obvious.” We see this when companies find a model that predicts what already works for them, sometimes even “by design,” and apply it to different scenarios.
For instance, a retail company determines that people who filled their cart online are more likely to purchase than people who didn’t, so they heavily market to that group. They are predicting the obvious!
Instead, they should apply models that help optimize what does not work well — converting first-time buyers who don’t already have items in their cart. By solving for the latter — or predicting the non-obvious — this retail company will be much more likely to impact sales and acquire new customers instead of just keeping the same ones.
To avoid the trap of creating self-fulfilling prophecies, here’s the process you should follow for applying AI to small data problems:
Natural Language Processing