I did my catch-up reading (the old fashion physical reading) over this past Memorial Day. One feature article in Time Magazine, on recommendation engines, jumped out at me. Here’s the online version.
Time reporter Lev Grossman correctly points out the laughable issues with the first generation recommendation engines and personalization using collaborative filtering. Just because people bought the iPad and Coke together doesn’t mean the two products are related.
The article hits home on what statisticians and cognitive scientists have known for years:
- There are small lies, damn lies, and then statistics
- Humans are only predictable within a narrow context or topic
Within any large data set, such as internet purchases and behaviors, you can always find two things being connected even when they are just random. There is a fine line between serendipity and randomness. To separate them, you have to know if there is any hidden affinity between the two. The latest cognitive discoveries show that people are animals of context. Relationships among products or content are not determined by who we are as individuals, but rather are biologically "forced" upon us by the contexts in which the products or content are being considered. Across unrelated topics or contexts, people are very unpredictable even you know them well – I've known my wife for years but everything I buy for her seems to be wrong.
However, there is a reliable way to predict people – one that is based on the collective wisdom of like-minded peers. These are not your normal Facebook friends. They are your contextual peers. If you are shopping for a red washer, all of the people in the world who are looking for similar colorful washers should be your implicit contextual "friends" at this moment. Their collective interests under the context of "red washer" can help you short-circuit the content overload on the internet.
Interestingly, the narrower the contexts, the more predictable people become. This brain pattern recognition ability happens to offer us the perfect tool to discover the truly hard-to-find long tail products and content!
Powered by Compendium Blogware
Comments for Your Grandma’s Recommendation Engine: If You Liked “The Sound of Music”, We Recommend “Rocky IV”