April 14, 2008

Video Store Clerk Game: A Crowd Wisdom Experiment

On-line movie recommendation systems (such as those at Amazon, Netflix, etc) are pretty good at guessing what movies you might like based on your movie history. Improvements to these systems are constantly being made, using ever more sophisticated algorithms. But how good are they compared to the wisdom of actual people? That’s what my friends Jay and Andy are trying to figure out. And they need your help.

Jay and Andy have created a game called Video Store Clerk in which you play a video store clerk. You are told how a real customer has rated previous movie rentals, and then you are shown another movie title that the person also rented. Can you guess how the customer rated that movie?

They are collecting all the user-generated data and comparing it to the real customers’ ratings. A computer has already played the game with millions of customers, and we know how well it did. The question is whether or not the wisdom of crowds can beat the computer. To gather enough data for an accurate comparison, they need a lot of people to play. So please, pass the link around. Digg it. Blog it. They tell me their server can handle the load.

The experiment’s findings will ultimately go toward building a better movie recommendation system. Hopefully you’ll find the game fun to play, too. And if you have any ideas about improving the game, you can leave a comment here or use the contact link on their site.

Link: Video Store Clerk


Hoohah! High score! Only among today’s high scores, and only number 16, but…Hoohah!

It’s like this game fufils my teenage dream of getting to work in a video rental store - very addicting!

So, I just got the all-time high score, and my last round I just entered all 1’s to end the thing. Apparently I’m good at this. I recommend paying close attention to their total rankings; I think I could make it through a round without looking at anything but that, although I wouldn’t have done as well.

I normally agree with you whole-heartedly. However, in this instance, I have to strongly disagree with one of your statements - that the recommedation systems are pretty good at guessing what movies you might like. In fact, I negatively blogged on this very topic (

Despite this, I think improving the service is a good idea and I am happy to give it a shot.

I like it. Ill give the game a spin.

I would assume that you have heard about the million dollar netflix prize? http://www.netflixprize.com/

It’s hard to tell if those little tiny stars on the computer screen are yellow or white, especially when glancing down the list (persistence of vision). Bigger, or better colors please!

I’m VERY curious about the results. Cool experiment.

Why do I love rating things so?

Netflix is amazing at predicting what I will like. It looks like I am just as good at predicting what others will like. This makes me hope that there is someone at Netflix, suggesting movies just for me. A Netflix gnome.

For the first couple of rounds I was sticking to what I knew (sci-fi, comedy, animation) and doing pretty well, for the most part attaining my 7 correct answers with customers to spare. I then went out on a limb and picked some genres I knew little or nothing about (musicals and documentaries were particularly tricky given that I’d not even *heard* of 90% of them) but still managed to pass without too much difficulty (only requiring all 10 customers during one round), and in the end simply gave up on the game out of boredom as it seemed there was no end in sight. (Whoever racked up 16000+ points obviously has way too much free time on their hands!)

It seems that just knowing the overall rankings for a particular customer is, in general, good enough to know how they’d rank an arbitrary film, and domain knowledge is much less important (and in some cases even proved detrimental); the assumption of “if they liked X then they’d probably like Y” depends, I guess, on specifically *why* they liked X, and you’re much safer with “they seemed to rate more stuff a 4 than anything else, so they’d probably give this a 4 too”.

More useful, I think, would be providing a slightly longer list of previously rated titles (as to be honest, three instances provides almost no insight at all) and then a selection of other titles they’d also rated and sorting them into order of preference. Not only would it provide deeper and more interesting data, but the game itself would be a bit more challenging and more likely to hold interest for participants…

Are your friends competing in the Netflix challenge?

I wonder if it would work better to have a more Pandora-like system that keeps track of individual elements you like. So a person who has thumbs-upped a lot of movies that have quick, witty banter and aliens will get other movies with those same qualities.

kinda reminds me of carnegie mellon’s library management game.


The Video Store Clerk Game was mentioned today in TechDirt:


man diz game iz borin