After Netflix

Well - a (combined) team has finally managed to get to the finishing line - many,many congratulations to them. I must admit I feel a mix of regret not to be slightly further up the leaderboard and relief that I can now (bar a few desperate throws of the dice) concentrate on taking the learnings from Netflix elsewhere.

The competition has been very good to me, and I'm now engaged on a variety of projects trying to leverage the skills learnt including:

• Producing a film and television recommendation system http://marketingfeeds.nl/TechCrunch/2009/06/03/beeTV_Raises_$8_Million_For_Stunning_Personal_TV_Recommendation_System
  
• Working for a number of dating agencies http://www.onlinepersonalswatch.com/news/2009/04/gavin-potter-and-nick-tsinonis-founders-of-intro-analytics.html trying to help them identify compatible people - the interesting twist here is that as well as the person having to like the movie the movie has to like the person as well - if you see what I mean)
  
• Identifying who might have to go to the accident and emergency department of a hospital so that careplans can be put in place to reduce the likelihood of an emergency admission thereby reducing costs and improving patient satisfaction. (the movie equivalent here is the treatments they received in the last year).
• Working on a project to predict the prices of ... (I'm afraid I can't talk about this one just yet).
  

The interesting thing is, that in all these cases, the application of the Netflix algorithms makes a substantial improvement over the status quo - I think the learnings from the Netflix competition have enormous applications both within recommendation systems and elsewhere. Hats off to Netflix for producing such a valuable advance in both the science (and probably more importantly) the number of people who can now tackle these kinds of problem.

If any of the Netflix contestants are interested in working on "real problems" please don't hesitate to get in touch. I've more work than I can handle at the moment.

How Netflix predicts the price of wine

I, and I know, a number of others are beginning to be sidetracked into other things that we might do with the knowledge that we have garned from the Netflix prize whilst we let our betters (go for it Pragmatic Theory) battle to get that last little of rmse that will land them the $1million prize.

I'll publish a number of ideas that I've been involved in over the last year. One that has surprised me is the ability to predict the price of a wine from comments collected from the web. Its early days yet, but a project that I've been involved in is looking to see whether we can predict the price of clarets (ranging from $3,000 a case to $300 a case) based solely on wine reviews.

Slightly surprisingly this is working very well. The picture above shows the fit (in £(UK)) of the price of around 100 wines to their actual values. In Netflix terms the rmse of the prices is around £370 a case (once the mean price is subtracted), once you include the contributions from the words the rmse falls to around £140 a case, so slightly over half the variance can be accounted for.

What is also interesting is some of the key words that indicate a high price. These words are in order of importance with the words at the bottom of the list being negative indicators of price.

woody
pencil
hard
fat
complex
spicy
tannin
cherry
smoky
fragrant
green
elegant
soft
balanced
tobacco
fruit
oak
blackberries
fruity
lingering
flabby
expressive
aromatic
Jammy
smooth
thin
rounded

So woody and pencil are the words to look for when choosing expensive wines from Bordeaux. Try it when you next purchase a wine, its already changed what I look for on a wine description.

Why do it. Well its a little bit of a labour of love, to see if we can produce a system that can identify underpriced wines to buy. However, the success so far has suggested that if we can find a more liquid market (no pun intended), then there might be the potential to make some money by identifying underpriced opportunities, and we are currently exploring a few other ideas that are looking promising.

The psychological meaning of billions of parameters

The leaders in the Neflix competition have made great strides since my last post.

Essentially my understanding is that they have done this by modelling thousands of factors on a daily basis. i.e for each person they model (say 2000) factors on an individual and individual day basis. The set of ratings provided for the competition gives enough information so that you can work out that a particular person had a particular preference of a particular strength on a particular day to watch something funny (or given that there are 2000 factors or so) something rather more obscure (maybe watch something in sepia or something). The ratings also enable you to calculate how much a film meets those requirements (again on a particular day - what seemed funny at one time period may not seem funny at another).

By combining the two sets of factors you can then work out how a person will rate a particular movie and improve your score in the competition. This is an undoubtedly impressive feat from a statistical / machine learning viewpoint.

It strikes me that this is also interesting from a psychological viewpoint - do we really believe that people have such nuanced preferences across such a large number of dimensions. I have an open mind about this - apriori I would have thought people would use many fewer factors in arriving at a rating decision - certainly 2000 factors (or even 20) can't all be combined consciously - the subconscious must be heavily involved. Maybe, on the other hand, there are only a few factors that we take into account - but they are different per person and the only way in which they can be explained is by taking a mix of the 2000 or so factors that are modelled.

It strikes me that depending on your view on the above your choice of research direction on the Netflix competition, recommendation systems and indeed psychological processes in general will vary.

I'd welcome views.

Signalling and sequences

One of my daughter's friends suggested that sequels would, on average, recieve lower scores than the original movies - as, at least in her experience, they were invariably worse. I thought I'd just confirm her suspicions so that I could let her know that she was thinking about the problem in a good way.

However, to my surprise the opposite appears to be true. Here is the mean score - the 0.5879992 number (adjusted for various things) for each episode of Sex in the City.


Sex and the City: Season 1 0.5879992 41138
Sex and the City: Season 2 0.5824835 43795
Sex and the City: Season 3 0.6523933 38983
Sex and the City: Season 4 0.7066851 34616
Sex and the City: Season 5 0.7359862 33380
Sex and the City: Season 6: Part 1 0.8097552 33532
Sex and the City: Season 6: Part 2 0.8241694 27914

As you can see the later the sequel the better the result. This seems, at least to me, counter intuitive - However the answer may lie in the second number which is the number of people who rated the movie. It seems that once there have been (in this case two episodes), then people who don't like the movie drop out and don't watch any further. So although less people watch it, they give a higher average rating.

This might be interpreted as some form of signalling. If a movie can accurately 'signal' to its potential audience that it is worth watching then the average rating will be higher. Interestingly this is - potentially - in conflict with the aim of the movie companies who might want to maximise the number of people watching irrespective of what they think of the movie (at least in the short term).

Wired Article

Thank you all who have sent suggestions as to how to improve my score on the Netflix prize as a result of reading about my attempts in the article in the latest edition of Wired. I'm very grateful and will incorporate any that I can figure out a way of converting into a computer program.

Just to say a little bit more about me. - I'm fascinated by the use of computers to understand how the mind works and how we can then use this knowledge to help predict human behaviour. The Netflix prize provides probably the largest dataset collected on human decision making that has been made publicly available. My attempts at the prize are based around a desire to understand how we can use such a dataset to better understand human decision making (and, of course, the outside chance of winning $1million).

My progress to date suggests that there is at least something in this approach and I'm open to offers to work on other datasets that incorporate a human decision making element - I currently have time available. As, an example, have a look at this company that I have helped setup recently, www.customerfusion.co.uk which aims to harness some of the learning from the Netflix competition to fuse market research data with customer information. Better still, if you have any market research data that you want to extract more value from - drop us an email.

Simple Heuristics that make us smart

Just back from holiday. Managed to finish Gigerenzer and Todd's "Simple Heuristics That make us smart", an interesting idea in a long book. Basiclly they list a number of simple ways in which people make decisions and demonstrate that the simple methods can be as accurate, or in some cases, more accurate than sophisticated statistical techniques.

Now if one could work out which people use when rating videos...

As an aside - In the UK a woman called Sally Clark was convicted of killing her children based on a completely false understanding of probabilities and none of the Judge, defending lawyers, prosecution lawyers or expert witnesses picked up on the gross misunderstandings that occured. She is now dead - undoubtly as a result of the miscarriage of justice. - A 'must read' is Gigerenzer's "Reckoning with Risk". Gigerenzer provides an extremely clear introduction to the mistakes that people make (primarily doctors and lawyers) and suggests ways of presenting evidence to make sure errors don't occur. If I'm ever in front of a doctor or a judge I'm going to make sure I assess my own probabilities or give them a copy of his book.

The Korbell papers

Decided to try and implement one of the Korbell algorithms. After much angst managed to get their IncFctr algorithm working (although not producing quite such good results). It doesn't seem to lead to much better results than the Funkian gradient approach but it sure is a lot faster. It must be at least 10* faster than the gradient approach. The Korbell team are to be congratulated.

I wish I'd tried implementing it earlier - it would haved saved considerable time.