My Involvement in Redweb’s Spirit of Christmas 2009

Over the last few weeks I've been beavering away with Redweb on a top secret project. My role involves some pretty cool Arduino and PHP work that all integrates nicely with Twitter. If I told you that a week ago I'd have probably been shot, but luckily for my health the radio-silence has been lifted and I can talk about the project to my heart's content.

Introducing the Spirit of Christmas

When Redweb approached me to do the development and electronics for a hush-hush project I was admittedly a little excited. Who doesn't like secrets? The project in question is a festive one that would run across the Christmas period. I was intrigued. After some begging and pleading for more details I was finally let in to the inner circle and all was divulged.

They wanted to measure Christmas spirit on Twitter across the festive period and display this data in the physical world via Arduino. A system was needed to do the heavy lifting in regards to Twitter and perform all the necessary calculations to work out the spirit level. With me so far? Then let me expand a little.

Process diagram for the Spirit of Christmas system

The system in question is to regularly scrape Twitter for all tweets including the word Christmas. From here the content of each tweet is searched for both good and bad words, referenced from a predefined list. Each of these words are weighted to indicate the severity of emotion, ranging from -1 for a particularly evil word to 1 for a positively beautiful word. By performing some mind-boggling calculations the tweet is classed as good or bad and an overall spirit level is spat out. This is repeated for each tweet until no more remain. From here the fun begins. Every 15 minutes an Arduino kit calls the system and is given an angle corresponding to the spirit level of that period. The angle is then used to rotate a servo which is connected to a pointer on a display in the office. The pointer gives a physical indication of the current level of Christmas spirit on Twitter. A webcam then transmits a feed of the display to the website, resulting in an overwhelming sense of joy being experienced by the viewer.

This is all well and good, I hear you ask, but what is the point? Well, the point is that Redweb are crazy like that and thought it was be a cool idea to measure how people feel about Christmas. They chose Twitter because of it's dominance in the micro-blogging field and the plain and simple fact that there is a shit-tonne (a technical term) of data that we can access. I'm with Redweb on this one, I think the idea is amazing and I absolutely love it.

Together, we launched The Spirit of Christmas 2009 a few days ago and it's due to run 24/7 until Christmas day. I challenge anyone to stand in-front of the finished website and not admit it's awesome.

I'll give you a few moments to come down off the high I'm sure you're experiencing. Finished? Great. Now, let me explain how I created the system.

Your tweets or your life

Fortunately for Twitter it's relatively straight forward to access all the tweets I needed. With a little poking of the Search API and some help from the Services_Twitter PEAR extension I was positively rolling in Christmas tweets. Finding the tweets I needed is the easy part, the real fun begins when I need to start manipulating that data.

Before I start playing around with the tweets I cache them all in a temporary database table. This cache is cleared every night and simply allows me to keep a record of the tweets for that day. I'll explain later how the cache is used by the Arduino.

Measuring the emotion of a tweet

So I have all the tweets I could ask for. Fantastic, but not incredibly useful, at least in their current state. So how do I go about persuading them into giving up their Christmas spirit? With some calculations beautiful for their simplicity of course.

There isn't much use trying to work out the sprit if there isn't anything to base the spirit on. Unfortunately my research concluded that there isn't some magical Christmas entity, a sobering realisation which I'll be having words with my parents about. Instead a selection of predefined words will have to suffice.

All the words are stored in 2 arrays, one for good words and one for bad words. Each individual word has a weighting associated with it that indicates it's emotional severity, ranging from -1 for extremely bad words to 1 for amazing words. For example, using the word "hate" when describing Christmas is more severe than using the word "dislike". The same goes with good words. For example, "love" is much more positive than "like". From this weighting table I can calculate an average weighting for the tweet in question resulting in a final value between -1 and 1, the Christmas spirit.

Sketch of the Christmas spirit calculations

You'd be excused for thinking this all sounds relatively simple but trust me, it took a lot of thinking and testing to refine the process. Originally the spirit was either -1 (negative), 0 (neutral), or 1 (positive), based purely on if there were more bad words, the same amount of, or neither, good or bad words, or more positive words respectively. This method proved to be incredibly restrictive. Often tweets were being classed as negative, even if they had really positive words in, simply because there were more bad words, regardless of their severity.

For posterity I store the total spirit level and other interesting data about the tweets in another database table. Perhaps I will use this data in the future for something interesting…

Displaying it all in the real world

The spirit-o-meter in action

Getting the Arduino kit to react to the spirit level involves a whole bunch of code which I won't go into here as I'll be covering it in detail in another entry. Needless to say that the kit is connected to the internet via an ethernet shield which then checks a PHP script every 15 minutes to be given an angle to rotate the servo to. As a pointer is attached to the servo this action results in a measurement being displayed in the real world. Pretty sweet.

Merry Christmas one and all

I hope this slightly lengthy insight into the development of the Spirit of Christmas 2009 has proven useful. The whole process has been a challenging yet thoroughly enjoyable one and has taught me much about emotion tracking and Arduino.

Why not jump on Twitter and give your opinion on Christmas.