Edge of Innovation

Project Oxford: from facial recognition to emotion recognition

Marek Hoffmann Autor, Hemd & Hoodie


Whenever a new facial recognition technology comes onto the market, or a technology that has the potential to scan and recognise faces, it almost feels like the outcry is a well-rehearsed performance.

Remember Google Glass? It provoked an especially dramatic reaction — see: 72 percent say no to Google Glass because of privacy. Ultimately, the camera mounted into Google’s eyewear was both its greatest feature and its biggest flaw, sparking concerns about secret surveillance, a violation of privacy or the misuse of data for criminal purposes.

But that hasn’t stopped the development of facial recognition elsewhere. A number of tech giants are working on it, ranging from Apple and Facebook to Google and Intel (specifically its RealSense 3D camera tech and True Key password management software).

Microsoft has ambitions in this area too, but they move beyond facial recognition and into emotion recognition. Redmond calls it: Project Oxford.

A learning machine that knows when you’re angry

Simply put, Microsoft Project Oxford is a set of tools that make it possible for a computer to identify emotions in photos using facial recognition technology.

The software is able to distinguish between anger, contempt, disgust, fear, happiness, sadness and surprise, as well as a “neutral” emotional state. Each image scanned is assigned a score between zero and one for each of these eight possibilities, where zero corresponds to a complete absence of the emotion in question and one is a solid emotional response.

You can see how it works in the two photos below.


“The exciting thing has been how much interest there is and how diverse the response is,” says Ryan Galgon, Senior Program Manager for the Oxford Project.

Diverse is right. Microsoft is at pains to point out that its emotion recognition technology “is experimental, and not always accurate” — the tool seems to produce a different result for the same photograph if it is uploaded in two different sizes.

Nevertheless, the emotion recognition system is an intelligent learning machine that is designed to improve — the more data it receives, the better the results will be.

With this in mind, the system can be trained using sample images to recognise specific properties, traits and facial characteristics. This knowledge can then be applied to new images and used to improve the system ability to recognise these features through constant adaptation.

Microsoft has already demonstrated how effective the “learn by doing” principle is with its real-time translator for Skype or the Windows 10 personal digital assistant, Cortana.

Capturing emotions in the retail sector 

The question now is what useful purpose can emotion recognition serve? After all, you could argue that we humans can interpret emotions in a photograph much faster and (for now) much more accurately than a computer can.

Microsoft’s Ryan Galgon sees the retail sector as a particularly important potential target group. He believes that emotion recognition could be used in retail environments to gauge the responses of consumers viewing products or commercials in a store window.


Galgon also sees potential for using the tool with messaging services, which could recommend different options based on the facial expressions they detect in photos being exchanged.

To take this line of thought one step further, the tool might be useful for revealing the mood of the person you’re talking to by scanning that person’s face using a smartphone camera. The app would, of course, need permission to use the camera first!

how old net2

This isn’t the first time that Microsoft has dabbled with facial recognition technology. You might remember its HowOldRobot AI, which aimed to guesstimate a person’s age based on a photo. You can still try it for yourself on the website how-old.net (although the results are often wildly and laughably inaccurate).

Similarly mixed results are also to be expected from this emotion recognition technology.

But why not give it a whirl and make up your own mind? Microsoft’s Project Oxford emotion recognition system is available to use for FREE. Simply upload an image (at least 36×36 pixels in size, smaller than 4 MB, in JPEG, PNG, GIF or BMP format) and then wait for the tool to identify the number of faces in it and allot each an emotional score.

You’ll see a blue frame around each face detected and emotion scores (rounded to five decimal places) can be viewed when you move your mouse pointer over a face. As always, we invite you to post your “results” in the comments below…

Cover photo and image 3: Microsoft Blog
Image 1 and 2: Microsoft — Project Oxford
Image 4: How-old.net

Share This Article

Related Topics

Tech Innovation

Read This Next

Read Full Story