Руденко Віталій Олексійович.

Харківський національний університет радіоелектроніки, Україна

Constructing services for human emotion recognition and their analysis for different clients

Nowadays services with various means of communication are very popular among vast social groups, these services are trying as much as possible to facilitate the exchange of information between people. That is why the ability to analyze emotions will fit into the modern realities as an awesome way to make your communication more interesting, more entertaining, more informative and, even, more educational.

When people communicate, important part is played by emotional background, which can be determined by analyzing facial expressions. Emotions can serve as a source of additional information, for example: indicate untruthfulness said, or unpleasant emotion like sadness would be recognize and probably immediately fix by your friends in conversation. Thus we are increasing quality of communication and decreasing possibility of some important emotions to remain unnoticed.

The object of study work is recognition algorithms and analysis of emotions and their possible implementation in software code.

The aim of this work is to create a software service to identify and, where possible, to analyze human emotions by facial expressions.

Recognition of emotions extremely difficult process, they are quite different from one another for different races and even nations. When we are trying to analyze facial expression we can encounter situations when expression is the same but its’ meaning is different or more frequent situation when face express smile for most of Europeans top lip is curved up, but for African nations we commonly see that lip is still in same state as it was without smile. Such differences should be ignored in analyzer, so that it could make conclusions without unstable information, or each of them should be resolved for each troublesome moment. We should also understand that there are common face types, common behavior mode and common reactions for all humans, so it is actually reasonable to use this common elements to create analyzer. Moreover, there are many methods and algorithms of machine learning, which will be able to create their own criteria for analysis later, when service will be running, to deal with unique moments such as scars or unusual faces. Taking into account previous facts it is necessary to use more complex two-way communication channel between client and service (preferably web-sockets), also we shouldn’t forget that the whole burden of counting data is on the server so it will be heavily loaded.

ASP.NET SignalR is a new library for ASP.NET developers that makes it incredibly simple to add real-time web functionality to your applications. What is "real-time web" functionality? It's the ability to have your server-side code push content to the connected clients as it happens, in real-time. SignalR provides a simple API for creating server-to-client remote procedure calls (RPC) that call JavaScript functions in client browsers (and other client platforms) from server-side .NET code. SignalR also includes API for connection management (for instance, connect and disconnect events), and grouping connections. SignalR uses the new WebSocket transport where available, and falls back to older transports where necessary. While you could certainly write your application using WebSocket directly, using SignalR means that a lot of the extra functionality you would need to implement will already have been done for you. Most importantly, this means that you can code your application to take advantage of WebSocket without having to worry about creating a separate code path for older clients. SignalR also shields you from having to worry about updates to WebSocket, since SignalR will continue to be updated to support changes in the underlying transport, providing your application a consistent interface across versions of WebSocket. While you could certainly create a solution using WebSocket alone, SignalR provides all of the functionality you would need to write yourself, such as fallback to other transports and revising your application for updates to WebSocket implementations.

OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products. Being a BSD-licensed product, OpenCV makes it easy for businesses to utilize and modify the code. The library has more than 2500 optimized algorithms, which includes a comprehensive set of both classic and state-of-the-art computer vision and machine learning algorithms. These algorithms can be used to detect and recognize faces, identify objects, classify human actions in videos, track camera movements, track moving objects, extract 3D models of objects, produce 3D point clouds from stereo cameras, stitch images together to produce a high resolution image of an entire scene, find similar images from an image database, remove red eyes from images taken using flash, follow eye movements, recognize scenery and establish markers to overlay it with augmented reality, etc. OpenCV has more than 47 thousand people of user community and estimated number of downloads exceeding 7 million. The library is used extensively in companies, research groups and by governmental bodies.

Emgu CV is a cross platform .Net wrapper to the OpenCV image processing library. Allowing OpenCV functions to be called from .NET compatible languages such as C#, VB, VC++, IronPython etc. The wrapper can be compiled in Mono and run on Windows, Linux, Mac OS X, iPhone, iPad and Android devices.

On the client side we need to support the opportunity to work with the resources of system (camera). So for web this meant complex logic in Flash, or given up the possibility of web clients. The latter is not an option especially because of high speed of development of the modern Web, fortunately in Html 5 version has support for working with system resources and some browsers currently support this possibilities. Moreover Html 5 allow us to use new transports: WebSocket (if the both the server and browser indicate they can support Websocket) WebSocket is the only transport that establishes a true persistent, two-way connection between client and server. However, WebSocket also has the most stringent requirements; it is fully supported only in the latest versions of Microsoft Internet Explorer, Google Chrome, and Mozilla Firefox, and only has a partial implementation in other browsers such as Opera and Safari; Server Sent Events, also known as EventSource (if the browser supports Server Sent Events, which is basically all browsers except Internet Explorer.)

Image processing – the main task of the server. Using EmguCV greatly simplify common operations and speeds up your system. In addition library allows us to use a lot of operations for transforming the image that may help in the statistics collecting to improve service in the future. There are several ways how emotions could be noticed and analyzed. The most common is creating a facial mask by certain points and analyzing their relative position. Another way is to apply special filters to image until we will have main curves of face and analyzing that curves mathematically will give us an answer about human emotions. Both these methods have pros and cons. Main problem of first method is that we need to save several photos of user to create proper facial mask. This method works perfectly on video streams but can be difficult to implement via web. Second have hard math beneath itself thus become completely non-flexible and non-extendible because for each change in work of our server we will have to change algorithms of recognizing and analyzing emotions completely.

It is proposed to implement a test system in the form of web client - a chat with support for HTML 5, which implement access methods for SignalR server, where frames from a webcam are processed using client library EmguCV and sent to other users with analyzed emotional state.

As a result of the work will be obtained service for the detection and analysis of the human emotion. Studies will be done on the analysis of facial images for emotion recognition capabilities. For recognition will be used library EmguCV (Open coputer vision library for C #). Emotions are recognized by fixing certain points on the face, followed by an analysis of their location. The application architecture is designed to operate as a web service using library SignalR or web sockets itself for access via Html 5 web client.