IT202100007499A1 - HYBRID TELEMATICS NETWORK ARCHITECTURE DEDICATED TO PROCESSES OF EXTRACTION AND EVALUATION OF OPINIONS THROUGH EXPRESSION ANALYSIS AND RELATED ELECTRONICS AND SOFTWARE THAT ARE PART OF IT - Google Patents

Description

Description of the invention entitled:

?HYBRID TELEMATICS NETWORK ARCHITECTURE DEDICATED TO PROCESSES OF EXTRACTION AND EVALUATION OF OPINIONS THROUGH EXPRESSION ANALYSIS AND RELATED ELECTRONICS AND SOFTWARE THAT ARE PART OF IT?

Description

Sector to which the invention pertains

The invention refers in general to the sector of Internet applications, in particular it refers to applications in the context of Expression Analysis, i.e. the extraction and analysis of the opinions expressed by people, regarding products or services, considering in Does it also examine the emotions detectable by the expressiveness? of the face.

Known technique

Internet yes? developed in an academic environment of mutual trust, where everyone could read, edit or add information. IoT (internet of things) is taking off in a faster environment savvy where the expectations of public opinion and governments in terms of security and privacy are very high, and information security? one of the major concerns of businesses. So you need to work in a secure environment from the start, protecting your personal information, business secrets and critical infrastructure as you work. IoT brings a new set of problems, such as device security, security and robustness of computer systems. New types of attacks, as well as new privacy and cybersecurity regulations, can destabilize many industries. However, the economic benefits of IoT as an enabler for analytics, automation, process and resource optimization cannot be underestimated. To succeed with the transformation that the IoT is driving, industries need to gather expertise and understand new threats and how to mitigate them.

For the IT department, the IoT creates the need manage a large number of different types of devices, many of which may not be able to ask a user for login credentials or perform the traditional security check. For hackers, the amount? and diversity? of these devices increase the potential attack surface, including among the potentially sensitive data, some completely new types of detections for this scenario.

The application of Expression Analysis ? a significant leap forward from the now standard study of user/customer behavior; also integrating emotions into the survey on the basis of expressiveness? of the face, ? clear that detecting data cos? sensitive with standard and commonly used tools represents a risk with incalculable consequences.

We usually understand Expression Analysis as the extraction and analysis of the opinions expressed by subjects regarding products or services, both to measure the perception that users have of a certain market brand, and for the reaction detected in non- necessarily linked to a situation of purchase of the good itself.

Artificial intelligence involves all those operations characteristic of the human intellect and performed by computers. These include planning, understanding language, recognizing objects and sounds, learning and problem solving. The brain, based on sensory inputs, makes decisions, sending signals to the body to command the relative movements.

The IoT is not nothing but a set of connected sensors, and thanks to artificial intelligence you have the possibility? to make sense of all the acquired data and through the control system, or the heart of the circuit, the CPU, it is possible to make decisions by cataloging the data detected by the sensor in a quantitative and qualitative way, where sensor must be understood in the sense more wide as possible.

machine learning ? essentially an avenue for implementing artificial intelligence; a kind of subgroup of the AI that focuses on the ability? of the machines to receive a series of data and to learn by themselves, modifying the algorithms as they receive more? data about what they are processing. Very often, the terms AI and machine learning (ML) are used interchangeably.

The term ?machine learning? ? been coined after all?AI, understood as ?the capacity? of a machine to learn without being explicitly programmed?. Machine learning ? therefore a way to ?educate? an algorithm so that it can learn from various environmental situations. Education, or even better training, involves the use of huge amounts of water? of data and an efficient algorithm in order to adapt (and improve) according to the situations that occur.

Machine learning automates the construction of the analytical model. Use neural network methods, statistical modeling, and operations research to find hidden insights in data. A neural network? inspired by the functioning of the human brain. ? a computing system made up of units? interconnected (such as neurons) that process information by responding to external inputs, thus transmitting the related information between different units.

A classic example of machine learning? represented by artificial vision systems, or the ability? of a computational system to recognize objects acquired digitally by image sensors. The algorithm used in these cases will have to? recognize certain objects, distinguishing them between animals, things and people, and at the same time learning from situations, or have memory of what? what yes? made to use it effectively in the next acquisitions of artificial vision used above all in Automotive systems.

Does deep learning use neural network models with various units? processing; leverages computational advances and training techniques to learn complex models across a huge amount of of data. Common applications include image and speech recognition. The concept of deep/deep learning is sometimes referred to simply as a ?deep neural network?, referring to the many layers involved.

Figure 1 shows the monthly percentage trend of stack overflow searches over the years, having reported:

? outlined an analysis procedure for machine learning;

? full stroke an analysis procedure for deep learning;

? and I draw - point an analysis procedure of the artificial intelligence type.

In recent years, both machine learning and deep learning have made huge strides for artificial intelligence. Both require huge quantities? of data collected by a myriad of sensors that continue to fill the ecosystem of the Internet of Things, thus making artificial intelligence better.

The improvement of the IoT ecosystem drives the artificial intelligence and therefore the approaches used for a successful implementation.

Advances in computer processing and data storage have made it possible to capture and analyze more data. data. Shrinking computer chips and improved manufacturing techniques mean smarter sensors? cheap and powerful. The connectivity wireless offers a high volume of data at very cheap rates, and allows all those sensors to send data to the cloud. The birth of the cloud has also allowed a virtually unlimited storage of this data and a considerable capacity? computational processing. All these advances have brought artificial intelligence more and more. close to the final goal of creating intelligent machines that are entering more and more? in daily life.

All the main players have invested heavily in these technologies and, at this point, are we expecting the launch of concretely usable or scatter products and services? the moment of disillusionment.

You often see examples of cognitive services that can be used by any user without installing anything locally. As for example happens with the Cognitive Services of the Azure platform: they allow to activate functionalities? recognition of faces, text in natural language, speech, images relying on the computational resources available on the cloud, similar services can be found on the cloud platforms of the other two giants Amazon and Google.

Practically everywhere in the world you have the possibility? that the external signals are treated and processed by a set of input nodes, in turn connected with multiple internal nodes (organized at levels): each node processes the signals received and transmits the result to the subsequent nodes.

Since neural networks are trained using data, the connections between neurons are thus strengthened and the output gradually forms patterns, i.e. well-defined schemes that can be used by the machine to make decisions, classify and identify.

Recognize facial expressions and emotions ? a? skill? basic that is learned at an early age? and important for human social interactions. Men can look at a person's face and quickly recognize the common emotions of anger, happiness, surprise, disgust, sadness and fear. Transfer this skill? on a car ? a complex task. Researchers have spent decades of engineering time writing computer programs that recognize a feature? accurately, having to repeat the same process to recognize a feature? slightly different.

Deep learning techniques show promise in reducing error rates in the recognition and classification of computer vision systems. The deep learning implementation in embedded systems can help machines interpret facial expressions visually, with near-human levels of accuracy. The recognition of facial expressions is, in fact, the sector that is gaining more importance and is? therefore at the center of studies and application of new algorithms almost constantly, although there are already? methods for identifying expressions using machine learning. Summary of the invention

The main objective of the present invention ? to implement a platform that integrates the User Sentiment Analysis function using and correlating photos to specific events. By way of example: check l? expressiveness? of the face of images posted at a sporting event, a demonstration or a concert; analyze the images through the platform, with the aim of classifying the emotions of the people filmed to trace the perception that the subjects have as a function of the specific related event.

The solution according to the present invention has the aim of overcoming the problems described above. To succeed in this goal? Was it necessary to apply a specific research process in the analysis of expressiveness? of the face and, above all, on the algorithms to be developed starting from the innovations in the field of artificial intelligence. ? It was necessary to adopt video analysis tools applied to the face, starting from basic Face Detection and Face Recognition algorithms, aggregating tools that can verify the orientation and posture assumed by the user captured in the images being analysed.

In particular ? implemented a hybrid telematic network architecture dedicated to the processes of extraction and analysis of opinions, regarding products and services, provided by people through the? of the face, including a? unit? central server and unit? discovery clients connected to it, distributed.

The unit server includes:

the. a?unit? process central;

ii. at least one unit? of memory for the execution of the various applications; iii. of the units? mass storage of the information detected; iv. of the ports of interfacing and communication with the unit? clients; the units? client in turn comprising each:

a) of the units? process peripherals;

b) of the cameras connected each to the respective unit? process device.

The images acquired from the client's Camera Units are transferred to the functionalities? of Processing of the server, subject to a process of analysis in real time including the phases relating to the recognition of faces, and to the determination of expressions, through dedicated applications using Deep Learning techniques. The acquisition of the images, as well as from the unit? client ? also carried out through the use of the connection APIs to the various social media, based on the tags of the events in progress. Video capture? implemented, detecting the videos uploaded on various social networks, which are considered as nodes of a social graph.

For the sole purpose of better understanding the? Invention and without thereby? wanting to limit the scope of validity?, the invention will come? illustrated based on a series of preferred embodiments with reference to the accompanying figures, in which:

- Figure 1 ? a graph highlighting current trends in searches on Stack Overflow;

- Figure 2 ? a block diagram referring to the process of transforming images into function vectors;

- Figure 3 ? a block diagram referring to a different process of transforming images into function vectors;

- Figure 4 ? a block diagram that highlights a comparison between the traditional machine learning process and the deep learning process;

- Figure 5 ? a block diagram highlighting the various stages of expression analysis;

- Figure 6 ? a block diagram referencing a classification process;

- Figure 7 ? referring to a partitioning technique in the context of a face localization procedure;

- Figure 8 ? a view of the assembled components, according to the invention, constituting the system for detecting and processing the data relating to the images;

- Figure 9 ? a general diagram of the telematic network architecture according to the invention;

- Figure 10 ? a combined view of circuit boards implementing the system according to the present invention,

- Figure 11 ? a schema showing the allocation of the clients' camera unit in a certain area;

- Figure 12 ? a representation of a set of filters used in Face Recognition procedures.

Description of preferred embodiments of the invention

To train a model, you need to implement a classifier; does this use the features? of an object to try to identify which class it belongs to. For example, the classifier can be trained to detect if the? image ? a:

Bicycle;

Boat;

Car;

Airplane.

The four objects above are the classes that the classifier must recognize. To build a classifier, ? You need to have some data as input and give it a label. The algorithm takes this data, finds a model, and then classifies it into the corresponding class.

This task ? called supervised learning. In supervised learning, the training data fed to the algorithm includes a label. Training an algorithm requires following some standard steps:

Collect data

Train the classifier(training)

Make previsions

The first step ? fundamental, the choice of the right data will do? yes that the algorithm succeeds or fails. The data you choose to form the model are called characteristics. In the example in question, the characteristics are the pixels of the images.

Each image? a line while each pixel ? a column. If the image has a size of 28x28, then the dataset contains 784 columns (28x28). In the following image, each image is been transformed into a function vector. The label tells the computer which object is in the image. In Figure 2 ? reported a diagram relating to the process of transforming images into function vectors. From the related functionality columns? relating to block 20 of the train data and block 21 of the test data, are the vectors of the characteristics 22 obtained? and 22? with the relative labels 23?, 23?, which could be Bicycle xxxx,

yyyy boat,

vvvv car,

airplane zzzz.

The target ? to use this training data to classify the type of object. The first step is to create the feature columns. The second step provides for the choice of an algorithm 25 to train the model which, through an inferential procedure 24, allows the output of the final instance 26, in this case the value ?car?.

At the end of the training, the model can figure out which image corresponds to which object.

Subsequently, ? easy to use the model to analyze new images. For each new image inserted in the model, the machine indicates the class to which it belongs, applying a certain percentage of membership. The brand new untagged image is scanned by the model. For a human being, ? trivial to identify the image as a car instead the machine uses its previous knowledge to understand that this image also represents a car.

In deep learning, what? as shown in Figure 3, the learning phase takes place through a neural network 27. In this figure, with respect to Figure 2, the three connected layers of the neural network 27 responsible for the deep learning process have been highlighted. Each input enters a neuron and ? multiplied by a ?weight?. The result of the multiplication flows to the next level and becomes the input. This process is repeated for each level of the network up to the final level, which ? called output level; provides an effective value for? activity? of regression and a probability? of each class for? activity? classification. The neural network uses a mathematical algorithm to update the weights of all neurons. The neural network? completely ?trained? when the value of the weights d? an output close to reality?. For example, a well-trained neural network can recognize the object on an image with greater precision than the traditional neural network. A data set can contain from a dozen to hundreds of features. Will the system learn? the relevance of these characteristics. However, not all features are significant to the algorithm. A crucial part of machine learning ? find a relevant set of characteristics to make the system learn something.

Any way to do this part in machine learning? use feature extraction to combine existing features to create a smaller set? relevant features. Can? be done with PCA, T-SNE or other dimensionality reduction algorithms.

For example, for processing an image, it is necessary to manually extract the ?feature? in? the image as the eyes, the nose, the lips and cos? Street. The extracted characteristics are sent to the classification model.

Deep learning solves this problem, especially for a convolutional neural network. The first layer of a neural network analyzes small details from the image; the following levels combine the previous analyzes to make the information more complex. In the convolutional neural network, feature extraction is done with the use of filtering. The network applies a filter to the image to see if there is a match, i.e. the shape of the function ? identical to a part of the image. If there is a match, the network will use this filter. The feature extraction process then runs automatically.

Figure 4 shows a comparative diagram that highlights the difference between deep and traditional machine learning. In it, starting from a given input 28, in the first scheme A relating to traditional machine learning, one enters a phase of feature extration 29 and a subsequent classification 30, which is followed at the end by the identifying output 26. Instead in procedure B, the feature extraction phase and the classification phase are integrated in a single interactive function 31 which produces, in any case, an identifying output 26.

Deep learning? in fact only the starting point for face recognition. The extraction 34 of human facial expressions, which in English is precisely defined as Face Expression Recognition (FER), ? a process that is mainly defined by three phases: pre-processing 32, feature extraction 29 and classification (classification) 30. The pre-processing phase 32 serves for the transformation of the input in such a way as to make the its subsequent processing pi? simpler and more accurate. In general, feature extraction is used to reduce the size of the information needed to describe an otherwise too large dataset. When the input data in an algorithm is too much to process and is suspected to be redundant, then it can be transformed into a reduced set. It starts with a basic set of measured data and creates derived values (characteristics) intended to be informative and non-redundant, facilitating subsequent learning and generalization phases. The ?classification? 30 ? the phase in which the system has the task of finding the membership class of a new input among a set of categories already? defined and generated by a set of observations made previously and already? categorized, called ?training set?. For example, given an email, a system can? decide whether this is spam or not based on the mail received previously and already? labeled. The classification? typically an instance of supervised learning, or rather a learning in which ? already A training set consisting of correctly identified images is available.

The corresponding unsupervised procedure ? known as clustering and involves grouping data into categories based on some inherent similarity or distance. L? algorithm implementing classification 30 ? known as classifier term which sometimes also refers to the mathematical function, implemented by a classification algorithm precisely, which maps the input data into a category.

Figure 5 shows the typical steps of an expression analysis process according to the invention. The actual classification procedure 30 foresees the input from two different paths upstream:

? on the one hand the input of the face image 28, a first pre-processing phase 32? and a feature extraction step 29?,

? on the other hand ? expected that from a database 33 of the facial images one proceeds through yet another pre-processing step 32? and a phase of extraction of the relative characteristics 29?. The result of the activities? of classification 30, on the basis of the two preparatory phases mentioned, ? a cataloging through pre-established taxonomies 34 which can be expressions such as surprise, smile, sadness, anger, fear, disgust. - Pre-processing

The preprocessing ? a process that has the finality? to improve the performance of a facial recognition system and which runs before the feature extraction process, i.e. ? the first processing step. Image preprocessing involves different types of techniques such as image resizing or contrast adjustment in order to improve the frames of faces representing expressions. You proceed with the description pi? details of some techniques.

- Bessel sampling

Bessel sampling is used for face image size reduction without for? going to compromise the perceptual value of the original input. Can you? use gaussian filter for resizing the input images. To reduce the problems related to lighting and variation that vary faces is a possible method? that of normalization with median filter. A median filter? a non-linear filter used for signal smoothing. ? especially useful for removing impulse-type noise from a signal. There are many variations of this filter, and a two-dimensional variant is often used in DSP systems to remove noise and speckles from images. The nonlinear function of the median filter pu? be expressed as:

y(n) = med [x (n-k),x (n-k+1),....,x (n),....,x (n k -1)

where y(n) ? the signal output and x(n) the input. The filter collects a portion of the signal containing N = 2k+1 input samples on which it calculates the median. In pre-processing an image, the filter replaces the gray level of each pixel with the median calculated from adjacent pixels. If the kernel takes an even number of pixels then the average of the two medians is calculated. Unlike other filters, such as the low-pass FIR, the median filter manages to preserve the edges of the image in a better way and for this reason ? widely used in digital image processing. Normalization is also used as a method of identifying the position of the eyes.

- Viola-Jones locator

One of the most popular pre-processing methods? robust for face tracking ? the Viola-Jons locator. Whereas a single classifier looks at a complete image and applies matrix operations to arrive at a classification in a Convolutional Neural Network, the Viola-Jones approach takes an ensemble approach. There? This means that Viola-Jones uses many different classifiers, each looking at a different portion of the image. As shown in Figure 6, it should be noted that from an image 28 a series of weak classifiers 35?, 35?... 35<n> can be deduced, to then arrive at a final instance 26 as output value.

Every single classifier ? more weak (less accurate, produces more false positives, etc.) than the final classifier because? receives less information. When the results of each classifier are combined, however, they produce a strong classifier. The algorithm consists of four phases:

a-Haar Feature Selection

b-Creation of an integral image

c-Adaboost algorithm as trainer

d-Cascade classifiers

An early contribution to the development of the Viola-Jones locator ? was the introduction of a number of image recognition functions. The human face has some general characteristics that are present in all faces. For example, the eye region ? more more dark, while that of the nose ? more clear. Furthermore, the position and distance between the eyes, mouth and nose are some recognizable features. It is precisely these traits that can be identified using Haar's characteristics. The rectangles that make up the Haar features are calculated and have a value equal to

? (pixel in area sorry) - ? (pixels in light area).

what? as shown in Figure 7 which indicates four different partitioning examples: A, B, C, D.

Using the integral image concept ? You can easily find out the area of the rectangle. A? full image? defined by the sum of the pixel values of the original image, where the value in position (x, y) ? given by the sum of the pixels above and to the left of that position.

A Haar feature classifier then uses the integral of the rectangle to compute the value. Multiply the weight of each rectangle by its area and the results come out as follows: added up.

Although evaluating a feature is quite fast, the total number of features that is too high makes the total process underperforming. For example, in a standard 24x24 pixel subwindow, there is a total of M = 162,336 possible features and therefore the evaluation processing becomes onerous. To work around this problem, the object detection framework uses a variant of the AdaBoost learning algorithm to both select the best features and train classifiers to use them. This algorithm constructs a strong classifier as a linear combination of weighted simple weak classifiers. Strong classifiers are cascaded in order of complexity, where each subsequent classifier is trained on only those selected samples that pass through the previous classifiers. When a step is passed, it moves on to the next phase. If a? image exceeds all stages then you can? say what? a face was detected.

The architecture that ? been defined to support the recognition process of facial expressions (and related emotions) requires three fundamental steps: i- acquisition of video streams on Camera Unit 70;

ii- transfer of the images to the processing server 60;

iii- image processing.

Regarding the implementation of the Computer Vision Server, yes? based on availability of examples present on the net, also assisted by the precious scientific material which endorses and validates many hypotheses made on the basis of project requirements. Starting from a consolidated base has made it possible to ?enrich? the feature scripts? such as image storage and communication with other devices present in the project platform.

As shown in Figure 8, in a preferred version the system ? was created with the combination and integration of the following hardware and software devices:

to. 41 computer based on Intel i7 CPU,

b. Linux Ubuntu 16.04 LTS 64Bit operating system,

c. Redis as an in-memory database 43 for data retention, d. 47 video shooting devices (webcam, ipcamera),

And. Computer vision frameworks and libraries:

i- Tensorflow

ii- Inception

iii- NCAppZoo

Figure 9 ? a representative diagram of the connection of the cameras in the entire system with hybrid telematic network architecture dedicated to the processes of extraction and analysis of opinions through the expressiveness of the face, according to the present invention.

The drawings shown indicate the components with which ? Has the image detection system been created locally at the place where the event is taking place? interested. The client-server type network architecture? schematized in Figure 9, in which the units are evident? of power 40, of the unit? central processing 41 server side 60, which are associated with the activities? numerical processing 44, equipped with units? memory 42 type microSD card with functionality? of storage, warehousing and archive 43. The connection through sockets 45 with a generic unit? peripheral 70 foresees the interworking with detection devices on the network which of the units? of client processing 46 in turn in connection with detection cameras 47 and/or with further sensors 48.

To allow the immediate transmission of camera images ? was it necessary to equip the circuit with a microcontroller capable of sending the content of the images, for the prototype ? an Arduino Nano was used, as per the connection diagram in Figure 10 for the redirection of the cameras.

The invention integrates the electronic technical characteristics of two different families of computers:

A- the Raspberry PIs, which are computers in all respects, equipped with software and an operating system with the ability? management of many heterogeneous processes that share the resources of the "machine", simultaneous management of web servers, databases, image acquisition and processing, network communication, file sharing and transfer, and

B- Arduino: what? a microcontroller with a series of inputs / outputs and communication buses available to run a single program permanently loaded into memory with a vast collection of libraries that simplify the task of interfacing the I/O and, above all, in ?application de quo, ? ideal for the transmission of videos and images.

As shown in Figure 11, it is preferable that the video cameras 47?, 47?, ?47<n > are arranged ? within the relative unit? client ? on a perimeter and are suitably interconnected with the server 60. Each sensor ? been positioned on the basis of the points where you can? divide the affected area as if it were a flat map. This makes it possible to have a matrix capable of providing important information on the zone in which the detected event occurs (expression). The diagram that follows shows a diagram of the distribution of the video cameras hypothesized in a medium-small area, in different contexts the position and number of video cameras can quietly vary. In an even more favorite microcomputer board 41 ? equipped with an ARM processor that is typically adopted in smartphones, tablets and various electronic devices. It does not have a hard disk, like traditional computers, but uses an SD 42 card on which? contained the operating system. Total memory available ? of 256MB which if compared to recent computers may seem very few, but even with this little memory the card? capable of providing excellent performance. The whole ? was designed so that the operating system is optimized in order to minimize the waste of quantity? of resources, both processor and memory.

Below is the Raspberry Pi 3 Raspberry Pi 3 Specification verbatim

Quad Core 1.2GHz Broadcom BCM2837 64bit CPU

1GB RAM

BCM43438 wireless LAN and Bluetooth Low Energy (BLE) on board

100Base Ethernet

40-pin extended GPIO

4 USB 2 ports

4 Pole stereo output and composite video port

Full size HDMI

CSI camera port for connecting a Raspberry Pi camera

DSI display port for connecting a Raspberry Pi touchscreen display

Micro SD port for loading your operating system and storing data

Upgraded switched Micro USB power source up to 2.5A

The Raspberry Pi Zero 46 ? an ultra-small and ultra-slim version of the well-known single-board computer, with a smaller form factor? small and connectivity? Wi-Fi and Bluetooth on board. Pi? 40% faster than the (original) Raspberry Pi 41 but is only 65mm long, 30mm wide and 30mm deep. of 5mm. The Raspberry Pi Zero 46 can? be equipped with a Raspberry Pi Camera Module v2, one of the cameras pi? popular available for the Raspberry family, with a resolution of 8 Mega Pixels and Sony IMX219 CMOS image sensor, with a quality? of the very high and stable image. ? capable of 3280 x 2464 pixel static images and also supports 1080p30, 720p60 and 640x480p90 video.

It connects to the Pi via one of the small sockets on the top surface of the board and uses the dedicated CSI interface, designed specifically for interfacing with cameras.

Technical specifications of the Raspberry Pi Zero

Raspberry Pi Zero Specification

1GHz single-core CPU

512MB RAM

Mini-HDMI port

Micro USB OTG port

Micro USB power

HAT compatible 40-pin header

Composite video and reset headers

CSI camera connector (v1.3 only)

From a logical point of view, the automatic system proposed for the recognition of facial expressions? characterized by a typical three-stage architecture, shown in Figure 5.

The first stage? the one responsible for face recognition 32 and is achieved through semi-automatic image processing techniques. The second one ? the one that deals with the extraction of the attributes 29 of the images useful for the recognition of the facial expression. Typically these are techniques that are computationally more? expensive and therefore applied only to areas of the image where there are faces. The last stage 30 ? that instead of classification, an area in which the best results are obtained through the application of techniques based on recurrent neural networks.

These processing stages are mapped in the hybrid architecture of the proposed system as specified in the following Table

Table 1: Architecture mapping

Architectural Component Processing Stage

Raspberry Pi Zero 46 Facial Recognition

Raspberry Pi 3 41 Feature Extraction

Centralized System 60 Classification of Expressions

From the point of view of the software components created for this purpose, the Raspberry 41 which has the role of Processing Unit acts as a server, waiting for connection requests from clients 70. For this purpose, it implements a main thread (Socket Server) 45 which listens on a predefined TCP port and, at each connection request, delegates the management of its life cycle to a child thread.

Luckily ? the Python-SocketIO library that implements this mechanism in the WSGI package. The source code below shows how easy it is to build a Multi-threaded server with this library.

In this example the listening TCP port ? the 500, while the? IP address ? 0.0.0.0, which implies that all connections from any interface available on the device will be accepted.

Acquisition of Video Streams

The acquisition of the video streams takes place on the Raspberry Pi Zero 46 which are equipped with a Raspberry Pi Camera Module v2 through the use of the mjpeg-streamer library.

Video capture can Furthermore, it can be implemented by fetching the videos uploaded on the various social networks 78, through the use of the Developer API Graph which they have. In this context, videos are considered as nodes of a social graph, to which it will be? It is possible to access, of course, only with the right authorizations.

As long as the video does not have special requirements given by music licenses, the permissions follow the following rules:

A page access token can see the videos on that page

A user access token can see every video created by the application through that user

Current user?s videos can only be viewed if the user has given user_videos and user_posts permission

Each valid access token can read the photographs on a public page Using, for example, the Instagram Graph API, an example of acquiring images from the social networks Facebook and Instagram, can be represented by the following code:

?from itertools import chain

from igql import InstagramGraphQL

import requests

def get_media_for_hashtag(hashtag):

igql_api = InstagramGraphQL()

hashtags = igql_api.get_hashtag(hashtags)

yield from chain.from_iterable(hashtag.recent_media())

def get_media_files_from_hashtag(hashtag):

for media in get_media_for_hashtag(hashtag):

url = media[?images?][?standard_resolution?][?url?]

yield requests.get(url).content?

Where ? the specific library has been previously installed:

pip install InstagramGraphQL

- Communication via Socket

Once the images have been acquired by the Raspberry Pi Zero 46, they must be transferred to the Raspberry Pi 3 41 for processing. This operation is performed with the aid of sockets, as schematized in the following figures. In particular, web sockets are used, an evolution of technology which allows full-duplex communication to be established within the same TCP connection. The library used for the purpose ? Python-SocketIO.

To ensure communication and data exchange between the microcontroller and the processing infrastructure? It was necessary to establish a wireless connection. The devised solution made it possible to create a bluetooth connection between the microcontroller 41 and a further pre-processing device 46 (Raspberry Pi Zero).

Raspberry allows you to start a Linux Like operating system, several distributions are available but for specific use the project provides for the use of Raspbian. The software packages needed to receive the data from the sensors and further harmonize them into a data flow that can be managed by the centralized systems were then installed.

For the purpose of realizing the sensing and processing system, what has been analyzed up to now, in the present description, leads to designing a first solution using the components which can meet the requirements dictated by the project specifications.

The following layout shows a logical scheme of the solution to be implemented, starting from the event to be analyzed up to the consultation interfaces which are of assistance to specialized personnel.

A series of detection video cameras have been implemented as shown in the diagrams shown in the figures and each of which ? connected to remote databases. The assembly of standard hardware ? supervised by suitably trained staff cos? to make up for the safety problems inherent in the objects present in the construction phase. In addition, each element suitably configured and tested so that the captured images never remain on the device but are immediately downloaded to the centralized platform. As for the activities? of preprocessing storage, the unit? of storage 43 receives the data through the communication port 45; ? complied with an encoding of the data sent in order to distinguish which is the source, the time reference value, and the location of the camera.

The unit Raspberry 41 has in its operating system (Raspian) a software module capable of interpreting and harmonizing these values it receives.

The system provides for the management of flows from third-party APIs. The acquisition of the images as well as from the suitably installed and placed devices is carried out through the use of the connection APIs to the various social media 78. Based on the event tags that match the one subjected to analysis via ad hoc cameras, it therefore becomes possible access to?huge amount? of the data provided by the major players in the field of social networks, clearly entering the domain of big data.

With regard to the methodology and data archiving services, the versatility of the data storage should be emphasized. of Python: a multi-paradigm language that has among the main objectives dynamism, simplicity? and flexibility. It supports object oriented paradigm, structured programming and many functional programming features. The peculiarity of this language? That ? been for years a viable alternative to scripting languages already? used quite commonly, carving out a niche for themselves in academia. But this has meant that at the time of the outbreak of the AI, what is it? developed above all in some renowned universities?, the porting of almost all the libraries in this? ambit ? been done using precisely this language, by splashing the rating among the languages pi? used.

Redis ? an open source (BSD licensed) in-memory data structure archive, used as a database, cache, and message broker. It supports data structures, such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and data streams. Redis has built-in replication, Lua scripting, LRU eviction, supports transactions and different levels of on-disk persistence, and offers high availability. via Redis Sentinel and automatic partitioning with Redis Cluster. ? You can perform atomic operations on these types, such as adding a string; increment the value in a hash; add an item to a list; calculus set and intersection, union and difference; or get the data with the score pi? tall in a neat set. For exceptional performance, Redis works with an in-memory dataset. Depending on the use case, ? You can save your data by occasionally dumping the dataset to disk or by adding each command to a registry. Persistence can optionally be disabled, if you only need a feature-rich, in-memory cache over the network. Redis also supports asynchronous master-slave replication which ? quite trivial to install, with a very fast non-blocking first sync, an automatic reconnection with partial resync on the network nodes.

Other features include:

Transactions

Pub / Sub

Lua scripting

Keys with a limited life time

Eviction of LRU keys

Automatic failover

? Is it possible to use Redis from most of the older programming languages? common. Redis ? written in ANSI C and runs in most POSIX systems such as Linux, *BSD, OS X with no external dependencies. Linux and OS X are the two operating systems where Redis ? developed and tested more? and many recommend using Linux for distribution, however pu? run in Solaris-derived systems such as SmartOS but there is no official support for Windows builds.

Redis with Python

redis-py ? one of many Python clients for Redis, but has the particularity? to be sponsored by the Redis developers themselves. It basically allows you to call Redis commands from Python and get Python objects in return.

To install, you need to download the Redis source code as a tarball: $ redisurl=?http://download.redis.io/redis-stable.tar.gz?

$ curl -s -o redis-stable.tar.gz $redisurl

Subsequently, ? You need to switch to rooted and extract the source code of the archive to /usr/local/lib/:

$ sudo to root

$ mkdir -p /usr/local/lib/

$ chmod a+w /usr/local/lib/

$ tar -C /usr/local/lib/ -xzf redis-stable.tar.gz

Once downloaded and unpacked the source code on /usr/local/lib/redis-stable/ since? Redis ? written in C, you need to compile, link and install the software with the make command:

cd /usr/local/lib/redis-stable/

$ make && make install

make install performs two actions:

The first make compiles and links the source code.

make install takes the binaries and copies them to /usr/local/bin/ so you can run them from anywhere.

At this point, confirmation is requested that Redis is in the PATH and its version is checked:

redis-cli --version

redis-cli 5.0.3

If the shell can't find redis-cli, you need to make sure that /usr/local/bin/ is on your environment PATH variable and add it if not.

The fundamental use of Redis in this project ? given that it has a client-server architecture and uses a request-response model. There? means that the user (the client) connects to a Redis server via TCP connection, on port 6379 by default. You request a certain action(request), and the server returns a resource(response) to us.

There can be many clients talking to the same server, which ? typical of an architecture that uses Redis for a client-server application. Each client performs a (usually blocking) read on a socket while waiting for the server's response.

The Python Redis client library redis-py gets the job done admirably. Encapsulates a TCP connection to a Redis server and sends raw commands, as serialized bytes using the REdis Serialization Protocol (RESP), to the server. It then takes the raw response and parses it back into a Python object.

By setting the db parameter, that ? the database number, ? is it possible to manage more? database at the same time and each ? identified by an integer. The maximum number of databases ? 16 by default.

When you just run redis-cli from the command line, you are referencing database 0. One very important point? the expiration of the key, that ? another distinguishing feature in Redis. When a key expires, that key and its corresponding value will automatically be deleted from the database after a certain number of seconds or at a certain timestamp.

The images acquired by the Camera Unit 70 and transferred to the Processing Unit 60 are subjected to an analysis process which consists of two phases:

i- Recognition of faces;

ii- Determination of expressions.

The following image shows an example application of the library.

The Python code needed to use it? very simple, and consists of a few lines of code:

import face_recognition

image = face_recognition.load_image_file(?)

face_locations = face_recognition.face_locations(image)

face_encodings = face_recognition.face_encodings(image, face_locations); for (y1, x1, y2, x2) in face_locations:

cv2.rectangle(image, (x1, y1), (x2, y2), (255, 255, 255), 5)

As regards the recognition of facial expressions, the Emotion-Detection-1 library is available, which uses the Haar Cascade Classifier?.

Object recognition using the Haar Cascade classifier ? a method proposed in 2001, based on machine learning. In this method, a ?cascade? it is trained starting from a set of positive and negative images, i.e. images which contain or do not contain the object to be recognised.

Libraries identical to convolutional kernels are usually used for face recognition. Each of these images corresponds to a single value obtained by subtracting the sum of the pixels in the white rectangles with the sum of the pixels in the black rectangles.

Figure 12 shows feature images used for facial recognition. The image (a) ? relating to edge features; the image (b) a of the line features; the last figure (c) ? related to a 4-rectangle feature. Starting from these images, their position and their dimensions within a photo, determines whether there is? or not a face inside the photo.

Advantages and industriality? of the found

The library contemplates the recognition of emotions both in modality? with singleface and multiface frames.

In the present invention the face recognition ? built using the Python Face Recognition library, introduced earlier. Based on the dlib C++ library, which uses Deep Learning techniques for face recognition, it has an accuracy of 99.38%.

After all, the application of the code just mentioned to a sample image of size 1024x768 took about 500ms for processing, which can? be considered a good performance if we consider that the image has several recognizable faces, a situation that is unlikely to arise? in cases that arise in reality?.

Claims (10)

Claims 1. Hybrid telematic network architecture dedicated to processes of extraction and analysis of opinions, regarding products and services, provided through the expressiveness of the face, characterized by the fact that it includes a? unity? central server (60) and units? client (70) of detection connected to it, distributed, in a given area, the unit? server (60) constituting the? unit? processing and analysis including in turn: i- a? unit? central process (41) based on one or more? microprocessors (44) with corresponding functions of interfacing and communicating with the relative peripherals; ii- at least one unit? memory (42) of such dimensions as to allow the execution of the processing and analysis in real time of the data coming from the various detection, acquisition and control functions; iii- of the unit? of memorization (43) of the detected information able to constitute a mass memory from which to recover the data for the elaboration and the analysis of the same; iv- of the ports (45) of interfacing and communication with the unit? client(70); the units? client (70?, 70?,?.70<n>) in turn including each: a- of the units? process peripherals (46?, 46?, ?46<n>) equipped with their own processor for local control and interfacing with the server (60); b- at least one camera (47?, 47?,?..47<m>) connected to the respective unit? process device (46?, 46?,?..46<n>); c- redirection devices (71) which are based on their own microcontroller for the immediate redirection of the images detected by the television camera (47) to the unit? central server (60); the? interfacing and? sending of data between the? unit? peripheral (70) and the server (60) taking place on the basis of a socket-based protocol in which the processor (41) of the unit? central (60) acts as a server, waiting for connection requests from clients (70), implementing a main thread, such as Socket Server, which listens on a predefined port and, at each connection request, delegating the management of the life cycle of the same to a child thread, being, the images acquired, from the Camera Unit of the client (70), and transferred to the functionalities? of Processing of the server (60), subject to a process of analysis in real time including the phases relating to the recognition of faces, and to the determination of expressions, through dedicated applications using Deep Learning techniques, being the acquisition of images, as well as from the units? clients (70), suitably installed on the territory, also carried out through the use of the APIs for connection to the various social media (78), based on the event tags that correspond to the one subjected to analysis via the cameras (47?, 47? , ?47<n>) placed, the video acquisition being implemented, with the detection of videos uploaded on various social networks, through the use of non-proprietary applications already? exist, videos being considered as nodes of a social graph, which can be accessed, with the envisaged levels of authorization, in order to search for a given image related to a given event. 2. Hybrid telematic network architecture dedicated to processes of extraction and analysis of expressiveness of the face of a group of people according to claim 1, characterized in that the television cameras (47?, 47?, ?47n) of the units? clients are distributed on a perimeter of the interested area considered as a flat map, arranged spaced evenly, interconnected with the central server (60), hence obtaining a matrix capable of providing information on which is the area in which ? present the detected event / facial expression. 3. Hybrid telematic network architecture dedicated to processes of extraction and analysis of expressiveness of the face of a group of people according to claims 1 and 2, characterized by the fact that the units? of storage (43) receiving the data through the communication port (45), memorize the instance of the event under analysis through the encodings of the data acquired with the information relating to the source, the time reference value, and the location of the camera (46, 70). 4. Hybrid telematic network architecture dedicated to processes of extraction and analysis of expressiveness? of the face of a group of people according to the previous claims characterized by the fact that the assembly of the standard hardware is configured and tested so that the images acquired by the unit? clients (70) are immediately sent to the centralized platform (60) by means of a dedicated microcontroller (71). 5. Hybrid telematic network architecture dedicated to processes of extraction and analysis of expressiveness? of the face of a group of people according to claim 1, characterized in that it comprises a first stage which implements the recognition of faces and ? implemented through semi-automatic image processing techniques, a second stage that deals with the extraction of image attributes useful for the recognition of facial expression, which being computationally more? onerous ? applied only to the areas of the image in which there are faces, and a third stage in charge of classification, in which optimal results are obtained through the application of techniques based on recurrent neural networks, said three processing stages being constituted in the architecture of the system based on the following correspondence: Architectural Component <--> Processing Stage Raspberry Pi Zero (46) <--> Facial Recognition Raspberry Pi 3 (41) <--> Feature Extraction Centralized system (60) <--> Classification of expressions 6. Method of analysis dedicated to the elaboration of processes of extraction and examination of the expressiveness of the face of one or more people according to the telematic network architecture of claims 1 to 5 characterized in that the extraction (34) of human face expressions is a process comprising three distinct phases, relating to: i- pre-processing (32), which operates the transformation of the input in such a way as to make its subsequent processing more? simpler and more accurate, through the possible reduction of the size of the information necessary to describe a set of data associated with each image; at? ii- feature extraction (29), in which the features or derived values, intended to be informative and non-redundant, are created starting from a basic set of measured data, so as to facilitate the subsequent learning and generalization phases; and to the iii- classification (30) in which the system operates in order to detect the belonging class of a new input among a set of categories already? defined and generated by a set of observations implemented previously and already? categorized, operating as a training set. 7. Method of analysis dedicated to the elaboration of processes of extraction and examination of the expressiveness of the face of one or more people based on the telematic network architecture according to the preceding claims characterized in that it provides for the acquisition of Video Streams, by acquiring videos uploaded on the various social networks (78), through the use of the available Developer API Graph, the videos being managed as the nodes of a social graph, to which ? It is possible to access having the necessary permissions available based on the following priority levels: i- an access token to a page sees videos of that page, ii- an access token of the user sees every video created by the application by that user, iii- current user?s videos can be viewed only if the user has given permission user_videos and user_posts, iv- any valid access token can? read the photographs on a public page. 8. Method of analysis dedicated to the elaboration of processes of extraction and examination of the expressiveness of the face of one or more persons according to the preceding claims characterized by the fact that in the pre-processing phase the performance of the facial recognition system performed before the feature extraction process is optimized, providing for the resizing of the images or the adjustment of the contrast so as to improve the frames of the faces represent expressions. 9. Method of analysis dedicated to the elaboration of processes of extraction and examination of the expressiveness of the face of one or more persons according to the preceding claims characterized by the fact that in the preprocessing phase a multiplicity of? of different classifiers, each of which ? aimed at a different portion of the image, each individual classifier being less accurate, than the final classifier as it receives less information, hence the results of each classifier being in turn combined, to provide a strong classifier, the construction of strong classifiers as a linear combination of weighted simple weak classifiers, leading to the arrangement of the same in cascade in order of complexity, each successive classifier being trained only on those selected samples that pass through the previous classifiers, through a sequentialization of the examination phases, only if the ?image passes all stages then the presence of a face is associated with it. 10. Method of analysis dedicated to the elaboration of processes of extraction and examination of the expressiveness of the face of one or more people according to the previous claims characterized by the fact of using the deep learning procedures with a convolutional neural network, the first layer of a dedicated neural network analyzing only minimal details from the image and the subsequent levels by combining the previous analyzes to make the information more? complex and articulated.