KR102493790B1 - Artificial intelligence-based place-linked facial recognition information generation method, device and system - Google Patents

Abstract

According to one embodiment, a method for generating artificial intelligence-based place-linked face recognition information comprises the steps of: taking an image with an image recognition device installed in a space (hereinafter referred to as 'designated space') where a plurality of people move; recognizing the plurality of people from the taken image; collecting predetermined behavior data from terminal devices of the plurality of people; simultaneously identifying emotional states (hereinafter referred to as 'group emotion data') of the plurality of recognized people; collecting current weather data; selecting recommendation content corresponding to the designated space based on the behavior data, the group emotion data, and the current weather data; and recommending the recommendation content regarding the designated space. According to the present invention, an emotion model is generated by considering characteristics of a space where a plurality of people moves so that the accuracy in identifying emotion can be improved.

Description

Artificial intelligence-based place-linked facial recognition information generation method, apparatus and system

The following embodiments relate to a method, apparatus, and system for generating face recognition information linked to an artificial intelligence-based place, and to identify public emotions based on people's expressions or gestures in a space where a large number of people flow, such as bookstores and department stores, Based on this, it relates to a technology for recommending curated content.

Interest in the technology of curating the contents (books, movies, performances, products, etc.) that customers need is greatly increasing.

In particular, there is a demand for content recommendation technology through predetermined curation for a large number of people who move through bookstores or department stores.

In the prior art such as Patent Document 1, a technique of recommending customized content for one member (individual) has been proposed.

However, in the prior art as described above, since curation is provided for each customer (member), there is a limit in that it is impossible to provide curation for marketing in a space where a large number of customers flow.

KR 10-2232957 B KR 10-2372017 B KR 10-2371119 B

The problem to be solved by an embodiment of the present invention is to identify the emotions of a large number of people in a designated space where a large number of people flow, such as a bookstore or department store, and combine them with other data such as weather to create content to be recommended in that space. It is to provide a method, device, and system for generating face recognition information based on artificial intelligence based place that curates.

According to one embodiment, the step of photographing an image with an image recognition device installed in a space in which a plurality of people move (hereinafter, referred to as 'designated space'); Recognizing a plurality of people from a photographed image; collecting predetermined behavioral data from terminal devices of the plurality of people; Simultaneously grasping the recognized emotional states of a plurality of people (hereinafter, referred to as 'group emotion data'); Collecting current weather data; selecting recommended content corresponding to the designated space based on the behavioral data, the collective emotion data, and current weather data; and recommending the recommended content for the designated space.

In addition, the recommended content is: When a predetermined book is selected and a request for outputting the recommended content is received at a predetermined kiosk disposed in the designated space, outputting a one-handed book extracted from a portion of the recommended content; Recording personal emotion data by measuring skin conductivity, oxygen saturation, and pulse wave of the person printing the one-handed book through the pulse rate moisture sensor provided in the kiosk (hereinafter, referred to as 'first personal emotion data recording step') ); a purchase step of moving to a recommended content purchase page of an Internet shopping mall or sending a purchase request message to a payment terminal in the designated space when the QR code printed on the one-handed book is scanned; When a purchase request message is transmitted to the payment terminal in the designated space in the purchase step, capturing an image of the recommended content buyer and recording personal emotion data (hereinafter, referred to as 'second personal emotion data recording step'); and updating a weight for each emotion of the recommended content based on the accumulated personal emotion data.

And, generating a first emotion model from a predetermined first emotion image set in order to determine an emotional state from the image; classifying strong emotional expressions and weak emotional expressions for each predetermined emotion classification from the first emotion model; generating a second emotion image set by excluding emotion images corresponding to strong emotional expressions from the first emotion image set; and generating a second emotion model from the second emotion image set; the group emotion data may be grasped based on the second emotion model.

In addition, the accumulated personal emotion data is built according to: the initial personal emotion data construction step performed before the first personal emotion data recording step and the second personal emotion data recording step are performed, and the initial personal emotion data The construction step includes: transmitting a predetermined movie scene to the tester while the tester is wearing the pulse moisture sensor; measuring skin conductance, oxygen saturation, and pulse wave of the tester; inputting the tester's skin conductance, oxygen saturation, and pulse wave into a Support Vector Machine (SVM); Based on the predicted emotion specified for each movie scene, an error rate is calculated by dividing the number classification extracted by the support vector machine by the number of misclassifications, and the gamma value (γ) and coefficient (c) of the kernel function applied to the support vector machine ) and dimension (d) to calculate each error rate; Joy, sadness, tension, and excitement are obtained by using a support vector machine to which a selected kernel function is applied among a plurality of kernel functions designated by adjusting the gamma value (γ), coefficient (c), and dimension (d) of the kernel function. generating third personal emotion data for the four emotions; and recognizing the tester's emotion based on the third personal emotion data, marking it on a predetermined display, and receiving feedback from the tester on which emotion among joy, sadness, tension, and excitement corresponds thereto. can

In addition, the initial personal emotion data establishment step may include: acquiring a predetermined personal video of the examiner; extracting a video image of a first point of view, which is the last point of time of the personal video, as a first image from the personal video; dividing an area occupied by the tester's face into a first area in the first image, and extracting a 1-1 image by dividing a portion of the first image where the first area is located; As a result of reproducing and analyzing the personal video in reverse chronological order from the first viewpoint, if it is confirmed that the tester's face does not deviate from the first region at the second viewpoint, the second viewpoint to the first viewpoint setting to 1 section; Recording fourth personal emotion data for determining the emotion for the personal video in the first section based on the 1-1 image (hereinafter, referred to as 'fourth personal emotion data recording step'); extracting a video image of a second viewpoint from the personal video as a second image; dividing an area occupied by the tester's face into a second area in the second image, and extracting a 2-1 image by dividing a portion of the second image where the second area is located; After the step of extracting the 2-1 image, as a result of reproducing and analyzing the personal video in reverse chronological order from the second viewpoint, if it is confirmed that the tester's face is not out of the second area at the third viewpoint, setting a period from the third time point to the second time point as a second interval; recording fifth personal emotion data for identifying emotions for the personal video of the second section based on the 2-1 image (hereinafter, referred to as 'fifth personal emotion data recording step'); comparing the emotion identified in the fourth personal emotion data recording step with the emotion identified in the fifth personal emotion data recording step, and recording personal emotion data classified as a weaker emotional expression as initial personal emotion data; can include

An apparatus according to an embodiment may be combined with hardware and controlled by a computer program stored in a medium to execute any one of the methods described above.

According to an embodiment, in a space where a plurality of people move, content may be recommended to people in the space based on emotions of the plurality of people.

In addition, content can be curated based on data more closely related to emotions by combining emotion data based on facial expressions, behavioral data (such as people's search records), and weather data.

In addition, by issuing a predetermined one-handed book for the content (book) selected by the customer among the recommended content, it is possible to help customers purchase the book based on the emotion of the printed matter, not the electronic device.

In addition, the accuracy of curation can be gradually improved by accumulatively recording behaviors for specific facial expressions (emotions), such as when a customer purchases a book.

In addition, by generating an emotion model considering the characteristics of a space in which a large number of people flow, it is possible to improve the accuracy of emotion identification.

In addition, before data is accumulated in a space where a large number of people actually move, initial personal emotion data may be established based on a pulse rate sensor or/and a personal video.

1 is a diagram schematically showing the configuration of an artificial intelligence-based place-linked facial recognition information generating system according to an embodiment.
2 is a diagram showing an example of a one-handed book output from an artificial intelligence-based place-linked facial recognition information generation system according to an embodiment.
3 is a diagram illustrating an example of a screen output while recognizing an emotion with a pulse rate and moisture sensor in a kiosk of an artificial intelligence-based place-linked facial recognition information generation system according to an embodiment.
4 is a diagram illustrating an 'initial personal emotion data construction step (initial emotion classifier construction process)' of the artificial intelligence-based place-linked face recognition information generation method according to an embodiment.
5 is a flowchart illustrating a method for generating face recognition information based on place linking based on artificial intelligence according to an embodiment.
6 is a flowchart illustrating a form in which one-handed book-type promotion is applied to an artificial intelligence-based place-linked facial recognition information generating method according to an embodiment.
7 is a flowchart illustrating in detail the initial personal emotion data construction step of the artificial intelligence-based place-linked face recognition information generating method according to an embodiment.
8 is a flowchart illustrating in detail the initial personal emotion data construction step of the artificial intelligence-based place-linked face recognition information generation method according to an embodiment.
9 is a flowchart illustrating a process of generating an emotion model excluding strong emotional expressions in an artificial intelligence-based place-linked facial recognition information generation method according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes can be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents or substitutes to the embodiments are included within the scope of rights.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be modified and implemented in various forms. Therefore, the embodiments are not limited to the specific disclosed form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Terms used in the examples are used only for descriptive purposes and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

In addition, in the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

The embodiments may be implemented in various types of products such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices.

An artificial intelligence (AI) system is a computer system that implements human-level intelligence, and unlike existing rule-based smart systems, machines learn and judge on their own. The more AI systems are used, the higher the recognition rate and the more accurate understanding of user preferences. Conventional rule-based smart systems are gradually being replaced by deep learning-based AI systems.

Artificial intelligence technology consists of machine learning and element technologies using machine learning. Machine learning is an algorithm technology that classifies/learns the characteristics of input data by itself, and element technology is a technology that uses machine learning algorithms such as deep learning to mimic functions such as recognition and judgment of the human brain. It consists of technical fields such as understanding, inference/prediction, knowledge expression, and motion control.

The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying/processing human language/characters, including natural language processing, machine translation, dialogue systems, question and answering, voice recognition/synthesis, and the like. Visual understanding is a technology for recognizing and processing objects like human vision, and includes object recognition, object tracking, image search, person recognition, scene understanding, space understanding, image improvement, and the like. Inference prediction is a technique of reasoning and predicting logically by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data creation/classification) and knowledge management (data utilization). Motion control is a technology for controlling the autonomous driving of a vehicle and the movement of a robot, and includes motion control (navigation, collision, driving), manipulation control (action control), and the like.

In general, in order to apply machine learning algorithms to real life, learning is performed in a trial and error method due to the nature of the basic methodology of machine learning. In particular, deep learning requires hundreds of thousands of iterations. It is impossible to execute this in an actual physical external environment, so instead, the actual physical external environment is virtually implemented on a computer and learning is performed through simulation.

1 is a diagram schematically showing the configuration of a system according to an embodiment.

Referring to FIG. 1 , a system according to an embodiment may include a plurality of customer terminals 2 and one or more image recognition devices 3 . In addition, a main server 1 that is connected to the customer terminal 2 and the image recognition device 3 by wire or wirelessly to transmit and receive information and process data, and a database server that manages data of the main server 1 ( 4) may be included. In addition, a predetermined kiosk 5 connected to the main server 1 and disposed in a designated space to interact with customers (people) may be included.

Most of the steps to be described later can be performed by the main server 1 unless otherwise noted, and in addition, it is possible to be distributed and processed in the customer terminal 2 or kiosk 5, etc. A skilled person will be able to easily convert and specify by referring to the description of the present invention.

According to one embodiment, in step (S100) of photographing an image with an image recognition device 3 installed in a space in which a plurality of people (customers) move (hereinafter referred to as a 'specified space'); in, CCTV or people Photos or videos may be taken with a camera installed separately at a predetermined height so that the face of the person is photographed from the front.

Then, in the step of recognizing multiple people from the captured image (S200), multiple people in the image can be recognized simultaneously through a predetermined object recognition library/algorithm.

The step (S300) of collecting predetermined behavioral data from the terminal devices of the plurality of people; may be performed when the customer terminals 2 such as smart devices equipped by the people are linked to the main server 1 .

For example, if the person is a member registered in the database of the main server 1, certain search records, content purchase history, shopping cart, content search history, frequently visited places, frequently visited corners (eg For example, data such as bookstore self-development, humanities, and religion divisions) can be collected.

Such behavioral data may also be calculated as an index for a designated space, such as collective emotion data described later, and accordingly, de-identification of behavioral data may proceed naturally.

For example, since customer A's 'depression'-related content search record, which has visited a designated space, is separated from customer A and functions only as data for the designated space, the fact that customer A has searched for 'depression' is not known to the outside world. becomes unknown in

In step S400 of simultaneously grasping the recognized emotional states (hereinafter referred to as 'group emotion data') of a plurality of recognized persons, the emotional states of each of the recognized plurality of persons are simultaneously grasped.

An artificial intelligence algorithm for identifying human emotions from images or photos is a well-known technology, so a detailed description thereof will be omitted.

Each of the emotional states identified for a plurality of people may be summed up and calculated as collective emotional data. For example, a dominant emotion may be selected through a predetermined process, such as identifying emotion indices for a predetermined emotion classification for each person and averaging or summing them, or a separate emotion index for a designated space may be calculated.

In the step of collecting current weather data (S500), weather information around a designated space at the current time is collected from a weather API of the Korea Meteorological Administration.

For example, for two designated spaces A and B to which the system according to an embodiment of the present invention is applied, different weather data may be collected from a designated space A located in Guro-gu, Seoul and a designated space B located in Yuseong-gu, Daejeon. .

A predetermined emotional index may be designated for each weather in the weather data. For example, a weather in which it rains intermittently can be designated as having a high sadness emotion index.

Selecting recommended content corresponding to the designated space based on the behavioral data, group emotion data, and current weather data (S600) means curating content based on three different types of data. .

Weights for each of the three different types of data may be differently designated according to a designated space or policy.

For example, in the case of a bookstore, since it can be determined that the current emotion has a relatively large influence on the selection and purchase of books, the weight of collective emotion data and weather data can be set high.

As another example, a low weight of group emotion data and a high weight of behavior data may be designated for content in which it is determined that the influence of emotion on content selection and purchase is relatively small.

The process of curating content by combining different data can be performed by those skilled in the art by referring to well-known curating methods, so a detailed description will be omitted.

As an example of the step of recommending the recommended content for the designated space (S700), a predetermined notification, message, or / and data related to the recommended content can be transmitted to the customer terminals 2 of customers who have visited the designated space. . As another example, an image or video related to recommended content may be transmitted to a predetermined display device (including the kiosk 5) installed in the designated space. The above two examples may be applied selectively or simultaneously.

One embodiment of the present invention can be appropriately used as a purpose of recommending books to customers visiting a bookstore. Accordingly, the recommended content may be selected as a predetermined book.

In addition, when a request for outputting recommended contents (recommended books) is received at a predetermined kiosk 5 disposed in the designated space (S800), a one-handed book extracted from a part of the recommended contents is output (S810); [0041] In [0083], a part of the recommended contents may be output on a predetermined thermal paper and provided to the customer.

Customers who have been recommended content in the above manner can approach the kiosk 5 placed around them and operate the kiosk 5, and manipulate the one-handed book output button of the recommended content in the kiosk 5 as described above. One-handed books may be provided.

Preferably, 2 to 5 pages of contents of a general book (recommended contents) may be extracted and output to the thermal paper.

According to one embodiment, a one-handed book can be printed on thermal paper with a size of 80mm X 300mm, a font size of 10px, and a line spacing of 1.0. 2.4 pages) may be included. An example of a one-handed book thus output is shown in FIG. 2 .

Recording personal emotion data by measuring skin conductance, oxygen saturation, and pulse wave of the person outputting the one-handed book through the pulse rate moisture sensor provided in the kiosk 5 (hereinafter, 'first personal emotion data recording step') '; S820);, the accuracy of emotion identification can be improved with the help of the customer.

For example, all customers who print one-handed books by limiting the operation of the pulse moisture sensor provided in the kiosk 5 for printing while providing the printing of one-handed books free of charge only to members. of personal emotion data can be collected. However, for this purpose, it is desirable to perform consent procedures for the collection and use of personal emotional data when signing up for membership or using the kiosk (5).

Then, when the QR code printed on the one-handed book is scanned (S830), the purchase step of moving to the recommended content purchase page of the Internet shopping mall (S841) or sending a purchase request message to the payment terminal in the designated space (S842) ( S840); may be performed.

In the above description, 'QR code' can be interpreted to include an identification image capable of specifying predetermined contents (products) such as barcodes in an equivalent range.

Since the one-handed book is an output for marketing/promotion created by extracting some contents or reviews of the book, the customer reads the corresponding one-handed book and then decides whether to purchase the recommended content.

When a customer wants to purchase the recommended content, he/she scans the QR code using the QR scanner of his or her customer terminal (2), and goes to the recommended content introduction page of the Internet shopping mall (including book shopping malls, book shopping apps, etc.) can connect

Alternatively, by scanning the QR code with a scanner connected to a payment-related device such as a POS machine in a designated space (bookstore), a payment process for the corresponding recommended content may be performed.

The one-handed book may be printed with the location of the recommended content in the designated space so that the customer or the staff of the designated space can easily find the location of the recommended contents.

When a purchase request message is transmitted to the payment terminal in the designated space in the purchase step (S840), capturing an image of the recommended content buyer and recording personal emotion data (hereinafter, referred to as 'second personal emotion data recording step') In ;S850), it is also possible to improve the accuracy of whether the recommended content is suitable for a person feeling any emotion by identifying the emotion of the customer who purchases the recommended content and reflecting it in the corresponding recommended content.

To this end, based on the accumulated personal emotion data, updating the weight for each emotion of the recommended content (S860); may include.

In general, people who move through bookstores or department stores show relatively passive emotional expressions (expressions, etc.) compared to the case where there is a counterpart of emotional expression.

Accordingly, when the emotion of people moving in a designated space is identified based on the emotion model learned by the data actively expressing expressions such as joy and sadness, the accuracy of emotion identification may be low.

In one embodiment of the present invention, by generating an emotion model based on passive emotion expression data, excluding these active emotion expression (facial expression) data, the emotions of people flowing in public places such as bookstores and department stores more accurately can figure it out

In order to determine the emotional state from the images, generating a first emotion model from a predetermined first emotion image set (S1); a plurality of emotion images are used as training data for generating an emotion model, and the above-described emotion images are used. Image data including both active and passive emotional expressions are referred to as a 'first emotion image set'.

Then, in the step (S2) of classifying strong emotional expressions and weak emotional expressions for each predetermined emotion category from the first emotion model, the emotional index is calculated to be relatively high for each emotion category such as joy, sadness, tension, and excitement. We classify strong emotional expressions that produce high levels of emotion and weak emotional expressions that have a relatively low emotional index. The value of the emotional index for distinguishing the strong emotional expression from the weak emotional expression is not separately designated, and can be classified by a general classification algorithm.

In the step (S3) of generating a second emotion image set by excluding emotion images corresponding to strong emotional expressions from the first emotion image set, the strong emotion expressions are classified from the first emotion image set according to the above classification process. The second emotion image set excluding the classified emotion image data may be filtered. When the same image data file exists in duplicate, problems such as data processing load and storage space may occur. Thus, 'generating a second emotion image set' does not mean copying the same image.

According to an embodiment, the group emotion data may be grasped based on the second emotion model, including generating a second emotion model from the second emotion image set (S4).

The first emotion model was created to filter image data corresponding to strong emotional expressions, and the second emotion model was created to more accurately classify emotions such as joy, sadness, tension, and excitement for relatively weak emotional expressions. It became.

In addition, the accumulated personal emotion data is: in the initial personal emotion data construction step (S900) performed before the first personal emotion data recording step (S820) and the second personal emotion data recording step (S850) are performed. can be built according to

The individual emotion data can also be used to identify group emotion data.

Such personal emotion data should be used based on arbitrarily input and collected data until the method, system, and apparatus according to an embodiment of the present invention are serviced for a certain period of time or more and data collected during actual service is accumulated. Therefore, the initial personal emotion data construction step (S900) for improving the accuracy of these arbitrary data will be described in detail as follows.

In the step of sending a predetermined movie scene to the tester in a state where the tester is wearing the pulse moisture sensor (S911), in order to measure body changes in specific emotions for each tester, a movie scene that induces a specific emotion in a person send them out

Thereafter (during transmission of a movie scene), a step (S912) of measuring the tester's skin conductance, oxygen saturation, and pulse wave is performed.

The pulse rate moisture sensor may include an electrode for measuring skin conductivity and a PPG type optical blood flow measurement sensor for measuring oxygen saturation and pulse wave. This pulse moisture sensor communicates with a PC (central processing unit) built into the kiosk 5 through RS232 (serial communication), and the sensor transmits a value in bit units to the PC through serial communication, and the PC transmits each bit Converts to a real number type and processes the signal value as data. These processes can be handled by utilizing the pywin32 and pySerial libraries.

Heart-related vital signs such as heart rate and electrocardiogram are generally used to analyze changes in the sympathetic and parasympathetic autonomic nervous system. Skin conductance is used to measure the response of the autonomic nervous system according to the change index of sweat glands. Oxygen saturation is used to measure a stress index by checking blood flow through an optical blood flow sensor. Bio-signal measurement using these three is a simple and useful method for measuring the autonomic nervous system and identifying emotional states because it is relatively simple to measure and has high repeatability compared to other methods (brain wave, behavior, iris). For quick access by users who do not have time, in an embodiment of the present invention, bio-signal measurement is performed based on pulse waves and electrocardiograms.

Inputting the tester's skin conductance, oxygen saturation, and pulse wave into a Support Vector Machine (SVM) (S913); processing the pulse wave data, skin conductance data, and oxygen saturation data, extracting features from them, and SVM Enter in

The process of processing pulse wave data and extracting features is as follows.

① Type of pulse wave

The rise and fall of the pulse wave is determined by the shape of the pulse wave. The shape of the pulse wave can be largely defined by two factors: regularity and speed. If the regularity of the pulse wave is small and fast, it is treated as ‘swelling pulse’, and if the regularity of the pulse wave is large and slow, it is treated as ‘sive pulse’. When the speed and regularity are standardized and divided, the closer to 1 is treated as ‘swelling pulse’, and the closer to 0, ‘sive pulse’.

② speed

It means the change in the size of the peak interval of the pulse wave. This is interpreted the same as the amount of change in pulse rate.

에서

는 맥파에서 첨두치가 발생한 상대적인 시간을 나타낸다.

at

represents the relative time at which the peak occurred in the pulse wave.

③ Pulse wave rule diagram

The smoothness of the pulse wave is determined using the regularity of pulse wave generation.

식을 통해, 맥파 주기 변화율이 클 경우 맥파는 불규칙하게 형성되어 ‘활맥’으로, 반대로 주기변화가 너무 작을경우 ‘색맥’으로 분류 처리한다.

According to the equation, when the rate of change of the pulse wave period is large, the pulse wave is irregularly formed and classified as a 'stride vein'.

은 측정한 심박수 간격의 평균값, N은 추출된 맥파 첨두치)(

is the average value of the measured heart rate intervals, N is the peak value of the extracted pulse wave)

The process of processing skin conductance data and extracting features is as follows.

① humidity level

Humidity is calculated in a standardized size to minimize the impact on the environment and sensing method.

, 샘플링 신호의 평균은

, 표준편차는 δ)(The ith sampled data is

, the average of the sampled signals is

, the standard deviation is δ)

② rate of change

The rate of change refers to the degree of change in the humidity level over time.

는 i번째의 조습도, 총 샘플수는 N)(

is the i-th humidity level, the total number of samples is N)

The process of processing oxygen saturation data and extracting features is as follows.

Blood oxygen saturation (SpO2) is a signal that indicates the oxygen content present in hemoglobin among blood components.

(AC component is the amount of light absorbed by blood passing through peripheral blood vessels, and DC component is the amount of light absorbed by tissues such as muscle, epidermis, and bone)

Based on the predicted emotion specified for each movie scene, an error rate is calculated by dividing the number classification extracted by the support vector machine by the number of misclassifications, and the gamma value (γ) and coefficient (c) of the kernel function applied to the support vector machine ) and dimension (d) to calculate each error rate (S914); By adjusting the gamma value (γ), coefficient (c), and dimension (d) of the kernel function, using a support vector machine to which a selected kernel function is applied among a plurality of designated kernel functions, joy, sadness, tension, and excitement generating third personal emotion data for the four emotions (S915); and recognizing the tester's emotion based on the third personal emotion data, displaying the tester's emotion on a predetermined display, and receiving feedback from the tester as to which emotion among joy, sadness, tension, and excitement corresponds thereto (S916); can include

In the step of receiving the feedback (S916), the emotion recognized by the customer may be different from the actual emotion, and since clicking is possible at random, the emotion that differs too greatly from the data obtained through the initial classifier design through emotion induction In the case of state data, the corresponding feedback can be excluded.

The process of the above initial personal emotion data construction step (initial emotion classifier; S910; S911 to S916) is as shown in FIG. 4 .

A Support Vector Machine (SVM) is one of the fields of machine learning and is a supervised learning model for pattern recognition and data analysis, and is mainly used for classification and regression analysis. A set of data belonging to one of the two categories is given When the data is lost, the SVM algorithm creates a non-probabilistic binary linear classification model that determines which category the new data belongs to based on the given data set.The classification model created is expressed as a boundary in the space where the data is mapped. It is an algorithm to find the boundary with the largest width among them. SVM can be used in nonlinear classification as well as linear classification. For nonlinear classification, it is necessary to map the given data into a high-dimensional feature space, and to do this efficiently, a kernel You can also use tricks.

In emotion identification according to an embodiment of the present invention, SVM is used to maximize the generalization ability between categories as well as to minimize the error rate. SVM was used because it is easy to classify data with high-dimensional mapping of data and has higher performance compared to other classifiers. The characteristics of bio-signals acquired through emotional sensing were used as inputs to the SVM classifier. In addition, in order to improve the separability of the decision curve showing a complex shape, the kernel function that converts the mapping to a higher dimension is used through the test as follows.

The kernel function candidates are

(1) Linear Function:

(2) Polynomial Function:

(3) Radial Basis Function:

(4) Sigmoid Function:

Here, u is the test set, v is the support vector, γ is the gamma value that controls the degree, c is the coefficient, and d is the dimension. Therefore, the optimal kernel function can be selected by adjusting γ, c, and d.

Based on the kernel function, adjust γ, c, and d, and select the SVM fit function that is optimal for the experimental data. At this time, all of the extracted biometric features are used as the input.

The accuracy of each kernel function is shown in Table 1 below.

Kernel Function accuracy Linear Function 72.97% Polynomial function 73.42% Radial Basis Function 77.03% Sigmoid Function 72.52%

The above table summarizes the maximum performance when using each kernel function in (1) ~ (4).

The radial basis kernel function showed the best performance of 77.03%. In one embodiment of the present invention, the radial basis function = 1 showing the highest accuracy of 77.03% was used as a kernel function.

Thereafter, the emotional data of the user may be classified (predicted) based on the pulse wave, skin conductance, and oxygen saturation data characteristics.

Hereinafter, steps (S900; S920 to S929) of constructing initial personal emotion data from image data collected in large quantities without a separate sensor will be described. The image data-based initial personal emotion data building step (S900; S920 to S929) may be used in parallel with the sensor-based initial personal emotion data building step (S900; S911 to S916) described above.

The initial personal emotion data construction step (S900) includes: acquiring a predetermined personal video of the examiner (S920);

The 'predetermined personal video filmed by the tester' refers to the collected image data, and when used in parallel with the sensor-based initial personal emotion data construction step (S900; S911 to S916), the personal video of the tester provided by the tester. can be data.

In addition, extracting a video image of a first view, which is the last view of the personal video, as a first image from the personal video (S921); dividing an area occupied by the tester's face in the first image into a first area, and extracting a 1-1 image by dividing a portion of the first image where the first area is located (S922); As a result of reproducing and analyzing the personal video in reverse chronological order from the first viewpoint, if it is confirmed that the tester's face does not deviate from the first region at the second viewpoint, the second viewpoint to the first viewpoint setting to 1 interval (S923); Recording fourth personal emotion data to determine the emotion for the personal video of the first section based on the 1-1 image (hereinafter, referred to as 'fourth personal emotion data recording step'; S924); extracting a video image of a second viewpoint from the personal video as a second image (S925); dividing an area occupied by the tester's face into a second area in the second image, and extracting a 2-1 image by dividing a portion of the second image where the second area is located (S926); After the step of extracting the 2-1 image, as a result of reproducing and analyzing the personal video in reverse chronological order from the second viewpoint, if it is confirmed that the tester's face is not out of the second area at the third viewpoint, Setting a period from the third time point to the second time point as a second interval (S927); Recording fifth personal emotion data for determining the emotion for the personal video in the second section based on the 2-1 image (hereinafter, referred to as 'fifth personal emotion data recording step'; S928); Comparing the emotion identified in the fourth personal emotion data recording step with the emotion identified in the fifth personal emotion data recording step, and recording personal emotion data classified as a weaker emotional expression as initial personal emotion data (S929) ; may be further included.

Through the above process, relatively strong emotional expressions and relatively weak emotional expressions can be classified from image data collected from individuals, and weak emotional expressions can be induced to be learned.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

1 : Main server
2: Customer terminal
3: image recognition device
4 : Database server
5 : Kiosk
S100: Step of taking a video
S200: Recognizing multiple people
S300: Step of collecting behavioral data
S400: step of simultaneously grasping collective emotion data
S500: Step of collecting weather data
S600: Step to select recommended content
S700: Step of recommending recommended content
S800: Determining whether to receive a request for outputting recommended content
S810: step of outputting a one-handed book
S820: First Personal Emotion Data Recording Step
S830: Step to determine whether to scan the QR code
S840: purchase stage
S841: Step to move to the recommended content purchase page of the Internet shopping mall
S842: Step of sending a purchase request message to the payment terminal in the designated space
S850: Second Personal Emotion Data Recording Step
S860: Updating the weight for each emotion of the recommended content
S900: initial personal emotion data construction step
S911: step of transmitting movie scenes
S912: step of measuring tester's skin conductivity, oxygen saturation and pulse wave
S913: Input to Support Vector Machine (SVM)
S914: Calculating the error rate of the kernel function applied to the support vector machine
S915: Step of generating third personal emotion data
S916: step of receiving feedback
S920: Acquisition of personal video
S921: Step of extracting a video image of a first viewpoint as a first image
S922: Step of extracting the 1-1st image
S923: Step of setting the second time point to the first time point as the first interval
S924: 4th Personal Emotion Data Recording Step
S925: Step of extracting a video image of a second viewpoint as a second image
S926: Step of extracting the 2-1st image
S927: Step of setting the third time point to the second time point as the second interval
S928: 5th Personal Emotion Data Recording Step
S929: Step of recording personal emotion data classified as weaker emotional expression as initial personal emotion data
S1: Step of generating a first emotion model
S2: Step of classifying strong emotional expressions and weak emotional expressions
S3: generating a second emotion image set
S4: Step of generating a second emotion model

Claims (3)

photographing an image with an image recognition device installed in a space where a plurality of people move (hereinafter, referred to as a 'designated space');
Recognizing a plurality of people from a photographed image;
Simultaneously grasping the recognized emotional states of a plurality of people (hereinafter, referred to as 'group emotion data');
Collecting current weather data;
selecting recommended content corresponding to the designated space based on the group emotion data and current weather data; and
Including; recommending the recommended content for the designated space;
The recommended content is: selected as a predetermined book,
When a request for outputting recommended content is received (input) to a predetermined kiosk disposed in the designated space, outputting a one-handed book extracted from a part of the recommended content;
Recording personal emotion data by measuring skin conductivity, oxygen saturation, and pulse wave of the person printing the one-handed book through the pulse rate moisture sensor provided in the kiosk (hereinafter, referred to as 'first personal emotion data recording step') );
a purchase step of moving to a recommended content purchase page of an Internet shopping mall or sending a purchase request message to a payment terminal in the designated space when the QR code printed on the one-handed book is scanned;
When a purchase request message is transmitted to the payment terminal in the designated space in the purchase step, capturing an image of the recommended content buyer and recording personal emotion data (hereinafter, referred to as 'second personal emotion data recording step'); and
Based on the accumulated personal emotion data, updating the weight for each emotion of the recommended content; artificial intelligence-based place-linked face recognition information generation method including delete The method of claim 1,
generating a first emotion model from a predetermined first emotion image set to determine an emotional state from the images;
classifying strong emotional expressions and weak emotional expressions for each predetermined emotion classification from the first emotion model;
generating a second emotion image set by excluding emotion images corresponding to strong emotional expressions from the first emotion image set; and
Generating a second emotion model from the second emotion image set; Including,
Artificial intelligence-based place-linked face recognition information generation method for identifying the group emotion data based on the second emotion model

Images