KR20200010680A - Automated Facial Expression Recognizing Systems on N frames, Methods, and Computer-Readable Mediums thereof - Google Patents

Abstract

The present invention relates to an emotion recognition system performing emotion recognition of a target based on N frames using a machine learning model, a method thereof, and a computer-readable medium thereof. According to one embodiment of the present invention, a natural language generation system comprises: a reference frame extraction unit deriving a reference frame of a target; a feature information extraction unit extracting N first feature information for the N frames and second feature information for the reference frame; and an emotion determination unit deriving the emotion result of the target by a learned circulatory neural network model based on the N first feature information and the second feature information.

Description

Automated Facial Expression Recognizing Systems on N frames, methods, and Computer-Readable Mediums Technical} that uses a machine learning model to perform object recognition based on N frames.

The present invention relates to an emotion recognition system, a method, and a computer-readable medium for performing an object recognition based on N frames using a machine learning model, and more particularly, to a user using a machine learning model. It is a technology that limits the influence of individual characteristics and recognizes the user's emotion from the images that are continuously input as time passes.Emotion recognition is performed based on N frames using a machine learning model. Systems, methods, and computer-readable media.

Conventionally, there has been a technique for recognizing the emotion of an object from the facial expression change of a person who is the object of image information by using the learned artificial neural network technology.

However, in the related art, the inherent characteristics and appearance of each person who is an object of image information may interfere with recognition of the emotion of the object. For example, since facial expressions and changes are different according to race and gender, basic or neutral facial expressions may be different, and since the intensity of facial expressions is different for each individual, artificial neural judgments may be inaccurate. .

In addition, since the pose may change depending on the angle of the face in the image information, and the influence of light at the time of image capturing may be different for each image, it may be difficult to determine the emotion of the object with a small number of frames. Can be.

Therefore, in recognizing the emotion of the object by using the learned artificial neural network technology, there is a problem in that a large amount of image information must be learned and input of the object to be discriminated to recognize the emotion of the object. .

Therefore, there is a demand for a new technology capable of recognizing the emotions of an object with a small amount of image information while minimizing the influence of the environment, such as personal appearance of the object, and light.

An object of the present invention is to use a machine learning model to limit the influence of the user's individual characteristics, and to recognize the user's emotions from the images that are continuously input over time, N machine using the machine learning model An emotion recognition system, a method, and a computer-readable medium for performing an emotion recognition of a subject based on a frame are provided.

In order to solve the above problems, in one embodiment of the present invention, as an emotion recognition system for performing the object recognition based on the N frames using a machine learning model, a reference for deriving the reference frame of the object Frame extracting unit; A feature information extracting unit for extracting N first feature information for the N frames and second feature information for the reference frame; And an emotion discrimination unit configured to derive an emotion result of the object based on the trained circulatory neural network model based on the N first feature information and the second feature information.

In one embodiment of the present invention, the reference frame extractor may derive one of the N frames as a reference frame.

In an embodiment of the present invention, the feature information extracting unit may include: a first feature information extracting unit extracting the N first feature information from the N frames; And the second feature information extracting unit extracting second feature information from the reference frame.

In an embodiment of the present invention, the first feature information extractor extracts the N first feature information from the N frames using the learned convolutional neural network model, and the second feature information extractor learns the convoluted learner. The second feature information may be extracted from the reference frame by using a neural network model.

In one embodiment of the present invention, the emotion discrimination unit, emotion sequence information circulatory neural network module for extracting emotion sequence information based on the N first feature information using the learned cyclic neural network model; A reference sequence information cyclic neural network module for extracting reference sequence information based on the second feature information using the learned cyclic neural network model; And an emotion result deriving unit for deriving an emotion result of the target based on the emotion sequence information and reference sequence information.

In one embodiment of the present invention, the emotion sequence information circulatory neural network module, the N first feature information is sequentially input to a plurality of LSTM units, using the learned cyclic neural network model of the LSTM method, the first N pieces of emotion sequence information may be extracted based on the feature information.

In one embodiment of the present invention, the reference sequence information circulatory neural network module, the second feature information is input to a plurality of LSTM units, respectively, by using the learned cyclic neural network model of the LSTM method, and the second feature information Based on the sequence information can be extracted.

In one embodiment of the present invention, the emotion sequence information circulatory neural network module, the N first feature information is sequentially input to a plurality of LSTM units, using the learned cyclic neural network model of the LSTM method, the first Extracting N pieces of emotion sequence information based on the feature information, and the second sequence information is input to the plurality of LSTM units, respectively, in the reference sequence information cyclic neural network module using a learned cyclic neural network model of the LSTM method. N reference sequence information may be extracted based on the second feature information, and the emotion result extracting unit may derive the emotion result of the target based on the N emotion sequence information and the N reference sequence information.

In an embodiment of the present disclosure, the emotion result deriving unit may derive the emotion result of the target by subtracting the N emotion sequence information corresponding to each of the N reference sequence information from the N reference sequence information. have.

In one embodiment of the present invention, the plurality of LSTM units of the emotion sequence information cyclic neural network module and the corresponding plurality of LSTM units of the reference sequence information cyclic neural network module may be learned by common learning data.

In order to solve the above problems, in one embodiment of the present invention, using the machine learning model implemented by a computing device including at least one processor and at least one memory to perform the object recognition based on N frames An emotion recognition method comprising: a reference frame extracting step of deriving a reference frame of the object; A feature information extraction step of extracting N first feature information for the N frames and second feature information for the reference frame; And an emotion discrimination step of deriving an emotion result of the object based on the trained circulatory neural network model based on the N first feature information and the second feature information.

In an embodiment of the present invention, the feature information extraction step may include: a first feature information extraction step of extracting the N first feature information from the N frames; And extracting second feature information from the reference frame, wherein the second feature information extracting step comprises extracting second feature information from the N frames using the learned convolutional neural network model. In the extracting of the first feature information and extracting the second feature information, the second feature information may be extracted from the reference frame using a trained convolutional neural network model.

In an embodiment of the present invention, the emotion discriminating step may include: an emotion sequence information extracting step of extracting emotion sequence information based on the N first feature information using the learned cyclic neural network model; A reference sequence information extracting step of extracting reference sequence information based on the second feature information using the learned cyclic neural network model; And an emotion result deriving step of deriving an emotion result of the target based on the emotion sequence information and reference sequence information.

In one embodiment of the present invention, in the emotion sequence information extraction step, the N first feature information is sequentially input to a plurality of LSTM units using a learned cyclic neural network model of the LSTM method, and the first feature Extracting the N pieces of emotion sequence information based on the information and extracting the reference sequence information, the second feature information is input to a plurality of LSTM units, respectively, by using a trained cyclic neural network model of the LSTM method. The N reference sequence information may be extracted based on the two feature information, and in the emotion result deriving step, the emotion result of the target may be derived based on the N emotion sequence information and the N reference sequence information.

In order to solve the above problems, in one embodiment of the present invention, as a computer-readable medium for implementing an emotion recognition method for performing the emotion recognition of the object based on N frames using a machine learning model, The computer-readable medium stores instructions for causing a computing device to perform the following steps, the steps comprising: extracting a reference frame for deriving a reference frame of the object; A feature information extraction step of extracting N first feature information for the N frames and second feature information for the reference frame; And an emotion discrimination step of deriving an emotion result of the object based on the learned cyclic neural network model based on the N first feature information and the second feature information.

According to an embodiment of the present invention, the emotion of the target included in the image information may be recognized using the learned artificial neural network model.

According to an embodiment of the present invention, the emotion of the object included in the image information may be classified according to a set class using the trained artificial neural network model.

According to an embodiment of the present invention, feature information may be extracted for each consecutive frame using a convolutional artificial neural network model.

According to an embodiment of the present invention, feature information on the intrinsic characteristic of the object may be extracted using the convolutional artificial neural network model from the extracted reference frame.

According to an embodiment of the present invention, by performing the operation of the learned artificial neural network model using the characteristic information on the unique characteristics of the object, there is an effect of reducing the amount of image information necessary to recognize the emotion of the object.

According to an embodiment of the present invention, by performing the operation of the trained neural network model using the characteristic information on the unique characteristics of the target, there is an effect of reducing the steps required to recognize the emotion of the target.

According to an embodiment of the present invention, by performing the operation of the learned artificial neural network model using the characteristic information on the characteristic of the characteristic of the object, to minimize the effect of the environment, such as personal characteristics, such as the appearance of the object, light It works.

1 schematically illustrates an operating environment of an emotion recognition system that performs emotion recognition of a subject according to an embodiment of the present invention.
2 schematically illustrates an internal configuration of an emotion recognition system according to an embodiment of the present invention.
3 exemplarily illustrates an operation of a reference frame extractor according to an embodiment of the present invention.
4 schematically illustrates a process of deriving an emotion result based on the first feature information and the second feature information according to an embodiment of the present invention.
5 exemplarily illustrates operations of the first feature information extractor and the second feature information extractor using the convolutional neural network model according to an exemplary embodiment of the present invention.
6 exemplarily illustrates a detailed operation of the convolutional neural network model according to an embodiment of the present invention.
7 exemplarily illustrates an operation of the emotion discrimination unit according to an embodiment of the present invention.
8 exemplarily shows characteristic information according to a unique characteristic of an object according to an embodiment of the present invention.
FIG. 9 exemplarily shows a performance comparison result of an emotion recognition system that performs an emotion recognition of a target based on N frames using a machine learning model according to an embodiment of the present invention.
10 exemplarily illustrates an internal configuration of a computing device according to an embodiment of the present invention.

In the following, various embodiments and / or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it will also be appreciated by one of ordinary skill in the art that this aspect (s) may be practiced without these specific details. The following description and the annexed drawings set forth in detail certain illustrative aspects of the one or more aspects. However, these aspects are exemplary and some of the various methods in the principles of the various aspects may be used and the descriptions described are intended to include all such aspects and their equivalents.

In addition, various aspects and features will be presented by a system that may include a number of devices, components, and / or modules, and the like. The various systems may include additional devices, components, and / or modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in connection with the drawings. It must be understood and recognized.

As used herein, “an embodiment”, “an example”, “aspect”, “an example” and the like may not be construed as any aspect or design described being better or advantageous than other aspects or designs. . The terms '~ part', 'component', 'module', 'system', 'interface', etc. used generally mean a computer-related entity. For example, hardware, hardware And a combination of software and software.

The terms "comprises" and / or "comprising" also mean that the feature and / or component is present, but exclude the presence or addition of one or more other features, components and / or groups thereof. It should be understood that it does not.

In addition, terms including ordinal numbers such as first and second may be used to describe various components, but the components are not limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

In addition, in the embodiments of the present invention, unless otherwise defined, all terms used herein including technical or scientific terms are generally understood by those skilled in the art to which the present invention belongs. Has the same meaning as Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and ideally or excessively formal meanings, unless explicitly defined in the embodiments of the present invention. Not interpreted as

The present invention relates to an emotion recognition system for performing an object recognition based on N frames using a machine learning model. The present invention relates to a machine learning model for limiting the influence of an individual characteristic of a user, Accordingly, the emotion of the user may be recognized from the images continuously input.

Hereinafter, in order to perform such emotion recognition, an emotion recognition system, a method, and a computer-readable medium for performing the emotion recognition of a target using a machine learning model will be described.

1 schematically illustrates an operating environment of an emotion recognition system that performs emotion recognition of a subject according to an embodiment of the present invention.

According to an embodiment of the present invention, as shown in (A) of FIG. 1, the emotion recognition system 100 derives an emotion result for the object based on the image information of the object capable of recognizing the emotion. can do.

In detail, the image information may be a dynamic image of the object including a facial expression change of the face to recognize the emotion of the object, or a plurality of static images of the object continuously recorded at a predetermined period. The image information should be interpreted broadly, including having two or more image frames. Such image information may be image information alone or image information including images and sounds.

The emotion recognition system 100 may derive an emotion result for the object based on the image information.

The emotion result for the object is classified into one or more of a dataset from which emotion information can be extracted, for example, a vector dataset, or one or more of a plurality of predetermined classes by recognizing the emotion of the object. May include results. In another embodiment of the present invention, an additional error compensation algorithm may be applied to the data set from which the emotion information can be extracted. That is, the emotional result should be interpreted as a concept including the emotional result itself or an intermediate result for deriving the emotional result.

According to an embodiment of the present invention, the predetermined plurality of classes may include one or more of anger, displeasure, fear, happiness, sadness, and surprise.

However, the emotion result for the object may include not only the plurality of preset classes, but also the intensity of each of one or more emotions included in the preset plurality of classes.

Specifically, according to one embodiment of the present invention, the strength of the emotion can be expressed even in happiness, which is one of the predetermined plurality of classes extracted from the target image information.

For example, when the emotion result of the object extracted from the image information of the object is detected as happiness, the emotion of happiness of the object may also have emotional intensity (size). The intensity of the emotion of the subject can be expressed by setting 1 as the time when the emotional intensity of happiness of the subject derived from the image information of the subject is 1 and 10 when the emotional intensity of the subject happiness is the highest. have. Through this, the emotion result for the object expressing the minute expression difference may be derived based on the image information of the object.

To this end, in learning various types of machine learning models for deriving the emotion result based on the image information as described below, image information including a label or an image labeled with a target emotion. Based on the learning progress.

That is, cosine similarity with other images or frames included in the image information may be calculated based on a labeled image or a frame with respect to the object's emotion, and each learning intensity may be obtained by calculating cosine similarity.

As a result, not only the predetermined plurality of classes using the various types of machine learning models learned by the images or frames included in the image information including the emotional intensity, but also the predetermined plurality of classes. An emotion intensity of each of one or more emotions included in the expression may be expressed.

Also, based on this, unlike the conventional technology, the image information from the neutral expression to the intermediate expression change is different from the neutral expression to the image information including all images or frames for which the object expression change is completed. Based on the above, the emotional result can be derived even with a small frame.

In addition, the emotion recognition system 100 may derive the emotion result based on the image information of the object using various types of machine learning models constituting the emotion recognition system 100.

Such a machine learning model is learned with the learned data, and then can be used to implement the automated emotion recognition system 100.

However, the emotion recognition system 100 may be implemented in a different manner from the prior art by using a machine learning model, so that the emotion result for the object may be more efficiently compared to the prior art.

In FIG. 1A, the emotion result is derived from the image information. In detail, the emotion recognition system 100 derives a reference frame from the image information, and derives an emotion result by compensating an intrinsic characteristic such as the appearance of the object or an external characteristic such as an environment based on the reference frame. In this way, more accurate emotion recognition can be performed.

Alternatively, as shown in FIG. 1B, reference frame information may be input to the emotion recognition system together with the image information.

In this case, a reference frame for compensating for intrinsic characteristics such as the external appearance of the target or external characteristics such as the environment is extracted from the reference frame information and / or image information, thereby intrinsic characteristics such as the external appearance of the target, or external characteristics such as the environment. By compensating for the derivation of emotion results, more accurate emotion recognition can be performed.

As the reference frame, for example, a frame having a neutral expression may be selected from a frame having a neutral expression or a part of a frame included in the image information or a frame included in the image information.

The reference frame information may include the reference frame itself or information for deriving a reference frame, or information for deriving a reference frame among frames of the image information, or the appearance and environment of the object. It may include information about. Here, the information on appearance may include race, sex, age, face color, face shape, emotion intensity, neutral expression, and the like, and the environmental information may include light quantity, image related information, and pose.

By implementing the emotion recognition system 100 in a way to limit the appearance and environment of the object based on the feature information, compared to the prior art by using a smaller amount of image information than the accurate result of the target emotion Can be derived.

2 schematically illustrates an internal configuration of an emotion recognition system according to an embodiment of the present invention.

The emotion recognition system according to the above embodiment may be implemented by one or more processors and one or more memories by a computing device.

Such a computing device may include a processor A, a bus (corresponding to a double-headed arrow between a processor, a memory, a network interface), a network interface B, and a memory C. The memory C may store artificial neural network learning data used in the inference or prediction module of the present invention described below as learning data learned in implementing an OS and an artificial neural network. Alternatively, the artificial neural network learning data may mean modeling information itself in which deep learning is performed. In the processor A, the reference frame extractor 1000, the feature information extractor 2000, and the emotion discriminator 3000 may be executed. In other embodiments the emotion recognition system may include more components than the components of FIG. 2.

The memory is a computer-readable recording medium, and may include a permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Such software components may be loaded from a computer readable recording medium separate from the memory using a drive mechanism (not shown). Such a separate computer-readable recording medium may include a computer-readable recording medium (not shown), such as a floppy drive, disk, tape, DVD / CD-ROM drive, memory card, and the like. In other embodiments, the software components may be loaded into the memory via the network interface B rather than the computer readable recording medium.

The bus may enable communication and data transfer between components of the computing device. The bus may be configured using a high-speed serial bus, a parallel bus, a storage area network and / or other suitable communication technology.

The network interface B may be a computer hardware component for connecting a computing device implementing the emotion recognition system to a computer network. The network interface B may connect the emotion recognition system to the computer network through a wireless or wired connection.

The processor A may be configured to process instructions of a computer program by performing input / output operations that implement the basic arithmetic, logic, and emotion recognition system 100. Instructions may be provided to the processor by memory C or network interface B and via a bus. The processor may be configured to execute program codes for the reference frame extractor 1000, the feature information extractor 2000, and the emotion discriminator 3000. Such program code may be stored in a recording device such as a memory.

The reference frame extractor 1000, the feature information extractor 2000, and the emotion discriminator 3000 may be configured to perform an emotion recognition method which will be described below. According to the emotion recognition method, some components may be omitted, additional components not shown may be included, or two or more components may be combined.

On the other hand, such a computing device preferably corresponds to a personal computer or a server, and in some cases, a smart phone, a tablet, a mobile phone, a video phone, an e-book reader (e) -book reader, desktop PC, laptop PC, netbook PC, personal digital assistant (PDA), portable A portable multimedia player (PMP, hereinafter referred to as 'PMP'), an mp3 player, a mobile medical device, a camera, a wearable device (eg, head-mounted) Head-mounted device (HMD), for example referred to as 'HMD', electronic clothing, electronic bracelet, electronic necklace, electronic accessory, electronic tattoo, or smart watch ) And the like.

That is, the emotion recognition system 100 that performs the emotion recognition of the object based on the N frames using the machine learning model according to the embodiment of the present invention includes a reference frame extracting unit (derived) to derive the reference frame of the object ( 1000); A feature information extracting unit 2000 for extracting N first feature information for the N frames and second feature information for the reference frame; And an emotion discriminating unit 3000 that derives an emotion result of the object based on the trained cyclic neural network model based on the N first feature information and the second feature information.

The reference frame extractor 1000 may derive the reference frame of the target. That is, as described above, the reference frame derived from the reference frame extracting unit 1000 is input to other components of the emotion recognition system 100 to provide characteristic information about the characteristic of the object such as its appearance or environment. It is the basis for the extraction. The extracted feature information is then used to compensate for the result derived by the artificial neural network. Such a reference frame may be selected from a frame having a neutral expression or a frame estimated to have a neutral expression among some of the frames included in the image information or a frame included in the image information. Alternatively, a frame reconstructed from reference frame information provided by the user may be selected.

The feature information extractor 2000 may extract N first feature information for the N frames and second feature information for the reference frame. That is, the N first feature information may be derived by the feature information extractor 2000 based on the N frames for the object, and the second feature information is based on the reference frame. It may be derived by the information extraction unit 2000.

The N first feature information and the second feature information may be input to the emotion discrimination unit 3000.

The emotion discrimination unit 3000 may derive an emotion result of the object by using a circulatory neural network model trained based on the first feature information of the N and the second feature information.

3 exemplarily illustrates an operation of a reference frame extractor according to an embodiment of the present invention.

According to the embodiment, as shown in (a) of FIG. 3, the reference frame extractor 1000 may derive the reference frame of the object. In more detail, a reference frame may be extracted from the entire image information. For example, a frame having a specific sequence number among all the image information may be extracted as a reference frame, or a frame determined to have a low expression strength may be selected as the reference frame by determining the intensity of the expression with respect to the frame included in the entire image information. The determination of the strength of the facial expression may be performed by the learned artificial neural network model, or may be performed by rule-based determination of predetermined feature information.

Preferably, as shown in (b) of FIG. 3, the reference frame extractor 1000 may derive one of N frames to be analyzed as a reference frame.

That is, the reference frame extractor 1000 derives one of the N frames for the target as a reference frame or derives the reference frame for deriving the reference frame of the target based on the image information of the target. Image preprocessing may be performed on any one of the frames to derive the reference frame. Preferably, the reference frame extractor 1000 may derive the first frame of the N frames as a reference frame.

In addition, the reference frame extractor 1000 may extract a reference frame through reference frame information input by the user of the emotion recognition system in FIG.

The reference frame information may correspond to the reference frame itself. For example, in the emotion recognition system, the user may be informed to make a neutral expression, and the image taken when the user makes the neutral expression may be a reference frame.

Alternatively, a neutral expression frame or a frame corresponding to a predetermined reference may be set as a reference frame from image information including a plurality of frames provided by the user.

Alternatively, the reference frame information may correspond to information about the user's appearance or environment, not image information, and may be based solely on the basis of such reference frame information or based on other data (for example, image information). The frame can be derived.

 That is, as described above, the reference frame derived from the reference frame extracting unit 1000 is input to other components of the emotion recognition system 100 so as to provide characteristics of inherent characteristics such as appearance and / or environment of the object. It is the basis for extracting the information, and the extracted feature information is used to compensate for the results obtained through the cyclic neural network.

That is, the second feature information is the feature information on the inherent characteristics of the object described above, and the second feature information extracted based on the reference frame is a conventional technique for recognizing the emotion of the object using a machine learning model. For example, the present invention can be used to solve a phenomenon in which efficiency is lowered due to an inherent characteristic of an object.

4 schematically illustrates a process of deriving an emotion result based on the first feature information and the second feature information according to an embodiment of the present invention.

According to the embodiment, the feature information extractor 2000 includes: a first feature information extractor 2100 for extracting the N first feature information from the N frames; And the second feature information extractor 2200 for extracting second feature information from the reference frame.

내지

의 상기 N개의 제1특징정보를 추출할 수 있다.The first feature information extractor 2100 may extract the N first feature information from the N frames. That is, as shown in FIG. 4, the N frames of # 1 to #N are input to the first feature information extractor 2100, and the first feature information extractor 2100 is the N frames. Based on each

To

The N first feature information of may be extracted.

Preferably, the first feature information extractor 2100 may extract the N first feature information from the N frames by using the learned artificial neural network model.

That is, the first feature information extractor 2100 may be implemented by a trained neural network model, and the N frames are input to the trained artificial neural network model of the first feature information extractor 2100. The N first feature information may be extracted from each of the N frames. Alternatively, the first feature information extractor 2100 may extract first feature information from a rule-based model that extracts predetermined information related to emotions, facial expressions, etc., from an input frame other than an artificial neural network model.

의 상기 제2특징정보를 추출할 수 있다. 혹은 상기 제2특징정보추출부(2200)는 인공신경망모델이 아닌 입력된 프레임으로부터 감정, 표정 등과 관련된 기설정된 정보를 추출하는 규칙기반 모델로부터 제1특징정보를 추출할 수 있다. The second feature information extractor 2200 may extract second feature information from the reference frame. That is, as shown in FIG. 4, the reference frame derived from the reference frame extractor 1000 is input to the second feature information extractor 2200, and the second feature information extractor 2200 is Based on the reference frame

The second feature information of may be extracted. Alternatively, the second feature information extractor 2200 may extract first feature information from a rule-based model that extracts predetermined information related to emotions, facial expressions, etc., from an input frame other than an artificial neural network model.

Preferably, the first feature information extracting unit and the second feature information extracting unit are models that can be learned with common learning data or can output the same or corresponding results for the same input value.

Preferably, the second feature information extractor 2200 may extract the second feature information from the reference frame by using the learned artificial neural network model.

That is, the second feature information extractor 2200 may be implemented by a trained neural network model, and the N frames are input to the trained artificial neural network model of the second feature information extractor 2200. The second feature information may be extracted from the reference frame.

5 exemplarily illustrates operations of the first feature information extractor and the second feature information extractor using a machine-learned model according to an embodiment of the present invention.

According to the embodiment, more preferably, the first feature information extracting unit 2100 extracts the N first feature information from the N frames by using a machine-learned model, and the second feature information. The extractor 2200 may extract the second feature information from the reference frame by using a machine-learned model.

That is, the first feature information extraction unit 2100 may extract the N first feature information from the N frames by using a machine-learned model. Specifically, the first feature information extractor 2100 may extract the N first feature information, which is a feature of the object, from each of the N frames using the machine-learned model constituting the first feature information extractor 2100.

을 추출할 수 있다. 이와 마찬가지로, #2 프레임이 입력되어 이에 대한 제1특징정보인

을 추출하고, #N 프레임이 입력되어

을 추출할 수 있다.As shown in (a) of FIG. 5, the # 1 frame is input to the machine-learned model of the first feature information extracting unit 2100, and is the first feature information.

Can be extracted. Similarly, frame # 2 is inputted so that the first feature information

, The #N frame is entered

Can be extracted.

In addition, the second feature information extracting unit 2200 may extract the second feature information from the reference frame using a machine-learned model. Specifically, the second feature information, which is a feature of the object, may be extracted from the reference frame by using the machine-learned model.

가 추출될 수 있다.As shown in (b) of FIG. 5, the reference frame derived from the reference frame extracting unit 1000 is input to a machine-learned model of the second feature information extracting unit 2200, and a second corresponding thereto is obtained. Feature Information

Can be extracted.

However, the machine-learned model constituting the first feature information extracting unit 2100 and the second feature information extracting unit 2200 may be an artificial neural network such as a convolutional neural network model (CNN) and a capsule network (CapsNet). It can be implemented by a model. An artificial neural network model such as a convolutional neural network model (CNN), a capsule network (CapsNet), etc. has a feature of extracting feature information of a target from input image information.

Hereinafter, an operation process of the convolutional neural network model (CNN) constituting the first feature information extractor 2100 and the second feature information extractor 2200 will be described.

6 exemplarily illustrates a detailed operation of the convolutional neural network model according to an embodiment of the present invention.

As shown in FIG. 6, the convolutional neural network model constituting the first feature information extractor and the second feature information extractor repeats convolution and pooling operations when image information of the target is input, thereby being fully connected. The first feature information and the second feature information may be extracted by generating a layer.

As a result, the first feature information and the second feature information extracted through such a convolutional neural network model extract information about each frame of the object to be input, thereby providing information based on the operation of the emotion discriminator described later. Can be

7 exemplarily illustrates an operation of the emotion discrimination unit according to an embodiment of the present invention.

According to the exemplary embodiment, the emotion discriminating unit 3000 may include an emotion sequence information circulatory neural network module 3100 for extracting emotion sequence information based on the N first feature information using the learned cyclic neural network model; A reference sequence information cyclic neural network module 3200 for extracting reference sequence information based on the second feature information using the learned cyclic neural network model; And an emotion result extracting unit 3300 for deriving an emotion result of the target based on the emotion sequence information and reference sequence information.

The emotion sequence information circulatory neural network module 3100 may extract emotion sequence information based on the N first feature information using the learned cyclic neural network model.

The emotion sequence information is extracted from the N first characteristic information, which can be recognized by the emotion recognition system 100 or can classify the emotions of the object by a predetermined class. Information.

The N first feature information is sequentially input to each unit constituting the trained cyclic neural network model, and a result value for one input is stored and input together with the next input, such that the emotion sequence information is the next result. There is a characteristic that affects the value.

내지

의 상기 제1특징정보는 상기 감정시퀀스정보순환신경망모듈(3100)의 학습된 순환신경망모듈에 차례로 입력되고, 상기 감정결과도출부(3300)는 이를 통하여 추출된

내지

의 상기 감정시퀀스정보를 기초로 상기 대상의 감정을 인식하거나 혹은 상기 대상의 감정을 기설정된 클래스(class)에 의하여 분류할 수 있다.As shown in (a) of FIG. 7, extracted from the frames # 1 to #N.

To

The first characteristic information of is sequentially input to the learned cyclic neural network module of the emotion sequence information circulatory neural network module 3100, and the emotion result extracting unit 3300 is extracted through this.

To

The emotion of the object may be recognized or the emotion of the object may be classified based on a predetermined class based on the emotion sequence information of.

The reference sequence information cyclic neural network module 3200 may extract reference sequence information based on the second feature information by using the learned cyclic neural network model.

The reference sequence information is extracted from the reference frame, and the emotion recognition system 100 recognizes the emotion of the object or classifies the emotion of the object by a predetermined class, as described above. It can be used to limit the effects of the subject's unique characteristics.

Such reference sequence information is sequentially input to each unit constituting the circulatory neural network model trained on the second feature information, and the result value for one input is stored and input together with the next input, thereby providing the next result value. There is a characteristic that affects.

의 상기 제2특징정보는, 상기 기준시퀀스정보순환신경망모듈(3200)에 순차로 복수번 각각 입력되고, 상기 감정결과도출부(3300)는 추출된

내지

의 상기 기준시퀀스정보를 기초로 상기 대상의 고유특성에 의한 영향을 제한할 수 있다.As shown in (a) of FIG.

The second feature information of the, is sequentially input to the reference sequence information circulatory neural network module 3200 a plurality of times, respectively, the emotion result extracting unit 3300 is extracted

To

Based on the reference sequence information of the can be limited by the unique characteristics of the target.

The emotion result deriving unit 3300 may derive the emotion result of the object of which the influence by the unique characteristic of the object is limited based on the emotion sequence information and the reference sequence information.

That is, the emotion result deriving unit 3300 recognizes the emotion of the subject or sets the emotion of the subject based on the emotion sequence information extracted from the emotion sequence information circulatory neural network module 3100. By classifying, the influence of the characteristic of the subject is limited by using the reference sequence information, so that the emotion result of the subject can be derived.

As a result, the emotion result obtaining unit 3300 recognizes the emotion of the object based on the emotion sequence information and the reference sequence information, or classifies the emotion of the object into a predetermined class. Likewise, it is possible to implement an improved emotion recognition system compared to the prior art.

The emotion sequence information circulatory neural network module 3100 may sequentially input the N first feature information to one or more circulatory neural network units using the learned cyclic neural network model, based on the first feature information. N emotion sequence information can be extracted.

In addition, the reference sequence information circulatory neural network module 3200 uses the learned cyclic neural network model to input the second feature information to one or more cyclic neural network units, respectively, and based on the second feature information, the reference sequence information. Can be extracted.

That is, the N first feature information and the second feature are included in a cyclic neural network unit constituting a trained cyclic neural network model constituting the emotion sequence information circulatory neural network module 3100 and the reference sequence information circulatory neural network module 3200. Information is input in parallel, and the emotion sequence information and the reference sequence information can be extracted based on the information.

Accordingly, as described above, the emotion sequence information and the reference sequence information may be input to the above-described emotion result generator 3300 to derive the emotion result of the subject with limited influence by the unique characteristics of the subject.

More preferably, the emotion sequence information circulatory neural network module 3100 sequentially inputs the N first feature information to at least one cyclic neural network unit by using the trained cyclic neural network model, and inputs the first feature information. Extracting the N emotion sequence information based on the above, and the reference sequence information circulatory neural network module 3200 inputs the second feature information to each of the one or more circulatory neural network units using the learned cyclic neural network model, and the second N reference sequence information may be extracted based on the feature information, and the emotion result obtaining unit 3300 may derive the emotion result of the target based on the N reference sequence information and the N emotion sequence information. .

More preferably, the emotion result deriving unit 3300 may derive the emotion result of the target based on data including the N reference sequence information and the N emotion sequence information.

The trained circulatory neural network model constituting the emotion sequence information circulatory neural network module 3100 and the reference sequence information circulatory neural network module 3200 may include a recurent neural network (RNN), a long-short term memory (LSTM), and It may be implemented by a GRU (Gated Recurrent Unit).

As shown in FIG. 4, more preferably, the emotion sequence information circulatory neural network module 3100 uses the learned circulatory neural network model of the LSTM method, to each of one or more LSTM units of the learned circulatory neural network model. N first feature information may be input, and N emotion sequence information may be extracted based on the first feature information.

In addition, the reference sequence information cyclic neural network module 3200 inputs the second characteristic information to each of one or more LSTM units of the learned cyclic neural network model by using the learned cyclic neural network model of the LSTM method. Reference sequence information can be extracted based on the feature information.

That is, the N first feature information and the second feature information are input in parallel to each of at least one LSTM unit constituting the emotion sequence information cyclic neural network module 3100 and the reference sequence information cyclic neural network module 3200. Based on this, the emotion sequence information and the reference sequence information can be extracted.

Accordingly, as described above, the emotion sequence information and the reference sequence information may be input to the above-described emotion result generator 3300 to derive the emotion result of the subject with limited influence by the unique characteristics of the subject.

More preferably, the emotion sequence information circulatory neural network module 3100 sequentially inputs the N first feature information to at least one LSTM unit by using the learned cyclic neural network model of the LSTM method, and the first feature. N emotion sequence information is extracted based on the information, and the reference sequence information cyclic neural network module 3200 uses the learned cyclic neural network model of the LSTM method to input the second characteristic information to each of the one or more LSTM units. And extracting N reference sequence information based on the second feature information, and the emotion result extracting unit 3300 extracts an emotion result of the target based on the N reference sequence information and the N emotion sequence information. Can be derived.

More preferably, the emotion result deriving unit 3300 may derive the emotion result of the target based on data including the N reference sequence information and the N emotion sequence information.

More preferably, the emotion result deriving unit removes all or each of the N reference sequence information from each or all of the N emotion sequence information on the N reference sequence information and the N emotion sequence information. Through the operation of the scheme, the emotion result is derived.

The emotion result may correspond to one or more emotion result data for the N frames. The emotion result may be derived for each of the N frames, or may be a comprehensive emotion result for the N frames to be analyzed.

As described above, the emotion result may be a data set from which emotion information may be extracted, for example, a vector data set, or a plurality of classes preset by recognizing the emotion of the object. It may include the results of classification into one or more of the classes.

Preferably, as shown in (b) of FIG. 7, the emotion result deriving unit 3300 subtracts the N emotion sequence information corresponding to the reference sequence information from the N reference sequence information to determine the target. F ₁ to F _N as an emotional result of N frames can be derived. In this case, a method of subtracting by simply subtracting or performing an additional operation such as a coefficient may be applied.

As a result, by applying the corresponding N pieces of emotion sequence information from the N pieces of reference sequence information and compensating them, inherent characteristics of the object's appearance and environment are limited, resulting in improved performance compared to the prior art. have.

In another embodiment of the present invention, the emotion result derivation unit is a result derived from the N pieces of reference sequence information and the reference sequence information, for example, in F ₁ to F _N of FIG. An additional error compensation algorithm can be applied to derive the emotion result.

Preferably, the emotion sequence information circulatory neural network module 3100 and the reference sequence information circulatory neural network module 3200 are implemented as a cyclic neural network model of the same type LSTM, each corresponding LSTM unit is a common learning data Can be learned.

As a result, each of the corresponding LSTM units of the emotion sequence information circulatory neural network module 3100 and the reference sequence information circulatory neural network module 3200 learned by the common learning data, as described above, may be the emotion sequence information and the reference sequence. In extracting the information, the operation can be performed with the same weight.

That is, the second feature information corresponding to the N first feature information may be calculated by the same weight to extract the emotion sequence information and the reference sequence information.

Preferably, the LSTM units of the emotion sequence information circulatory neural network module 3100 and the reference sequence information circulatory neural network module 3200 correspond to the same or corresponding forms.

However, the emotion sequence information circulatory neural network module 3100 and the reference sequence information circulatory neural network module 3200 have the same weight based on the first feature information and the second feature information, respectively, according to the emotion sequence information and the reference sequence information. However, the state information of each LSTM unit is not shared with each other.

As a result, the emotion extracting unit 3300 may be configured based on the emotion sequence information and the reference sequence information extracted by the emotion sequence information circulatory neural network module 3100 and the reference sequence information circulatory neural network module 3200. The emotional results of the subject can be derived.

In this manner, by subtracting the corresponding N pieces of emotion sequence information from the N pieces of reference sequence information extracted by applying the same weight to a plurality of LSTM units of an LSTM trained cyclic neural network model, the subject's uniqueness The emotion of the object may be recognized in a state in which the characteristic is limited, or the emotion of the object may be classified by a predetermined class.

8 exemplarily shows characteristic information according to a unique characteristic of an object according to an embodiment of the present invention.

In the above embodiment, FIG. 8 exemplarily shows feature information extracted according to an object of the image information in the conventional emotion recognition system using the learned cyclic neural network module.

Specifically, each frame of the object is copied and input into the learned cyclic neural network model of the LSTM method. When one frame of the set of frames for the object generated by copying in this manner is input to the learned cyclic neural network model of the LSTM method, each value of the first column of FIG. 8 is 15-dimensional having a value of 0 to 1.0. It represents the value of the sequence information for the target expressed by. It can be seen that the value of such sequence information converges to one value as the number of input frames increases (Time Step). In this way, the value of the multi-dimensional sequence information that inputs and converges the frame set of the object represents the characteristic information of the unique characteristic of the object.

The value of the multi-dimensional sequence information expressing the characteristic information on the unique characteristic of the object has a different value depending on the object. That is, as shown in FIG. 8, when the frame set, which differs from the subject as a white male and a yellow female, is input to the learned circulatory neural network model of the LSTM method, convergence to different values can be confirmed.

That is, the value of the multi-dimensional sequence information may be identified as feature information according to the unique characteristics of the object having different values according to the object of the image information input to the learned cyclic neural network module.

Since such characteristic information on the intrinsic characteristic of the object degrades the performance of the conventional emotion recognition system, the present invention of the method as described above has been proposed.

FIG. 9 exemplarily shows a performance comparison result of an emotion recognition system that performs an object recognition based on N frames using a machine learning model according to an embodiment of the present invention.

), ASDF(

+

+

))이 더 적은 프레임으로도 같은 정답률을 갖는 것을 확인 할 수 있다.As shown in FIG. 9, the emotion recognition system ASDF according to the embodiments of the present invention is compared with the conventional LSTM.

), ASDF (

+

+

We can see that)) has the same percentage of correct answers even with fewer frames.

)은 하나의 대상으로 상기 감정인식시스템을 구성하는 신경망모델을 훈련시켰을 때 얻는 결과값을 나타내고, ASDF(

+

+

)은 세개의 대상으로 상기 감정인식시스템을 구성하는 신경망모델을 훈련시켰을 때 얻는 결과값을 나타낸다.In the table shown in FIG. 9, the ASDF (

) Represents the result obtained when the neural network model constituting the emotion recognition system is trained with one object.

+

+

) Represents the result obtained when training the neural network model constituting the emotion recognition system with three subjects.

Also, (a) the subject's feelings of anger, (b) the subject's feelings of displeasure, (c) the subject's feelings of fear, (d) the subject's feelings of happiness, (e) the subject's feelings of Sadness (f) shows the result when the subject's emotion is image information about anger.

10 exemplarily illustrates an internal configuration of a computing device according to an embodiment of the present invention.

As shown in FIG. 10, the computing device 11000 may include at least one processor 11100, a memory 11200, a peripheral interface 11300, and an input / output subsystem ( I / Osubsystem 11400, power circuit 11500, and communication circuit 11600. In this case, the computing device 11000 may correspond to a user terminal A connected to the emotion recognition system 100 or the computing device B described above.

The memory 11200 may include, for example, high-speed random access memory, magnetic disk, SRAM, DRAM, ROM, flash memory, or nonvolatile memory. have. The memory 11200 may include various data included in a software module, an instruction set, or a learned embedding model required for the operation of the computing device 11000.

In this case, accessing the memory 11200 from another component such as the processor 11100 or the peripheral interface 11300 may be controlled by the processor 11100.

The peripheral interface 11300 may couple the input and / or output peripherals of the computing device 11000 to the processor 11100 and the memory 11200. The processor 11100 may execute a software module or an instruction set stored in the memory 11200 to perform various functions for the computing device 11000 and process data.

Input / output subsystem 11400 can couple various input / output peripherals to peripheral interface 11300. For example, the input / output subsystem 11400 may include a controller for coupling a peripheral device such as a monitor or keyboard, a mouse, a printer, or a touch screen or a sensor, as necessary, to the peripheral interface 11300. According to another aspect, the input / output peripherals may be coupled to the peripheral interface 11300 without passing through the input / output subsystem 11400.

The power circuit 11500 may supply power to all or part of the components of the terminal. For example, power circuit 11500 may be a power management system, one or more power sources such as batteries or alternating current (AC), charging systems, power failure detection circuits, power converters or inverters, power status indicators or power sources. It can include any other components for creation, management, distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, the communication circuit 11600 may include an RF circuit to transmit and receive an RF signal, also known as an electromagnetic signal, to enable communication with other computing devices.

This embodiment of FIG. 10 is only one example of the computing device 11000, and the computing device 11000 may include some components shown in FIG. 10, or may include additional components not shown in FIG. It may have a configuration or arrangement that combines two or more components. For example, the computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor, in addition to the components illustrated in FIG. 10, and various communication schemes (WiFi, 3G, LTE) in the communication circuit 1160. , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components included in the computing device 11000 may be implemented in hardware, software, or a combination of both hardware and software, including integrated circuits specialized for one or more signal processing or applications.

Methods according to an embodiment of the present invention may be implemented in program instruction form that can be executed by various computing devices and recorded in a computer readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed in a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmitter (not shown) for transmitting the file at the request of the user terminal.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments are, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs). Can be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of these, and configure the processing device to operate as desired, or process it independently or in combination. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computing devices so that they are stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or, even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (15)

An emotion recognition system that performs an object recognition based on N frames using a machine learning model,
A reference frame extracting unit for deriving the reference frame of the target;
A feature information extracting unit for extracting N first feature information for the N frames and second feature information for the reference frame;
And an emotion discrimination unit configured to derive an emotion result of the object based on the learned circulatory neural network model based on the N first feature information and the second feature information.
The method according to claim 1,
The reference frame extraction unit,
Emotion recognition system for deriving any one of the N frames as a reference frame.
The method according to claim 1,
The feature information extraction unit,
A first feature information extraction unit for extracting the N first feature information from the N frames; And
And the second feature information extracting unit for extracting second feature information from the reference frame.
The method according to claim 3,
The first feature information extraction unit extracts the N first feature information from the N frames by using a machine-learned model,
The second feature information extraction unit extracts the second feature information from the reference frame using a machine-learned model.
The method according to claim 1,
The emotion discriminating unit,
An emotion sequence information circulatory neural network module for extracting emotion sequence information based on the N first feature information using a trained cyclic neural network model;
A reference sequence information cyclic neural network module for extracting reference sequence information based on the second feature information using the learned cyclic neural network model; And
And an emotion result deriving unit for deriving an emotion result of the object based on the emotion sequence information and reference sequence information.
The method according to claim 5,
The emotion sequence information circulatory neural network module,
Using the trained circulatory neural network model, the N first feature information is sequentially input to at least one cyclic neural network unit, and extracts the N emotion sequence information based on the first feature information.
The method according to claim 5,
The reference sequence information cyclic neural network module,
Emotion recognition system using the trained circulatory neural network model, the second feature information is respectively input to at least one cyclic neural network unit, and extracts the reference sequence information based on the second feature information.
The method according to claim 5,
The emotion sequence information circulatory neural network module,
The N first feature information is sequentially input to at least one cyclic neural network unit by using a trained cyclic neural network model, and the N emotion sequence information is extracted based on the first feature information.
The reference sequence information cyclic neural network module,
The second feature information is input to at least one cyclic neural network unit by using the learned cyclic neural network model, and N reference sequence information is extracted based on the second feature information.
The emotion result derivation unit,
And an emotion result of the object is derived based on the N emotion sequence information and the N reference sequence information.
The method according to claim 8,
The emotion result derivation unit,
And the emotion result of the object is derived by subtracting the N emotion sequence information corresponding to each of the N reference sequence information from the N reference sequence information.
The method according to claim 8,
And at least one cyclic neural network unit of the emotion sequence information circulatory neural network module and corresponding one or more cyclic neural network units of the reference sequence information circulatory neural network module are learned by common learning data.
An emotion recognition method of performing an object recognition on the basis of N frames using a machine learning model implemented by a computing device including at least one processor and at least one memory.
A reference frame extracting step of deriving the reference frame of the object;
A feature information extraction step of extracting N first feature information for the N frames and second feature information for the reference frame;
And an emotion discrimination step of deriving an emotion result of the object based on the trained circulatory neural network model based on the N first feature information and the second feature information.
The method according to claim 11,
The feature information extraction step,
A first feature information extraction step of extracting the N first feature information from the N frames; And
And a second feature information extraction step of extracting second feature information from the reference frame.
In the extracting of the first feature information, the N first feature information is extracted from the N frames by using a trained convolutional neural network model.
In the extracting of the second feature information, the emotion recognition method extracts the second feature information from the reference frame using a trained convolutional neural network model.
The method according to claim 11,
The emotion discrimination step,
An emotion sequence information extracting step of extracting emotion sequence information based on the N first feature information using the learned cyclic neural network model;
A reference sequence information extracting step of extracting reference sequence information based on the second feature information using the learned cyclic neural network model; And
And an emotion result derivation step of deriving an emotion result of the object based on the emotion sequence information and reference sequence information.
The method according to claim 13,
The emotion sequence information extraction step,
The N first feature information is sequentially input to at least one cyclic neural network unit by using a trained cyclic neural network model, and the N emotion sequence information is extracted based on the first feature information.
The reference sequence information extraction step,
The second feature information is input to at least one cyclic neural network unit by using the learned cyclic neural network model, and N reference sequence information is extracted based on the second feature information.
The emotion result derivation step,
And an emotion result of the object is derived based on the N emotion sequence information and the N reference sequence information.
A computer-readable medium for implementing an emotion recognition method of performing an object recognition on the basis of N frames using a machine learning model, wherein the computer-readable medium causes a computing device to perform the following steps. Save the instructions to be executed, the steps being:
A reference frame extracting step of deriving the reference frame of the object;
A feature information extraction step of extracting N first feature information for the N frames and second feature information for the reference frame;
And an emotion discrimination step of deriving an emotion result of the object based on the trained circulatory neural network model based on the N first feature information and the second feature information.

Images