KR102110393B1 - Emotion recognition method and computer program for excuting the method, emotion recognizer generation method and computer program for excuting the method - Google Patents

Abstract

According to an embodiment of the present invention, there is provided a method of recognizing a first user's emotion for a content to be recognized that includes at least one type of object, collecting the at least one type of object for each type of object; Generating a first feature vector for each type of object based on the collection result; Generating a first feature vector for personal information based on the general personal information of the first user and the content viewing information of the first user; Generating a second feature vector by merging the first feature vector for each type and the first feature vector for the personal information in a preset manner; And determining an emotion vector corresponding to the second feature vector by using an emotion recognizer learned based on a plurality of training data.

Description

Emotion Recognition Method and Program, and Emotion Recognizer Generation Method and Program

Embodiments of the present invention relate to a method and a program for recognizing emotions and a method and a program for generating emotion recognizers.

Due to the rapid development of information and communication technology, various types of terminals, such as mobile communication terminals and personal computers, are implemented to perform various functions.

In particular, these terminals have been developed to use various functions such as generating secondary information from the received information by developing from the conventional function of simply receiving and displaying the information.

Recently, information providers who provide information through these terminals have increased the number of users who want to use the advertisements, marketing, etc. by grasping the emotional state of the information recipient.

According to the related art, when a plurality of types of objects are included in the content, the user's feelings for the corresponding content are inferred based on only one type of object, or the emotions for each type of object are individually inferred. , This was statistically processed to infer the user's feelings about the content.

In this way, even though each type of object is all about the same subject, it lacks understanding of the content flow or context because it infers feelings for the entire content based on the results of individual emotion recognition for each type of object. There was a problem leading to the result.

In particular, this method processes individual emotion recognition results for each type of individual based on a statistical model, so even though somewhat different emotion recognition results are derived from some types of individuals, more detailed emotion recognition is ignored by ignoring these recognition results. There was an impossible problem.

The present invention is to solve the above-mentioned problems, and emotion recognition methods and programs that collect objects included in the same content for each type of object and integrate feature information of each type of collected objects at a lower level, It is intended to provide a method and program for generating an emotion recognizer.

In addition, the present invention performs emotion recognition on content based on feature information of each type of entity integrated at a lower level, and thus an emotion recognition method capable of deriving an emotion recognition result with an understanding of content flow or context. And a program and a method and program for generating an emotion recognizer.

Furthermore, it is intended to provide an emotion recognition method and program with improved accuracy and an emotion recognizer generation method and program.

According to an embodiment of the present invention, a method of recognizing a first user's emotion for a content to be recognized including at least one type of object includes: collecting the at least one type of object for each type of object; Generating a first feature vector for each type of object based on the collection result; Generating a first feature vector for personal information based on the general personal information of the first user and the content viewing information of the first user; Generating a second feature vector by merging the first feature vector for each type and the first feature vector for the personal information in a preset manner; And determining an emotion vector corresponding to the second feature vector by using an emotion recognizer learned based on a plurality of training data. In this case, the emotion recognizer may be a data set that learns a correlation between a plurality of test contents including a personal information of a viewer and at least one or more types of objects, and a plurality of second user emotion vectors for the plurality of test contents. .

When a plurality of first types of objects are included in the content to be recognized, collecting the objects may include collecting all of the plurality of first types of objects and generating a first feature vector for each type of the objects. Can generate a first feature vector reflecting the features of all the first types of objects included in the content to be recognized.

The preset method may be a method of serially merging one or more first vectors to be merged.

The personal information includes at least one of the first user's gender, age, age, education, residential area, and occupation, and the content viewing information includes when and where the first user viewed the recognized target content. , It may include at least one of how long you have enjoyed and through which route.

Generating a first feature vector for each type of object may include separating objects of the video type into objects of an image type and objects of an audio type, for the collected one or more video type objects; Generating a first feature vector for an object of the image type based on the object of the image type separated from the object of the video type and the object of at least one image type in the content to be recognized; And generating a first feature vector for an object of the audio type based on the object of the audio type separated from the object of the video type and the object of at least one audio type in the content to be recognized.

A method of recognizing a first user's emotion with respect to the recognition target content based on multimodal information included in the recognition target content according to an embodiment of the present invention includes recognition including at least one type of object Receiving target content; Generating a first feature vector for each type of object included in the content to be recognized; Generating a second feature vector, which is a feature vector of the content to be recognized, based on the first feature vector for each type; And determining emotions of the first user's content to be recognized based on the second feature vector and the emotion recognizer. In this case, the emotion recognizer may be a data set expressing a correlation between a plurality of test contents including at least one type of object and a plurality of second users' emotions on the plurality of test contents.

The recognition target content may include at least one object of at least one type of an object of a text type, an object of an image type, and an object of an audio type.

The generating of the first feature vector may generate at least one of a first feature vector for the text type object, a first feature vector for the image type object, and a first feature vector for the audio type. have.

Generating the second feature vector comprises: a first feature vector in a first dimension for the object of the text type, a first feature vector in a second dimension for the object of the image type, and a third dimension for the audio type. At least one of the first feature vectors may be merged to generate the second feature vector in a fourth dimension. In this case, the fourth dimension may be determined based on a sum of at least one of the first dimension, the second dimension, and the third dimension.

The emotion recognizer may be a data set that further considers personal information of the plurality of second users in expressing correlations of emotions of the plurality of second users with respect to the plurality of test contents. In this case, the step of receiving the content further receives the first user's personal information and the step of generating the first feature vector further generates a first feature vector for the first user's personal information, and the In the step of generating a feature vector, the second feature vector may be generated by further considering the first feature vector for the personal information.

A method of generating an emotion recognizer that recognizes a user's emotion based on multimodal information included in test content according to an embodiment of the present invention includes at least one type of object, and the user's emotion information Obtaining one or more labeled test contents; Generating, for each test content, a first feature vector for each type of object included in the test content; Generating, for each test content, a second feature vector that is a feature vector of the test content, based on the first feature vector for each type; And training the emotion recognizer based on the second feature vector and the emotion information.

The emotion recognizer may be a data set expressing a correlation between a plurality of test contents including at least one type of object and a plurality of user emotions on the plurality of test contents.

Each of the test contents may include at least one object of at least one type of an object of a text type, an object of an image type, and an object of an audio type.

The generating of the first feature vector may generate at least one of a first feature vector for the text type object, a first feature vector for the image type object, and a first feature vector for the audio type. have.

Generating the second feature vector comprises: a first feature vector in a first dimension for the object of the text type, a first feature vector in a second dimension for the object of the image type, and a third dimension for the audio type. At least one of the first feature vectors may be merged to generate the second feature vector in a fourth dimension. In this case, the fourth dimension may be determined based on a sum of at least one of the first dimension, the second dimension, and the third dimension.

The emotion recognizer may be a data set that further considers the personal information of the plurality of users in expressing correlations of emotions of the plurality of users with respect to the plurality of test contents. At this time, the step of acquiring the test content acquires one or more test contents in which the user's personal information is further labeled, and the step of generating the first feature vector further generates a first feature vector for the user's personal information. And, in the step of generating the second feature vector, the second feature vector may be generated by further considering the first feature vector for the personal information.

The emotion information includes a third feature vector, and the training step may update the data set of the emotion recognizer so that the second feature vector and the third feature vector correspond.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to embodiments of the present invention, the emotion recognition method and program and the emotion recognizer generation method capable of collecting objects included in the same content for each type of object and integrating characteristic information of each type of collected object at a lower level And programs.

In addition, the present invention performs emotion recognition on content based on feature information of each type of entity integrated at a lower level, and thus an emotion recognition method capable of deriving an emotion recognition result with an understanding of content flow or context. And a program and a method and program for generating an emotion recognizer.

Furthermore, it is possible to implement emotion recognition methods and programs with improved accuracy and emotion recognizer generation methods and programs.

1 schematically illustrates an emotion recognition system according to an embodiment of the present invention.
2 schematically shows a configuration of an emotion recognition device according to an embodiment of the present invention.
3 is a diagram for explaining a method of generating an emotion recognizer by an emotion recognition apparatus according to an embodiment of the present invention.
4 to 5D are flowcharts illustrating a method for generating an emotion recognizer in an emotion recognition apparatus according to an embodiment of the present invention.
6 is a view for explaining a method for recognizing the emotion of the content to be recognized by the emotion recognition apparatus using the emotion recognizer according to an embodiment of the present invention.
7 is a flowchart illustrating an emotion recognition method performed by an emotion recognition apparatus according to an embodiment of the present invention.
8 is an example of a method for generating a second feature vector by the emotion recognition apparatus according to an embodiment of the present invention.

The present invention can be applied to various transformations and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention and methods for achieving them will be clarified with reference to embodiments described below in detail with reference to the drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various forms.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, and the same or corresponding components will be denoted by the same reference numerals, and redundant description thereof will be omitted. .

In the following examples, terms such as first and second are not used in a limited sense, but for the purpose of distinguishing one component from other components. In the following embodiments, the singular expression includes the plural expression unless the context clearly indicates otherwise. In the examples below, terms such as include or have are meant to mean the presence of features or components described in the specification, and do not preclude the possibility of adding one or more other features or components in advance. In the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, since the size and shape of each component shown in the drawings are arbitrarily shown for convenience of description, the present invention is not necessarily limited to what is illustrated.

1 schematically illustrates an emotion recognition system according to an embodiment of the present invention.

Referring to FIG. 1, an emotion recognition system according to an embodiment of the present invention may include a server 100, a user terminal 200, an external device 300, and a communication network 400 connecting them.

In the emotion recognition system according to an embodiment of the present invention, the server 100 may receive the content to be recognized from the user terminal 200 and / or the external device 300, and determine emotion for the received content. In addition, the emotion recognition system may obtain one or more test contents marked with emotion information from the user terminal 200 and / or the external device 300, and train the emotion recognizer based on the emotion recognition system. A more detailed description will be given later.

In the present invention, 'Contents', such as content to be recognized and test content, are information that can be transmitted and received electronically, and may include various types of objects. The content may include at least one object of at least one type of an object of the text type, an object of the image type, and an object of the audio type. Of course, the content may also include video type objects.

For example, content such as a 'web document' may include text-type objects for conveying information and image-type objects for understanding information. At this time, each of the text type object and the image type object may be singular or plural.

In the present invention, an 'object' such as an object of a text type, an object of an image type, an object of an audio type, and an object of a video type may be a small unit of information constituting content. .

For example, in the case of a text type object, it may mean a paragraph of a paragraph included in the content. Also, in the case of an image type object, it may mean a single image included in content. For the audio type object and the video type object, it may mean one audio (video) clip, an audio (video) track, and the like.

In the present invention, the user terminal 200 may refer to various devices capable of transmitting and receiving the aforementioned content with the server 100. At this time, the terminal may be a personal computer 202 or a portable terminal 201. 1, the portable terminal 201 is illustrated as a smart phone, but the spirit of the present invention is not limited thereto.

Meanwhile, the user terminal 200 may include display means for displaying content, and input means for obtaining a user's input to the content. At this time, the input means and the display means may be configured in various ways. For example, the input means may include, but is not limited to, a keyboard, mouse, trackball, microphone, button, touch panel, and the like.

In the present invention, the external device 300 may refer to various devices that transmit and receive data through the server 100 and the user terminal 200 and the communication network 400.

According to an embodiment of the present invention, the external device 300 may be a device that provides test content for training an emotion recognizer provided in the server 100. At this time, the external device 300 may be singular or plural. For example, the external device 300 may be a plurality of servers that provide content in response to web crawling of the server 100.

 On the other hand, according to another embodiment of the present invention, the external device 300 transmits the identification information for the content provided by the external device 300 itself to the user terminal 200 to the server 100, and from the server 100 It may be a device that receives the result of emotion recognition for the content. For example, the external device 300 may be a server of a service provider who wants to know a user's feelings about content provided by the user.

However, the above-mentioned two cases are both exemplary, and the spirit of the present invention is not limited thereto.

In the present invention, the communication network 400 serves to connect the server 100, the user terminal 200, and the external device 300. For example, the communication network 400 provides an access path so that the user terminal 200 can transmit and receive packet data after accessing the server 100. The communication network 400 includes, for example, wired networks such as Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), and Integrated Service Digital Networks (ISDNs), wireless LANs, CDMA, Bluetooth, satellite communication, etc. It may cover a wireless network, but the scope of the present invention is not limited thereto.

In the present invention, the server 100 may receive the content to be recognized from the user terminal 200 and / or the external device 300, and determine emotion on the received content. In addition, the server 100 may acquire one or more test contents marked with emotion information from the user terminal 200 and / or the external device 300, and train the emotion recognizer based on this. To this end, the server 100 may include an emotion recognition device according to an embodiment of the present invention.

2 schematically shows the configuration of the emotion recognition device 110 according to an embodiment of the present invention.

Referring to FIG. 2, the emotion recognition device 110 according to the present embodiment may include a communication unit 111, a control unit 112, and a memory 113. In addition, although not shown in the drawings, the emotion recognition device 110 according to the present embodiment may further include an input / output unit, a program storage unit, and the like.

The communication unit 111 includes hardware required for the emotion recognition device 110 to transmit / receive signals such as control signals or data signals through wired / wireless connections with other network devices such as the user terminal 200 and / or the external device 300, and It may be a device that includes software.

The controller 112 may include all types of devices capable of processing data, such as a processor. Here, a 'processor' may mean a data processing device embedded in hardware having physically structured circuits, for example, to perform functions represented by codes or instructions included in a program. As an example of such a data processing device embedded in hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, and an application-specific integrated (ASIC) Circuit), FPGA (Field Programmable Gate Array), and the like, but the scope of the present invention is not limited thereto.

The memory 113 performs a function of temporarily or permanently storing data processed by the emotion recognition device 110. The memory may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto.

Hereinafter, it is assumed that the emotion recognition device 110 is provided in the server 100, but the emotion recognition device 110 may be separately provided from the server 100 according to role allocation.

On the other hand, as described above, the server 100, that is, the emotion recognition device 110 may receive the content to be recognized from the user terminal 200 and / or the external device 300, and determine emotion on the received content. In addition, the emotion recognition device 110 may acquire one or more test contents marked with emotion information from the user terminal 200 and / or the external device 300, and generate an emotion recognizer based on the test content.

Hereinafter, a method of generating the emotion recognizer from the one or more test contents by the emotion recognition apparatus 110 will be described first, and a method of determining emotion for the received content based on the generated emotion recognizer will be described later.

<Emotion recognition device generating method of the emotion recognition device 110>

In the present invention, the 'emotional recognizer' may be a data set expressing a correlation between a plurality of test contents and a plurality of users' emotions on the plurality of test contents.

According to an embodiment of the present invention, the above-described emotion recognizer may be a correlation between a plurality of test contents generated by a machine learning technique and emotions on the corresponding contents, that is, mapping information of both. Accordingly, the emotion recognizer uses algorithms and / or algorithms such as Logistic regression, Decision tree, Nearest-neighbor classifier, Kernel discriminate analysis, Neural network, Support Vector Machine, Random forest, Boosted tree to classify multiple test contents and / or objects. Method (technique) can be used.

The emotion recognizer may use algorithms and / or methods (techniques) such as linear regression, regression tree, kernel regression, support vector regression, and deep learning to predict a user's emotion for a certain content.

In addition, the emotion recognizer may use algorithms and / or methods (techniques) such as Principal component analysis, Non-negative matrix factorization, Independent component analysis, Manifold learning, SVD, etc. for the calculation of vectors described later.

The emotion recognizer may use algorithms and / or methods (techniques) such as k-means, hierarchical clustering, mean-shift, and self-organizing maps (SOMs) for grouping a plurality of test contents and / or objects.

The emotion recognizer may use algorithms and / or methods (techniques) such as bipartite cross-matching, n-point correlation two-sample testing, and minimum spanning tree for data comparison.

However, the above-described algorithms and / or methods (techniques) are exemplary and the spirit of the present invention is not limited thereto.

Meanwhile, the emotion recognizer may be a data set as described above. For example, the emotion recognizer may be a data set composed of a plurality of numbers, such as a matrix. However, this is exemplary and the spirit of the present invention is not limited thereto.

3 is a diagram for explaining a method of generating an emotion recognizer by the emotion recognition apparatus 110 according to an embodiment of the present invention.

Referring to FIG. 3, the control unit 112 of the emotion recognition device 110 according to an embodiment of the present invention includes at least one or more types of objects for generating a recognizer, and the user's emotion information is labeled One or more test contents 310 may be acquired. At this time, the control unit 112 may acquire the test content 310 from the external device 300 through web crawling as described above, or from the memory 113 of the emotion recognition device 110. In other words, the control unit 112 may acquire the test content 310 from the outside of the emotion recognition device 110 and / or the inside of the emotion recognition device 110. However, this is exemplary and the spirit of the present invention is not limited thereto.

In the present invention, 'the emotion information is marked on the content' may mean that the user's emotion for the content is added as additional information of the content. In other words, the fact that the emotion information is labeled may mean that the emotion information of the viewer for the content is further added separately from the content of the content. At this time, the test content may include information about emotion itself, such as 'joy' and 'sadness', in the form of a vector. Also, the test content may be labeled with information to infer emotions, such as "The content is really fun." In the latter case, the controller 112 may analyze the labeled information and infer information about the emotion itself in the form of a vector.

Meanwhile, in addition to the above-described emotion information, personal information of a user who has felt the emotion may be added to the test content. At this time, personal information includes general personal matters such as gender, age, age, educational background, residential area and occupation, and content-related matters such as when the content was viewed, where it was viewed, how long it was viewed, or through which channel. It may include. However, this is exemplary and the spirit of the present invention is not limited thereto.

The control unit 112 according to an embodiment of the present invention may generate a first feature vector for each type of object included in the test content, for each test content obtained through the above-described process. In this case, 'creating a first feature vector for each type' may mean collecting all objects of the same type in any one test content and generating a first feature vector based on the same.

For example, suppose that the test content 311 includes an object 321 of a text type, an object 322 of an image type, and an object 323 of an audio type. In this case, the control unit 112 removes the first feature vector 331 from the text type object 321, the first feature vector 332 from the image type object 322, and the audio type object 323. One feature vector 333 can be generated.

Looking at the process of generating the first feature vector 331 from the text-type object 321 by the control unit 112 in more detail, first, the control unit 112 may include all or some text-type objects 321 in the content 311. ). Subsequently, the controller 112 may extract characteristics of the text-type object 321 through parsing. For example, the controller 112 may extract a keyword related to emotion in the text-type object 321 through parsing. Subsequently, the control unit 112 may generate the first feature vector 331 based on the extracted keyword. At this time, the control unit 112 may extract keywords of the object 321 of a text type for each predetermined unit (eg, a paragraph), and generate the first feature vector 331 based on the keyword, or within the test content 311. The keyword may be extracted from the entire text-type object 321, and the first feature vector 331 may be generated based on the keyword.

Looking at the process of generating the first feature vector 332 from the object 322 of the image type by the control unit 112 in more detail, first, the control unit 112 may control all or part of the image type object in the test content 311 ( 322). Subsequently, the control unit 112 may extract characteristics of the object 322 of the image type through image analysis. For example, the control unit 112 may extract a feature related to emotion in the object 322 of the image type through a feature point extraction technique and / or an object recognition technique. Subsequently, the control unit 112 may generate the first feature vector 332 based on the extracted feature. At this time, the controller 112 may extract features for each image, and generate a first feature vector 332 based on the features, or extract features from all images and generate a first feature vector 332 based on the features. You may.

If the control unit 112 looks at the process of generating the first feature vector 333 from the audio-type object 323 in more detail, first, the control unit 112 controls all or part of the audio-type object 323 in the content 311. ). Subsequently, the control unit 112 may extract characteristics of the object 323 of the audio type through audio analysis. For example, the controller 112 may extract a feature related to emotion in the object 323 of the audio type through a feature sound extraction technique and / or a voice recognition technique. Subsequently, the control unit 112 may generate the first feature vector 333 based on the extracted feature.

The first feature vectors 331 to 333 generated from each type of entity may all have different dimensions. For example, the first feature vector 331 generated from the text-type object 321 is the first dimension, the first feature vector 332 generated from the image-type object 322 is the second dimension, and the audio type is The first feature vector 333 generated from the object 323 may be a third dimension. Of course, at least some of each of the first feature vectors 331 to 333 may have the same dimension. However, this is exemplary and the spirit of the present invention is not limited thereto.

Meanwhile, when the test content 311 includes an object (not shown) of the video type, the control unit 112 according to an embodiment of the present invention separates an object of the video type into an object of the image type and an object of the audio type. , A first feature vector may be generated according to a method of generating a first feature vector for each separated entity. In other words, the controller 112 generates a first feature vector according to a first feature vector generation method for an image type object, for an image type object separated from the video type object, and is separated from the video type object. For an audio type object, a first feature vector may be generated according to a first feature vector generation method for an audio type object. However, this type of video object is an example of combining two or more different types of objects, and the spirit of the present invention is not limited thereto.

If the personal information 313 is further displayed on the test content 311, the controller 112 according to an embodiment of the present invention may further generate a first feature vector 334 for personal information. At this time, the controller 112, as described above, related to general personal matters such as gender, age, age, educational background, residential area, and occupation, and content such as when, where, or through which route the content was viewed. The first feature vector 334 may be generated in consideration of all matters.

The controller 112 according to an embodiment of the present invention may generate a second feature vector 341 that is a feature vector of test content based on the first feature vectors 331 to 333 for each type of the generated object. In this case, the second feature vector 341 may have a meaning as a vector considering characteristics of all objects included in the test content 311.

According to the prior art, when a plurality of types of objects are included in the content, the user's feelings for the corresponding content are inferred on the basis of only text-type objects, or after the emotions for each type of object are individually inferred, This was statistically processed to infer the user's feelings about the content.

In this way, even though each type of object is all about the same subject, it lacks understanding of the content flow or context because it infers feelings for the entire content based on the results of individual emotion recognition for each type of object. There was a problem leading to the result.

In particular, this method processes individual emotion recognition results for each type of individual based on a statistical model, so even though somewhat different emotion recognition results are derived from some types of individuals, more detailed emotion recognition is ignored by ignoring these recognition results. There was an impossible problem.

The present invention can derive the emotion recognition result by adding the understanding of the flow or context of the content by integrating the feature information of each type of individual at a lower level and performing emotion recognition on the content based on this. Can produce detailed emotion recognition results.

Meanwhile, the control unit 112 according to an embodiment of the present invention includes a first feature vector 331 generated from an object 321 of a text type, a first feature vector 332 generated from an object 322 of an image type, The second feature vector 341 may be generated by merging the first feature vector 333 generated from the audio-type entity 323 and the first feature vector 334 generated from personal information. At this time, the control unit 112 may generate the second feature vector 341 from the first feature vectors 331 to 334 according to various methods.

For example, the controller 112 may generate a second feature vector 341 by merging each of the first feature vectors 331 to 334 in series in a manner similar to the method illustrated in FIG. 8. At this time, the number of dimensions of the second feature vector 341 may be determined based on the sum of the number of dimensions of each of the first feature vectors 331 to 334.

On the other hand, unlike the above-described method, the control unit 112 may generate a second feature vector 341 having a fixed number of digits by dimensional transformation operations and / or vector operations of the first feature vectors 331 to 334.

However, this is exemplary and the spirit of the present invention is not limited thereto, and any operation that generates one second feature vector based on a plurality of first feature vectors may be used without limitation.

The control unit 112 according to an embodiment of the present invention may generate a first feature vector of an object of a type that is not included according to a preset method when some types of objects are not included in the test content.

For example, when the test content does not include the object 321 of the text type, the control unit 112 may generate the first feature vector 331 for the object of the type that is not included as a zero vector. Can be.

In other words, when the second feature vector is generated, when the specific type of object is not included in the test content, the control unit 112 may replace the first feature vector for the object of the corresponding type with a preset vector. At this time, the preset vector may be a vector filled with '0' as described above. However, this is exemplary and the spirit of the present invention is not limited thereto.

The controller 112 according to an embodiment of the present invention trains the emotion recognizer 351 based on the generated second feature vector 341 and the emotion information 312 added to the test content 311. Can be. At this time, the emotion information 312 may include information about emotion in the form of a vector. Therefore, hereinafter, it is assumed that the above-described emotion information 312 includes a third feature vector.

As described above, the emotion recognizer 351 may be correlation information between a plurality of test contents 310 generated by a machine learning technique and corresponding contents, that is, mapping information of both.

Therefore, the control unit 112 according to an embodiment of the present invention can correspond (map) the second feature vector 341 generated by the above-described process and the third feature vector including emotion information 312 to each other. The recognizer 351 may be trained by updating the data set of the emotion recognizer 351.

For example, when the data set of the emotion recognizer 351 is composed of a plurality of coefficients (Weight), the control unit 112 determines the emotion recognizer 351 based on the second feature vector 341 and the third feature vector of each test content. Coefficients constituting can be adjusted. At this time, the control unit 112 generates and / or acquires the second feature vector 341 and the third feature vector from the plurality of test contents and updates the coefficients based thereon, thereby providing an emotion recognizer with improved accuracy. (351) can be obtained.

Accordingly, the present invention can derive the emotion recognition result by adding the understanding of the flow or context of the content by integrating the feature information of each type of entity at a lower level and performing emotion recognition on the content based on this, An emotion recognizer 351 capable of deriving a more detailed emotion recognition result may be generated.

4 to 5D are flowcharts illustrating a method for generating an emotion recognizer of the emotion recognition apparatus 110 according to an embodiment of the present invention. Hereinafter, an embodiment of the present invention will be described with reference to FIGS. 4 and 5D together.

Referring to FIG. 3, the control unit 112 of the emotion recognition device 110 according to an embodiment of the present invention includes at least one or more types of objects for generating a recognizer, and the user's emotion information is labeled One or more test contents 310 may be acquired. (S41) At this time, the control unit 112 may acquire the test contents 310 from the external device 300 through web crawling as described above. It can also be obtained from the memory 113 of the recognition device 110. In other words, the control unit 112 may acquire the test content 310 from the outside of the emotion recognition device 110 and / or the inside of the emotion recognition device 110. However, this is exemplary and the spirit of the present invention is not limited thereto.

Meanwhile, in addition to the above-described emotion information, personal information of a user who has felt the emotion may be added to the test content. At this time, personal information includes general personal matters such as gender, age, age, educational background, residential area and occupation, and content-related matters such as when the content was viewed, where it was viewed, how long it was viewed, or through which channel It may include. However, this is exemplary and the spirit of the present invention is not limited thereto.

The control unit 112 according to an embodiment of the present invention may generate a first feature vector for each type of object included in the test content, for each test content obtained by the above-described process. (S42) In this case, 'creating a first feature vector for each type' may mean collecting all objects of the same type in any one test content and generating a first feature vector based on the same.

For example, suppose that the test content 311 includes an object 321 of a text type, an object 322 of an image type, and an object 323 of an audio type. In this case, the control unit 112 removes the first feature vector 331 from the text type object 321, the first feature vector 332 from the image type object 322, and the audio type object 323. One feature vector 333 can be generated.

Referring to FIG. 5A, when the control unit 112 generates the first feature vector 331 from the text-type object 321 in more detail, first, the control unit 112 may control all or part of the content 311. The text-type object 321 may be collected. (S421a) Subsequently, the control unit 112 may extract characteristics of the text-type object 321 through parsing. (S422a) For example, the control unit 112 may Through parsing, keywords related to emotions in the object 321 of the text type may be extracted as features. Subsequently, the control unit 112 may generate the first feature vector 331 based on the extracted keyword. (S423a) At this time, the control unit 112 may be a text type object 321 for each predetermined unit (for example, a paragraph). The keyword of, and the first feature vector 331 may be generated based on this, and the keyword may be extracted from the entire text-type object 321 in the test content 311, and based on the first feature vector ( 331).

Referring to FIG. 5B, when the control unit 112 generates the first feature vector 332 from the image-type object 322 in more detail, first, the control unit 112 may control all or part of the test content 311. Objects 322 of the image type may be collected. (S421b) Subsequently, the control unit 112 may extract the characteristics of the object 322 of the image type through image analysis. (S422b) For example, the control unit 112 may extract the image type through the feature point extraction technique and / or the object recognition technique. Features related to the emotion in the object 322 of can be extracted. Subsequently, the controller 112 may generate a first feature vector 332 based on the extracted features. (S423b) At this time, the controller 112 extracts features for each image, and based on this, the first feature vector ( 332) may be generated, features may be extracted from all images, and a first feature vector 332 may be generated based on the features.

Referring to FIG. 5C, the process of generating the first feature vector 333 from the audio object 323 by the control unit 112 will be described in more detail. First, the control unit 112 may control all or part of the audio in the content 311. The object 323 of the type may be collected. (S421c) Then, the control unit 112 may extract characteristics of the object 323 of the audio type through audio analysis (S422c). Features related to emotion in the object 323 of the audio type may be extracted through a sound extraction technique and / or a speech recognition technique. Subsequently, the control unit 112 may generate the first feature vector 333 based on the extracted feature. (S423c)

The first feature vectors 331 to 333 generated from each type of entity may all have different dimensions. For example, the first feature vector 331 generated from the text-type object 321 is the first dimension, the first feature vector 332 generated from the image-type object 322 is the second dimension, and the audio type is The first feature vector 333 generated from the object 323 may be a third dimension. Of course, at least some of each of the first feature vectors 331 to 333 may have the same dimension. However, this is exemplary and the spirit of the present invention is not limited thereto.

Meanwhile, when an object (not shown) of a video type is included in the test content 311, the control unit 112 according to an embodiment of the present invention collects an object of the video type (S421d) and collects the object of the video type. Is separated into an image type object and an audio type object (422d), and a first feature vector may be generated according to a first feature vector generation method for each separated object. (423d) In other words, the control unit 112 ) Generates a first feature vector according to a first feature vector generation method for an image type object, and an audio type object separated from the video type object, for an image type object separated from the video type object. With respect to this, the first feature vector may be generated according to the first feature vector generating method for the audio type object. However, this type of video object is an example of combining two or more different types of objects, and the spirit of the present invention is not limited thereto.

If the personal information 313 is further displayed on the test content 311, the controller 112 according to an embodiment of the present invention may further generate a first feature vector 334 for personal information. At this time, the controller 112, as described above, related to general personal matters such as gender, age, age, educational background, residential area, and occupation, and content such as when, where, or through which route the content was viewed. The first feature vector 334 may be generated in consideration of all matters.

The controller 112 according to an embodiment of the present invention may generate a second feature vector that is a feature vector of the test content based on the first feature vector for each type of the generated object. (S43) The vector may have a meaning as a vector considering characteristics of all objects included in the test content.

According to the prior art, when a plurality of types of objects are included in the content, the user's feelings for the corresponding content are inferred on the basis of only text-type objects, or after the emotions for each type of object are individually inferred, This was statistically processed to infer the user's feelings about the content.

In this way, even though each type of object is all about the same subject, it lacks understanding of the content flow or context because it infers feelings for the entire content based on the results of individual emotion recognition for each type of object. There was a problem leading to the result.

Particularly, this method processes individual emotion recognition results for each type of individual based on a statistical model, so even though somewhat different emotion recognition results are derived from some types of individuals, more detailed emotion recognition is possible by ignoring these recognition results. There was an impossible problem.

The present invention can derive the emotion recognition result by adding the understanding of the flow or context of the content by integrating the feature information of each type of individual at a lower level and performing emotion recognition on the content based on this. Can produce detailed emotion recognition results.

Meanwhile, the control unit 112 according to an embodiment of the present invention includes a first feature vector 331 generated from an object 321 of a text type, a first feature vector 332 generated from an object 322 of an image type, The second feature vector 341 may be generated by merging the first feature vector 333 generated from the audio-type entity 323 and the first feature vector 334 generated from personal information. At this time, the control unit 112 may generate the second feature vector 341 from the first feature vectors 331 to 334 according to various methods.

For example, the controller 112 may generate a second feature vector 341 by merging each of the first feature vectors 331 to 334 in series in a manner similar to the method illustrated in FIG. 8. At this time, the number of dimensions of the second feature vector 341 may be determined based on the sum of the number of dimensions of each of the first feature vectors 331 to 334.

On the other hand, unlike the above-described method, the control unit 112 may generate a second feature vector 341 having a fixed number of digits by dimensional transformation operations and / or vector operations of the first feature vectors 331 to 334.

However, this is exemplary and the spirit of the present invention is not limited thereto, and any operation that generates one second feature vector based on a plurality of first feature vectors may be used without limitation.

The controller 112 according to an embodiment of the present invention trains the emotion recognizer 351 based on the generated second feature vector 341 and the emotion information 312 added to the test content 311. (S44) At this time, the emotion information 312 may include information about the emotion in the form of a vector. Therefore, hereinafter, it is assumed that the above-described emotion information 312 includes a third feature vector.

As described above, the emotion recognizer 351 may be correlation information between a plurality of test contents 310 generated by a machine learning technique and corresponding contents, that is, mapping information of both.

Therefore, the control unit 112 according to an embodiment of the present invention can correspond (map) the second feature vector 341 generated by the above-described process and the third feature vector including emotion information 312 to each other. The recognizer 351 may be trained by updating the data set of the emotion recognizer 351.

For example, when the data set of the emotion recognizer 351 is composed of a plurality of coefficients (Weight), the control unit 112 determines the emotion recognizer 351 based on the second feature vector 341 and the third feature vector of each test content. Coefficients constituting can be adjusted. At this time, the control unit 112 generates and / or acquires the second feature vector 341 and the third feature vector from the plurality of test contents and updates the coefficients based thereon, thereby providing an emotion recognizer with improved accuracy. (351) can be obtained.

Accordingly, the present invention can derive the emotion recognition result by adding the understanding of the flow or context of the content by integrating the feature information of each type of entity at a lower level and performing emotion recognition on the content based on this, An emotion recognizer 351 capable of deriving a more detailed emotion recognition result may be generated.

<Emotion recognition method using emotion recognition device 110>

In the above, the method of generating the emotion recognition apparatus 110, particularly the emotion recognizer, has been described with reference to FIGS. 3 to 5D. Hereinafter, a method of recognizing the emotion for the content to be recognized by the emotion recognition apparatus 110 generated by the above-described process through FIGS. 6 to 8 will be described. Therefore, hereinafter, it will be described on the assumption that the emotion recognizer is generated by the emotion recognition device 110 by the above-described process.

6 is a view for explaining a method of recognizing the emotion of the content to be recognized by the emotion recognition device 110 according to an embodiment of the present invention using an emotion recognizer.

Referring to FIG. 6, the control unit 112 of the emotion recognition device 110 according to an embodiment of the present invention may receive the content to be recognized 611 including at least one type of object.

At this time, the control unit 112 may receive the content to be recognized 611 itself from the user terminal 200. For example, the control unit 112 receives the content 611 including the text type object 621, the image type object 622, and the audio type object 623 from the user terminal 200, as shown in FIG. I can receive it.

Also, the control unit 112 may receive only identification information of the content 611 to be recognized from the user terminal 200. For example, the control unit 112 may receive the URL of the content to be recognized 611 from the user terminal 200. In this case, the control unit 112 may obtain the corresponding content 611 from the content provider (not shown) based on the identification information received from the user terminal 200.

According to another embodiment of the present invention, the control unit 112 may receive the identification information of the recognition target content 611 itself and / or the recognition target content 611 from the external device 300. At this time, the external device 300 may be a device that transmits identification information about the content provided to the user terminal 200 to the server 100 and receives an emotion recognition result for the content from the server 100 have. For example, the external device 300 may be a server of a service provider who wants to know a user's feelings about content provided by the user.

Meanwhile, the personal information 612 of the first user who views the corresponding content 611 may be added to the recognition target content 611 as additional information. At this time, the personal information 612 includes general personal matters such as gender, age, age, educational background, residential area, and occupation, as described above, and content such as when the content was viewed, where it was viewed, or through what channel. It may include related matters. However, this is exemplary and the spirit of the present invention is not limited thereto.

The controller 112 according to an embodiment of the present invention may generate the first feature vectors 631 to 634 for each type of object included in the content to be recognized obtained by the above-described process. In this case, 'creating the first feature vector for each type' may mean collecting all objects of the same type in the content to be recognized and generating the first feature vector based on the same.

For example, suppose that the content to be recognized 611 includes an object 621 of a text type, an object 622 of an image type, and an object 623 of an audio type. In this case, the control unit 112 removes the first feature vector 631 from the text type object 621, the first feature vector 632 from the image type object 622, and the audio type object 623. One feature vector 633 can be generated. Of course, when the personal information 612 is further displayed on the content 611 to be recognized, the control unit 112 according to an embodiment of the present invention may further generate a first feature vector 634 for personal information.

The control unit 112 according to an embodiment of the present invention, when the recognition target content 611 includes an object of the video type, separates the object of the video type into an object of the image type and an object of the audio type, each separated A first feature vector may be generated according to a method of generating a first feature vector for an individual of.

As described above, since the control unit 112 generates the first feature vectors 631 to 634 from each individual and / or personal information, detailed descriptions thereof will be omitted.

The controller 112 according to an embodiment of the present invention generates a second feature vector 641 that is a feature vector of the content to be recognized 611 based on the first feature vectors 631 to 634 for each type of the generated object. can do. At this time, the second feature vector 641 may have a meaning as a vector considering characteristics of all objects included in the content to be recognized 611.

According to the prior art, when a plurality of types of objects are included in the content, the user's feelings for the corresponding content are inferred on the basis of only text-type objects, or after the emotions for each type of object are individually inferred, This was statistically processed to infer the user's feelings about the content.

In this way, even though each type of object is all about the same subject, it lacks understanding of the content flow or context because it infers feelings for the entire content based on the results of individual emotion recognition for each type of object. There was a problem leading to the result.

Particularly, this method processes individual emotion recognition results for each type of individual based on a statistical model, so even though somewhat different emotion recognition results are derived from some types of individuals, more detailed emotion recognition is possible by ignoring these recognition results. There was an impossible problem.

The present invention can derive the emotion recognition result by adding the understanding of the flow or context of the content by integrating the feature information of each type of individual at a lower level and performing emotion recognition on the content based on this. Can produce detailed emotion recognition results.

Meanwhile, the control unit 112 according to an embodiment of the present invention includes a first feature vector 631 generated from an object 621 of a text type, a first feature vector 632 generated from an object 622 of an image type, The second feature vector 641 may be generated by merging the first feature vector 633 generated from the audio-type entity 623 and the first feature vector 634 generated from the personal information 612. At this time, the control unit 112 may generate the second feature vector 641 according to various methods.

For example, as illustrated in FIG. 8, the controller 112 may generate the second feature vector 641 by merging each of the first feature vectors 631 to 634 in series. At this time, the number of dimensions of the second feature vector 641 may be determined based on the sum of the number of dimensions of each of the first feature vectors 631 to 634.

On the other hand, unlike the above-described method, the control unit 112 may generate the second feature vector 641 of a fixed number of digits by the dimensional transformation operation and / or the vector operation of the first feature vectors 631 to 634. However, this is exemplary and the spirit of the present invention is not limited thereto, and any operation that generates one second feature vector based on a plurality of first feature vectors may be used without limitation.

The controller 112 according to an embodiment of the present invention may determine the emotion of the first user's recognition target content 611 based on the generated second feature vector 641 and the emotion recognizer 351.

As described above, the emotion recognizer 351 may be a correlation between a plurality of test contents generated by a machine learning technique and emotions on the corresponding contents, that is, mapping information of both.

Accordingly, referring to the emotion recognizer 351, the controller 112 according to an embodiment of the present invention may check emotion information 651 mapped to the generated second feature vector 641.

At this time, the emotion information 651 may include information about emotion in the form of a vector. For example, the emotion information 651 is a vector such as [0.1, 0.2, 0.99, 0.4 ..], the first value is a value related to 'joy', the second is a value related to 'sadness', and the third is' Let's say it's a value related to boredom. In this case, the control unit 112 may determine the user's emotion for the content to be recognized as 'boredom'.

Accordingly, the present invention can derive an emotion recognition result to which an understanding of the flow or context of the content is added from the content to be recognized, and to derive a more detailed emotion recognition result.

7 is a flowchart illustrating an emotion recognition method performed by the emotion recognition device 110 according to an embodiment of the present invention.

The control unit 112 of the emotion recognition device 110 according to an embodiment of the present invention may receive the recognition target content 611 including at least one type of object from the user terminal 200 (S71). At this time, the control unit 112 may receive the content to be recognized 611 itself from the user terminal 200. For example, the control unit 112 receives the content 611 including the text type object 621, the image type object 622, and the audio type object 623 from the user terminal 200, as shown in FIG. I can receive it.

Also, the control unit 112 may receive only identification information of the content 611 to be recognized from the user terminal 200. For example, the control unit 112 may receive the URL of the content to be recognized 611 from the user terminal 200. In this case, the control unit 112 may obtain the corresponding content 611 from the content provider (not shown) based on the identification information received from the user terminal 200.

According to another embodiment of the present invention, the control unit 112 may receive identification information of the recognition target content 611 itself and / or the recognition target content 611 from the external device 300, as illustrated in FIG. 7. It might be. At this time, the external device 300 may be a device that transmits identification information about the content provided to the user terminal 200 to the server 100 and receives an emotion recognition result for the content from the server 100 have. For example, the external device 300 may be a server of a service provider who wants to know a user's feelings about content provided by the user.

Meanwhile, the personal information 612 of the first user who views the corresponding content 611 may be added to the recognition target content 611 as additional information. At this time, the personal information 612 includes general personal matters such as gender, age, age, educational background, residential area, and occupation, as described above, and content such as when the content was viewed, where it was viewed, or through what channel. It may include related matters. However, this is exemplary and the spirit of the present invention is not limited thereto.

The controller 112 according to an embodiment of the present invention may generate the first feature vectors 631 to 634 for each type of object included in the content to be recognized obtained by the above-described process. (S72) 'Creating a first feature vector for each type' may mean collecting all objects of the same type in the content to be recognized and generating a first feature vector based on the same.

For example, suppose that the content to be recognized 611 includes an object 621 of a text type, an object 622 of an image type, and an object 623 of an audio type. In this case, the control unit 112 removes the first feature vector 631 from the text type object 621, the first feature vector 632 from the image type object 622, and the audio type object 623. One feature vector 633 can be generated. Of course, when the personal information 612 is further displayed on the content 611 to be recognized, the control unit 112 according to an embodiment of the present invention may further generate a first feature vector 634 for personal information.

The control unit 112 according to an embodiment of the present invention, when the recognition target content 611 includes an object of the video type, separates the object of the video type into an object of the image type and an object of the audio type, each separated A first feature vector may be generated according to a method of generating a first feature vector for an individual of.

As described above, since the control unit 112 generates the first feature vectors 631 to 634 from each individual and / or personal information, detailed descriptions thereof will be omitted.

The controller 112 according to an embodiment of the present invention generates a second feature vector 641 that is a feature vector of the content to be recognized 611 based on the first feature vectors 631 to 634 for each type of the generated object. (S73) In this case, the second feature vector 641 may have a meaning as a vector considering characteristics of all objects included in the content to be recognized 611.

Meanwhile, the control unit 112 according to an embodiment of the present invention includes a first feature vector 631 generated from an object 621 of a text type, a first feature vector 632 generated from an object 622 of an image type, The second feature vector 641 may be generated by merging the first feature vector 633 generated from the audio-type entity 623 and the first feature vector 634 generated from the personal information 612. At this time, the control unit 112 may generate the second feature vector 641 according to various methods.

The control unit 112 according to an embodiment of the present invention may determine the emotion of the first user's recognition target content 611 based on the generated second feature vector 641 and the emotion recognizer 351. S74)

As described above, the emotion recognizer 351 may be a correlation between a plurality of test contents generated by a machine learning technique and emotions on the corresponding contents, that is, mapping information of both.

Accordingly, referring to the emotion recognizer 351, the controller 112 according to an embodiment of the present invention may check emotion information 651 mapped to the generated second feature vector 641. At this time, the emotion information 651 may include information about emotion in the form of a vector. For example, the emotion information 651 is a vector such as [0.1, 0.2, 0.99, 0.4 ..], the first value is a value related to 'joy', the second is a value related to 'sadness', and the third is' Let's say it's a value related to boredom. In this case, the control unit 112 may determine the user's emotion for the content to be recognized as 'boredom'.

Accordingly, the present invention can derive an emotion recognition result to which an understanding of the flow or context of the content is added from the content to be recognized, and to derive a more detailed emotion recognition result.

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program can be recorded on a computer-readable medium. At this time, the medium may be to store a program executable by a computer. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks, And program instructions including ROM, RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention or may be known and available to those skilled in the computer software field. Examples of computer programs may include not only machine language codes produced by a compiler, but also high-level language codes executable by a computer using an interpreter or the like.

The specific implementations described in the present invention are exemplary embodiments, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings are illustrative examples of functional connections and / or physical or circuit connections. In the actual device, alternative or additional various functional connections, physical It can be represented as a connection, or circuit connections. In addition, unless specifically mentioned, such as "essential", "important", etc., it may not be a necessary component for the application of the present invention.

Therefore, the spirit of the present invention is not limited to the above-described embodiment, and should not be determined, and the scope of the spirit of the present invention, as well as the claims to be described later, as well as all ranges that are equivalent to or equivalently changed from the claims. Would belong to

100: server
110: emotion recognition device
111: Communication Department
112: control unit
113: memory
200: user terminal
300: external device
400: network

Claims (5)

A method for recognizing a first user's emotion for a content to be recognized that includes at least one type of object,
Learning a emotion recognizer to learn a correlation between the personal information, the at least one type of object, and the emotion information by using a plurality of test contents including user's personal information, at least one type of object, and emotion information Letting;
Collecting the at least one type of object in the content to be recognized for each type;
Generating a first feature vector for each type of object based on the collection result;
Generating a first feature vector for personal information based on the general personal information of the first user and the content viewing information of the first user;
Generating a second feature vector by merging the first feature vector for each type and the first feature vector for the personal information in a preset manner; And
And determining an emotion vector corresponding to the second feature vector using the learned emotion recognizer. The method according to claim 1
When a plurality of first types of objects are included in the content to be recognized,
Collecting the object is
Collecting all of the plurality of first type of objects,
The step of generating a first feature vector for each type of the object is
A method of recognizing emotion based on multi-modal information, generating a first feature vector reflecting features of all first types of objects included in the content to be recognized. The method according to claim 1
The test content is
Emotion recognition method based on multi-modal information, content obtained from external devices through web crawling. The method according to claim 1
The personal information above
It includes at least one of the first user's gender, age, age, educational background, residential area and occupation,
The content viewing information
A method for recognizing emotion based on multi-modal information, wherein the first user includes at least one of when the content to be recognized was viewed, where it was viewed, how long it was viewed, and through what route. The method according to claim 1
The step of generating a first feature vector for each type of the object is
For objects of one or more video types collected,
Separating the video type object into an image type object and an audio type object;
Generating a first feature vector for an object of the image type based on the object of the image type separated from the object of the video type and the object of at least one image type in the content to be recognized; And
The method comprising: generating a first feature vector for an audio type object based on an audio type object separated from the video type object and at least one audio type object in the content to be recognized; Emotion recognition method based on.

Images