KR102565321B1 - Character emotion recognition and tagging appratus, character emotion recognition and tagging method, and character emotion recognition and tagging system including the character emotion recognition and tagging apparatus - Google Patents

Abstract

In one aspect of the present disclosure, a transceiver; at least one processor; and at least one memory operatively connected to the at least one processor to store at least one instruction to cause the at least one processor to perform operations comprising: Input image data including a character object is received from a user terminal through a transceiver, and the character object is classified into one emotion category among a plurality of emotion categories based on a feature of the input image data and a pre-learned classification model. classifying, tagging the input image data with the one emotion category to generate tagged input image data, and transmitting the tagged input image data to the user terminal through the transceiver; The classified model is an apparatus for recognizing and tagging character emotions, which is learned based on an integrated action unit (AU) of a character object included in the training image data stored in the at least one memory.

Description

Character emotion recognition and tagging system including a character emotion recognition and tagging device, a character emotion recognition and tagging method, and a character emotion recognition and tagging device SYSTEM INCLUDING THE CHARACTER EMOTION RECOGNITION AND TAGGING APPARATUS}

The present disclosure relates to a character emotion recognition and tagging system including a character emotion recognition and tagging device, a character emotion recognition and tagging method, and a character emotion recognition and tagging device in the field of artificial intelligence visual (situation) processing.

Content analysis, search and provision are rapidly developing with the development of deep learning-based artificial intelligence technology, and are expected to be applied and utilized in various platform fields. For example, in order to reduce uncertainty about the quality and taste of experiential content, user-customized content can be provided based on artificial intelligence technology in the decision-making process of digital content consumption. User-customized content is emerging as an important keyword in artificial intelligence technology, especially content-related technology. In the content market where economies of scale are important, artificial intelligence technology is an essential element.

In addition, non-face-to-face communication through content distribution/service mobile is increasing in recent years. With the development of the mobile content industry, it is necessary to classify and analyze uploaded content according to the speed and quantity of content uploads that are rapidly increasing.

However, current technology focuses on text or speech content among various contents, and technological development for various multimedia is insufficient. In particular, in the case of creative content such as characters or emoticons, there is no user-customized content providing service capable of automatically recognizing, classifying, and tagging the emotions of characters included in the creation during the process of uploading and distributing them.

Republic of Korea Patent Publication 10-2018-0111467 Korean Registered Patent No. 10-2110393

Various examples of the present disclosure include a character emotion recognition and tagging device, a character emotion recognition and tagging method, and a character emotion recognition and tagging device capable of automatically recognizing and tagging a character's emotion included in content in the process of uploading and distributing content. This is to provide a character emotion recognition and tagging system including

The technical problems to be achieved in various examples of the present disclosure are not limited to those mentioned above, and other technical problems not mentioned above can be solved by those skilled in the art from various examples of the present disclosure to be described below. can be considered by

In one aspect of the present disclosure, a transceiver; at least one processor; and at least one memory operatively connected to the at least one processor to store at least one instruction to cause the at least one processor to perform operations comprising: Input image data including a character object is received from a user terminal through a transceiver, and the character object is classified into one emotion category among a plurality of emotion categories based on a feature of the input image data and a pre-learned classification model. classifying, tagging the input image data with the one emotion category to generate tagged input image data, and transmitting the tagged input image data to the user terminal through the transceiver; The classified model is an apparatus for recognizing and tagging character emotions, which is learned based on an integrated action unit (AU) of a character object included in the training image data stored in the at least one memory.

The unified AU may include a human AU and a character AU.

The human AU includes a plurality of eyebrow AUs, a plurality of eye AUs having different distances and ratios of at least one of a plurality of feature points included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object included in the training image data; It may include at least one of a plurality of nose AUs, a plurality of mouth AUs, and a plurality of cheek AUs.

The character AU may include at least one of a background object AU included in the training image data and an emotional object AU included in the training image data.

The emotion object AU corresponds to an object including an edge at least partially overlapping with an edge of the character object among objects other than the character object in the training image data, and the background object AU corresponds to the training image Data may correspond to objects other than the character object and the emotion object AU.

The operations include: classification option data from the user terminal via the transceiver, the classification option data including selection information for selecting at least one of the background object AU and the emotion object AU; , selects an AU corresponding to the selection information from among the background object AU and the emotion object AU, and the character object based on the human AU, the AU corresponding to the selection information, and the pre-learned classification model It may further include classifying into the one emotion category.

The operations include: receiving evaluation data for the tagged input image data through the transceiver, and if the evaluation data is less than a preset threshold value, at least one of the human AU, the background object AU, and the emotion object AU and classifying the character object into the one emotion category based on the pre-learned classification model.

Each of the unified AUs may be labeled with at least one of the plurality of emotion categories.

In another aspect of the present disclosure, a character emotion recognition and tagging method performed by a character emotion recognition and tagging apparatus, comprising: receiving input image data including a character object from a user terminal; classifying the character object into one emotion category among a plurality of emotion categories based on a feature of the input image data and a pre-learned classification model; tagging the input image data with the one emotion category to generate tagged input image data; and transmitting the tagged input image data to the user terminal, wherein the pre-learned classification model converts an integrated action unit (AU) of character objects included in training image data stored in the at least one memory to the characteristic can be learned by

The integrated AU includes a human AU and a character AU, and the human AU is at least one of distances and ratios between a plurality of feature points included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object included in the training image data. At least one of a plurality of eyebrows AU, a plurality of eyes AU, a plurality of nose AUs, a plurality of mouth AUs, and a plurality of cheek AUs having one or the other, and the character AU is a background object included in the training image data It may include at least one of an AU and an emotional object AU included in the training image data.

The emotion object AU corresponds to an object including an edge at least partially overlapping with an edge of the character object among objects other than the character object in the training image data, and the background object AU corresponds to the training image Data may correspond to objects other than the character object and the emotion object AU.

classification option data from the user terminal, wherein the classification option data includes selection information for selecting at least one of the background object AU and the emotion object AU; receive; select an AU corresponding to the selection information from among the background object AU and the emotion object AU; and classifying the character object into the one emotion category based on the human AU, the AU corresponding to the selection information, and the pre-learned classification model.

receive evaluation data for the tagged input image data; and classifying the character object into the one emotion category based on at least one of the human AU, the background object AU, and the emotion object AU and the pre-learned classification model when the evaluation data is less than a preset threshold. may include more.

Each of the unified AUs may be labeled with at least one of the plurality of emotion categories.

In another aspect of the present disclosure, a user terminal; and a character emotion recognition and tagging device for classifying emotions of character objects included in input image data received from the user terminal, wherein the character emotion recognition and tagging device includes: a transceiver; at least one processor; and at least one memory operatively connected to the at least one processor to store at least one instruction to cause the at least one processor to perform operations comprising: Receiving the input image data from the user terminal through a transceiver, and classifying the character object into one emotion category among a plurality of emotion categories based on features of the input image data and a pre-learned classification model; The one emotion category is tagged to the input image data to generate tagged input image data, and the tagged input image data is transmitted to the user terminal through the transceiver, and the pre-learned classification model is a character emotion recognition and tagging system that is learned based on the characteristic of an integrated action unit (AU) of a character object included in the training image data stored in the at least one memory.

The various examples of the present disclosure described above are only some of the preferred examples of the present disclosure, and various examples in which the technical features of the various examples of the present disclosure are reflected are detailed descriptions to be detailed below by those of ordinary skill in the art. It can be derived and understood based on.

According to various examples of the present disclosure, the following effects are obtained.

According to various examples of the present disclosure, a device for recognizing and tagging character emotions, a method for recognizing and tagging character emotions, and a character emotion recognition and tagging capable of automatically recognizing and tagging emotions of characters included in content in the process of uploading and distributing content A character emotion recognition and tagging system including an apparatus may be provided.

In addition, it is possible to recommend an optimal emotion recognition tag by identifying user-customized information based on user (creator) information and analyzing character images.

In addition, tag information is automatically recommended for content uploaded to the character image platform, and the accuracy of emotional category classification can be increased.

In addition, the efficiency of content management can be improved.

Effects obtainable from various examples of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clearly derived to those skilled in the art based on the detailed description below and can be understood.

The accompanying drawings are provided to aid understanding of various examples of the present disclosure, and provide various examples of the present disclosure together with detailed descriptions. However, technical features of various examples of the present disclosure are not limited to specific drawings, and features disclosed in each drawing may be combined with each other to form a new embodiment. Reference numerals in each figure mean structural elements.
1 is a block diagram of a character emotion recognition and tagging device according to an example of the present disclosure.
2 to 5 are for explaining an action unit (AU) defined according to various examples of the present disclosure.
6 is for explaining emotion category labeling.
7 is a schematic diagram of a character emotion recognition and tagging system according to an example of the present disclosure.
8 is a flowchart of a character emotion recognition and tagging method according to an example of the present disclosure.

Hereinafter, implementations according to the present invention will be described in detail with reference to the accompanying drawings. The detailed description set forth below in conjunction with the accompanying drawings is intended to describe exemplary implementations of the invention, and is not intended to represent the only implementations in which the invention may be practiced. The following detailed description includes specific details for the purpose of providing a thorough understanding of the present invention. However, one skilled in the art recognizes that the present disclosure may be practiced without these specific details.

In some cases, in order to avoid obscuring the concept of the present disclosure, well-known structures and devices may be omitted or may be shown in block diagram form centering on core functions of each structure and device. In addition, the same reference numerals are used to describe like elements throughout the present disclosure.

Since various examples according to the concept of the present invention can be made with various changes and have various forms, various examples will be illustrated in the drawings and described in detail in the present disclosure. However, this is not intended to limit the various examples according to the concept of the present invention to specific disclosed forms, and includes modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component, Similarly, the second component may also be referred to as the first component.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle. Expressions describing the relationship between components, such as "between" and "directly between" or "directly adjacent to" should be interpreted similarly.

In various examples of this disclosure, “/” and “,” should be interpreted as indicating “and/or”. For example, “A/B” may mean “A and/or B”. Furthermore, “A, B” may mean “A and/or B”. Furthermore, “A/B/C” may mean “at least one of A, B and/or C”. Furthermore, “A, B, C” may mean “at least one of A, B and/or C”.

In various examples of this disclosure, “or” should be interpreted as indicating “and/or”. For example, “A or B” may include “only A”, “only B”, and/or “both A and B”. In other words, "or" should be interpreted as indicating "in addition or alternatively."

The terms used in this disclosure are only used to describe specific various examples, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, the terms "comprise" or "having" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present disclosure, it should not be interpreted in an ideal or excessively formal meaning. don't Hereinafter, various examples of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a character emotion recognition and tagging device according to an example of the present disclosure.

Referring to FIG. 1 , a character emotion recognition and tagging apparatus 100 according to an example of the present disclosure includes a transceiver 110 , at least one processor 120 and at least one memory 130 .

The transceiver 110 may be connected to the processor 120 and may transmit and/or receive wired/wireless signals. For example, the transceiver 110 may be connected to the user terminal 200 through a wireless communication network. Here, the wireless communication network may include a mobile communication network, a wireless LAN, a local area wireless communication network, and the like. For example, a wireless communication network may include LTE, LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system (GSM). for Mobile Communications) and the like. For example, the wireless communication network may include at least one of wireless fidelity (WiFi), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, near field communication (NFC), or radio frequency (RF).

The transceiver 110 may include a transmitter and a receiver. The transceiver 110 may be used interchangeably with a radio frequency (RF) unit. The transceiver 110 may transmit and receive various signals to and from the user terminal 200 under the control of the processor 120 .

The processor 120 controls the memory 130 and/or the transceiver 110 and may be configured to implement the descriptions, functions, procedures, suggestions, methods and/or operational flowcharts of the present disclosure. For example, the processor 120 may receive a radio signal through the transceiver 110 and store information included in the radio signal in the memory 130 . In addition, the processor 120 may process information stored in the memory 130 to generate a radio signal, and transmit the radio signal through the transceiver 110 .

The memory 130 may be connected to the processor 120 and may store various information related to the operation of the processor 120 . For example, memory 130 may perform some or all of the processes controlled by processor 120, or may provide instructions for performing descriptions, functions, procedures, suggestions, methods, and/or operational flowcharts of the present disclosure. It can store the software code that contains it.

Hereinafter, various operation examples of the character emotion recognition and tagging apparatus 100 will be described. The following various operation examples may be included in the operation of at least one processor 120 described above.

The processor 120 receives input image data including a character object from the user terminal 200 through the transceiver 110 . In the present disclosure, a character object may be an object corresponding to a character or an emoticon created by a creator, not a real person. The input image data includes a character object, and may include an emotion object for representing an emotional state of a corresponding character and/or a background object corresponding to a background image.

The processor 120 assigns a character object included in the input image data to one emotion category among a plurality of emotion categories based on a feature of the input image data received from the user terminal 200 and a pre-learned classification model. can be classified. In other words, the processor 120 may recognize and classify the emotion of the character object by using the characteristics of the input image data as the input data of the pre-learned classification model.

A plurality of emotion categories for emotion recognition and classification of character objects may be set or defined in various ways. For example, emotion categories include joy, eagerness, excitement, surprise, anger, fear, sadness, boredom, tired, disappointment, shyness, pouting, comfort, worry, generosity, sullenness, hunger, pain, embarrassment, and mischievousness. etc. can be set or defined.

Whether to be classified into any one of the emotion categories as in the above example may be performed based on various artificial intelligence (AI) algorithms. In other words, the learning algorithm for learning the above-described pre-learned classification model is a variety of machine learning such as supervised learning, unsupervised learning, reinforcement learning, and transfer learning. (machine learning) algorithms.

A feature for learning a classification model may be an integrated action unit (AU) of a character object. That is, the processor 120 may learn a classification model by characterizing the integrated AU of the character object, and generate a pre-learned classification model as a learning result. In the present disclosure, an AU may mean one element formed by at least one of an edge extracted from a character object and a feature point included in the edge.

Hereinafter, the unified AU defined in the present disclosure will be described in more detail.

2 to 5 are for explaining an action unit (AU) defined according to various examples of the present disclosure.

The unified AU is extracted from training image data or input image data, and may include a human AU and a character AU. Here, the training image data is image data used for learning the classification model, and may be pre-stored in the memory 130 or received from a user device other than the user terminal 200 .

The human AU may correspond to elements representing facial expressions, such as facial features, and the character AU corresponds to elements that directly or indirectly represent the emotions of a character, although not related to facial expressions. For convenience, human AU and character may be referred to as the enemy AU.

Referring to FIG. 2, the human AU is at least one of distances and ratios between a plurality of feature points included in each of the eyebrows, eyes, nose, mouth, and cheeks of a character object included in training image data or input image data for learning a classification model. It may include at least one of a plurality of eyebrow AUs, a plurality of eye AUs, a plurality of nose AUs, a plurality of mouth AUs, and a plurality of cheek AUs, which otherwise have one.

The eyebrow AU, eye AU, nose AU, mouth AU, and cheek AU may include at least one of a plurality of feature points and edges included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object. Eyebrow AU, eye AU, nose AU, mouth AU, and cheek AU may be set or defined in various ways according to emotions.

Referring to FIG. 3 , for example, a character object corresponding to the sadness emotion category includes a frowning eyebrow AU 311, a closed eye AU 321, an enlarged nose AU 331, and/or an M-shaped mouth AU 341. etc. may be included.

For example, a character object corresponding to the angry emotion category may include an angry eye AU 322 , an enlarged nose AU 332 , and/or an open mouth AU 342 .

For example, a character object corresponding to the joy emotion category may include a smiling eye AU 323 and/or a U-shaped mouth AU 343.

Referring to FIGS. 4 and 5 , the character AU may include at least one of a background object AU and an emotion object AU included in training image data for learning a classification model or input image data. Each of the background object AU and the emotion object AU may include at least one of a plurality of feature points and edges included in each of the above-described background object and emotion object. The background object AU and the emotion object AU may be set or defined in various ways according to emotions.

More specifically, the emotion object AU may correspond to an object including an edge at least partially overlapping an edge of a character object among other objects other than the character object in the training image or input image data. Also, the background object AU may correspond to objects other than the character object and the emotion object AU in the training image data or the input image data.

For example, the emotion object AU may include a tear AU 411 , an anger display AU 412 , an indifference AU 413 , and a sweat drop AU 414 .

For example, the background object AU may include a flower pattern AU 421 and a text AU 422. Here, the text AU 422 may include, for example, text indicating joy, such as 'I'm so happy', as shown in the case of a joy emotion category.

The above-mentioned unified AUs are merely exemplary, and various other factors capable of effectively representing corresponding emotions for each emotion category may be set or defined as the unified AU.

6 is for explaining emotion category labeling.

Referring to FIG. 6 , each of the aforementioned unified AUs may be labeled with at least one of a plurality of emotional categories. In this case, the training image data may be, for example, (emotion category, unified AU) pairs. That is, when learning a classification model, supervised learning using an integrated AU labeled in an emotion category as training image data may be used.

The processor 120 may generate tagged input image data by tagging the input image data with one classified emotion category. Here, the tagged input image data may include both the input image data and one classified emotion category information.

The processor 120 may transmit the tagged input image data generated through the transceiver 110 to the user terminal 200 .

According to the character emotion recognition and tagging apparatus 100 described above, emotions of objects included in creative works such as characters or emoticons can be effectively recognized and classified. Specifically, the apparatus 100 for recognizing and tagging character emotions may more accurately recognize and classify emotions of character objects by considering character elements in features used in AI algorithms such as machine learning.

Alternatively, according to another example of the present disclosure, the apparatus 100 for recognizing and tagging character emotions may select a feature to be used for emotion classification when emotion classification of input image data. Here, the selected feature may be any one of character AUs.

Specifically, the processor 120 may receive classification option data from the user terminal 200 through the transceiver 110 . Here, the classification option data may be selection information for selecting at least one of the background object AU and the emotion object AU.

The processor 120 may select an AU corresponding to the received selection information from among the background object AU and the emotion object AU.

The processor 120 may classify the character object into one emotion category based on the human AU, the AU corresponding to the received selection information, and the pre-learned classification model.

Accordingly, the character emotion recognition and tagging device 100 according to another example of the present disclosure may allow a creator to select a feature for emotion recognition and classification, and perform emotion recognition and classification based on the selected feature, thereby enabling a machine to perform emotion recognition and classification. can increase learning satisfaction.

Here, the processor 120 may additionally search for similar images corresponding to emotional categories classified and tagged for market research of content, and transmit the searched similar images to the user terminal 200 through the transceiver 110. .

In addition, the processor 120 may determine whether content copyright is violated by checking the similarity between the searched similar image and the uploaded image. Accordingly, if it is determined that content copyright is violated through the similarity test, the processor 120 may transmit a separate warning message to the user terminal 200 through the transceiver 110 .

Alternatively, according to another example of the present disclosure, the apparatus 100 for recognizing and tagging character emotions may perform reclassification by adding features according to the evaluation degree of classified emotions.

Specifically, the processor 120 may receive evaluation data for tagged input image data through the transceiver 110 . Here, the evaluation data may be data obtained by quantifying the user's satisfaction with the primarily classified emotion.

If the received evaluation data is less than a preset threshold, the processor 120 classifies the character object into one emotion category based on at least one of a human AU, a background object AU, and an emotion object AU and a pre-learned classification model. can For example, when the maximum value of the evaluation value corresponding to the evaluation data is 10 points, the predetermined threshold value is 5 points, and the value of the evaluation value corresponding to the received evaluation data is 4 points, the processor 120 determines the human AU, Emotion classification may be performed again by characterizing at least one of the background object AU and the emotion object AU.

According to another example of the present disclosure described above, the apparatus 100 for recognizing and tagging character emotions may be particularly useful when primarily performed emotion classification is characterized by only human AUs. That is, the character emotion recognition and tagging apparatus 100 primarily performs emotion classification by characterizing the human AU, and then, when the user's satisfaction with the classified emotion is low, character emotion recognition and tagging apparatus 100 characterizes the character AU together and performs emotion classification. By performing the classification again, user satisfaction can be increased.

7 is a schematic diagram of a character emotion recognition and tagging system according to an example of the present disclosure.

Hereinafter, detailed descriptions of overlapping parts with those described above will be omitted.

Referring to FIG. 7 , a character emotion recognition and tagging system according to an example of the present disclosure includes a character emotion recognition and tagging device 100 and a user terminal 200 .

The character emotion recognition and tagging apparatus 100 classifies emotions of character objects included in input image data received from the user terminal 200 .

The user terminal 200 may exchange data with the character emotion recognition and tagging device 100 . For example, the user terminal 200 may transmit input image data for emotion classification. At this time, the user terminal 200 may input design information such as the image style, color, shape, and drawing style of the character and transmit the corresponding design information together. For example, the image style may include a still image or a moving image.

When design information is received from the user terminal 200 , the character emotion recognition and tagging apparatus 100 may match the received design information to input image data and store it in the memory 130 .

8 is a flowchart of a character emotion recognition and tagging method according to an example of the present disclosure.

Referring to FIG. 8 , in S110 , the apparatus 100 for recognizing and tagging character emotions may receive input image data including a character object from the user terminal 200 .

In operation S120 , the apparatus 100 for recognizing and tagging character emotions may classify the character object into one emotion category among a plurality of emotion categories based on the characteristics of the input image data and the pre-learned classification model. Here, the pre-learned classification model may be learned by characterizing the integrated AU of the character object included in the training image data.

The unified AU may include a human AU and a character AU. In this case, the human AU includes a plurality of eyebrow AUs, a plurality of eye AUs, having at least one of the distances and ratios between the plurality of feature points included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object included in the training image data. It may include at least one of a plurality of nose AUs, a plurality of mouth AUs, and a plurality of cheek AUs.

The character AU may include at least one of a background object AU included in the training image data and an emotion object AU included in the training image data. In this case, the emotion object AU may correspond to an object including an edge at least partially overlapping an edge of a character object among other objects other than the character object in the training image data. The background object AU may correspond to objects other than the character object and the emotion object AU in the training image data.

In operation S130 , the apparatus 100 for recognizing and tagging character emotions may tag input image data with one emotion category to generate tagged input image data.

In S140 , the apparatus 100 for recognizing and tagging character emotions may transmit the tagged input image data to the user terminal 200 .

Here, the character emotion recognition and tagging method according to another example of the present disclosure may further include a step of selecting a feature to be used for emotion classification.

For example, the method for recognizing and tagging character emotions includes receiving classification option data from the user terminal 200 by the apparatus 100 for recognizing and tagging character emotions, selecting an AU corresponding to selection information among a background object AU and an emotion object AU. The step of selecting and classifying the character object into one emotion category based on the human AU, the AU corresponding to the selection information, and the pre-learned classification model may be further included.

In this case, the classification option data may include selection information for selecting at least one of the background object AU and the emotion object AU.

Here, the character emotion recognition and tagging method according to another example of the present disclosure may further include a step of selecting a feature to be used for emotion classification.

For example, the method for recognizing and tagging character emotions includes receiving evaluation data for tagged input image data by the apparatus 100 for recognizing and tagging character emotions, and if the evaluation data is less than a preset threshold, human AU and background object The method may further include classifying the character object into one emotion category based on at least one of the AU and the emotion object AU and a pre-learned classification model.

Since the examples of the proposed schemes in the above description may also be included as one of the implementation methods of the present disclosure, it is obvious that they can be regarded as a kind of proposed schemes. In addition, the above-described proposed schemes may be implemented independently, but may also be implemented in a combination (or merged) form of some proposed schemes.

The examples of the present disclosure disclosed as described above are provided to enable those skilled in the art to implement and practice the present disclosure. Although described above with reference to examples of the present disclosure, a person skilled in the art may variously modify and change the examples of the present disclosure. Thus, the present disclosure is not intended to be limited to the examples set forth herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

100: emotion recognition and tagging device
110: transceiver 120: processor
130: memory
200: user terminal

Claims (15)

transceiver;
at least one processor; and
at least one memory operatively connected to the at least one processor to store at least one instruction that causes the at least one processor to perform operations;
The above actions are:
Receiving input image data including a character object from a user terminal through the transceiver,
Classifying the character object into one emotion category among a plurality of emotion categories based on a feature of the input image data and a pre-learned classification model;
tagging the input image data with the one emotion category to generate tagged input image data; and
Transmitting the tagged input image data to the user terminal through the transceiver;
The pre-learned classification model is learned by characterizing an integrated action unit (AU) of a character object included in the training image data stored in the at least one memory,
The integrated AU includes a human AU and a character AU,
The human AU includes a plurality of eyebrow AUs, a plurality of eye AUs having different distances and ratios of at least one of a plurality of feature points included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object included in the training image data; At least one of a plurality of nose AUs, a plurality of mouth AUs, and a plurality of cheek AUs;
The character AU includes at least one of a background object AU included in the training image data and an emotion object AU included in the training image data,
The above actions are:
classifying the human AU among the integrated AUs into the one emotion category as the characteristic;
receiving evaluation data for the tagged input image data through the transceiver; and
Further comprising classifying the character object into the one emotion category based on the human AU, the character AU, and the pre-learned classification model if the evaluation data is less than a preset threshold value.
Character emotion recognition and tagging device.
delete delete delete According to claim 1,
The emotion object AU corresponds to an object including an edge at least partially overlapping an edge of the character object among objects other than the character object in the training image data, and
The background object AU corresponds to objects other than the character object and the emotion object AU in the training image data.
Character emotion recognition and tagging device.
According to claim 1,
The above actions are:
classification option data from the user terminal through the transceiver, wherein the classification option data includes selection information for selecting at least one of the background object AU and the emotion object AU; receive,
selecting an AU corresponding to the selection information from among the background object AU and the emotion object AU; and
Further comprising classifying the character object into the one emotion category based on the human AU, the AU corresponding to the selection information, and the pre-learned classification model.
Character emotion recognition and tagging device.
delete According to claim 1,
Each of the integrated AUs is labeled with at least one of the plurality of emotional categories,
Character emotion recognition and tagging device.
A character emotion recognition and tagging method performed by a character emotion recognition and tagging device,
Receiving input image data including a character object from a user terminal;
classifying the character object into one emotion category among a plurality of emotion categories based on a feature of the input image data and a pre-learned classification model;
tagging the input image data with the one emotion category to generate tagged input image data; and
Transmitting the tagged input image data to the user terminal,
The pre-learned classification model is learned by characterizing an integrated action unit (AU) of a character object included in training image data,
The integrated AU includes a human AU and a character AU,
The human AU includes a plurality of eyebrow AUs, a plurality of eye AUs having different distances and ratios of at least one of a plurality of feature points included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object included in the training image data; At least one of a plurality of nose AUs, a plurality of mouth AUs, and a plurality of cheek AUs;
The character AU includes at least one of a background object AU included in the training image data and an emotion object AU included in the training image data,
The character emotion recognition and tagging method:
classifying the human AU among the integrated AUs as the one emotion category;
receive evaluation data for the tagged input image data; and
Further comprising classifying the character object into the one emotion category based on the human AU, the character AU, and the pre-learned classification model if the evaluation data is less than a preset threshold value.
Character emotion recognition and tagging methods.
delete According to claim 9,
The emotion object AU corresponds to an object including an edge at least partially overlapping an edge of the character object among objects other than the character object in the training image data, and
The background object AU corresponds to objects other than the character object and the emotion object AU in the training image data.
Character emotion recognition and tagging methods.
According to claim 9,
classification option data from the user terminal, wherein the classification option data includes selection information for selecting at least one of the background object AU and the emotion object AU; receive;
select an AU corresponding to the selection information from among the background object AU and the emotion object AU; and
Further comprising classifying the character object into the one emotion category based on the human AU, the AU corresponding to the selection information, and the pre-learned classification model.
Character emotion recognition and tagging methods.
delete According to claim 9,
Each of the integrated AUs is labeled with at least one of the plurality of emotional categories,
Character emotion recognition and tagging methods.
user terminal; and
A character emotion recognition and tagging device for classifying the emotion of a character object included in the input image data received from the user terminal,
The character emotion recognition and tagging device:
transceiver;
at least one processor; and
at least one memory operatively connected to the at least one processor to store at least one instruction that causes the at least one processor to perform operations;
The above actions are:
Receiving the input image data from the user terminal through the transceiver;
Classifying the character object into one emotion category among a plurality of emotion categories based on a feature of the input image data and a pre-learned classification model;
tagging the input image data with the one emotion category to generate tagged input image data; and
Transmitting the tagged input image data to the user terminal through the transceiver;
The pre-learned classification model is learned by characterizing an integrated action unit (AU) of a character object included in the training image data stored in the at least one memory,
The integrated AU includes a human AU and a character AU,
The human AU includes a plurality of eyebrow AUs, a plurality of eye AUs having different distances and ratios of at least one of a plurality of feature points included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object included in the training image data; At least one of a plurality of nose AUs, a plurality of mouth AUs, and a plurality of cheek AUs;
The character AU includes at least one of a background object AU included in the training image data and an emotion object AU included in the training image data,
The above actions are:
classifying the human AU among the integrated AUs into the one emotion category as the characteristic;
receiving evaluation data for the tagged input image data through the transceiver; and
Further comprising classifying the character object into the one emotion category based on the human AU, the character AU, and the pre-learned classification model if the evaluation data is less than a preset threshold value.
Character emotion recognition and tagging system.

Images