KR20220053358A - Method and apparatus for analyzing sentiment related to object - Google Patents

Abstract

The present invention relates to a device for analyzing a sentiment related to an object, which includes: a video collecting unit collecting a reaction video of a person viewing a target video when information on the target video is received; a frame extracting unit extracting each frame included in the collected reaction video and the target video; a sentiment analysis unit identifying a frame that is included in the target video and induces a change in the person's sentiment based on the expression of the person in each frame extracted from the collected reaction video; an object analysis unit determining an object that induces a change in the person's sentiment among the frames in the target video and quantifying the degree of change in the sentiment of the person of the determined object based on the degree of change in the sentiment; and an information providing unit storing the determined object and the degree of change in the sentiment of the person corresponding to the determined object and outputting the stored content.

Description

Method and apparatus for analyzing emotional objects

The present invention relates to a method and apparatus for analyzing an emotional object, and more particularly, to a method and apparatus for analyzing a viewer's emotion for an object appearing in a target video.

As the spread of smart phones increases and the speed of wireless communication improves, the consumption of video content is gradually increasing. At the same time, the proportion of viewers who watched video content expressing their emotions in the form of a video instead of in the form of a text such as a comment is also increasing. Here, in order for viewers to express their emotions, an image produced by filming their viewing of video content is referred to as a reaction video.

On the other hand, if it is possible to understand the emotional change of viewers who have watched the target video, it may be easier to produce video content that can elicit a specific emotion from the viewers. Therefore, research is underway on a technology to quickly and effectively analyze the emotions of the people in the reaction video, identify the object to which viewers respond emotionally in the target video, and analyze the relationship between the specific object and the viewer's emotion. .

An object of the present invention is to derive a result of analyzing a relationship between a specific object included in a target video and the emotion of a person in a reaction video by providing a method and apparatus for analyzing a viewer's emotions for a specific object appearing in a target video.

In addition, based on the relationship between a specific object and the emotion of a person in the reaction video, we want to build an object dictionary for each emotion.

The emotional object analysis apparatus of the present invention is configured to extract, when information on a target video is received, an image collection unit that collects a reaction video shot by a person watching the target video, and each frame included in the collected reaction video and the target video a frame extraction unit, an emotion analysis unit for identifying a frame that induces a change in emotion of the person from among the frames included in the target video based on the facial expression of the person included in each frame extracted from the collected reaction video; and an object analyzer configured to determine an object inducing a change in emotion of the person from among the frames included in the target video based on the degree of change in emotion, and quantify the degree of change in emotion of the person in the determined object; and an information providing unit for storing the determined object and the degree of emotional change of the person corresponding to the determined object and outputting the stored content.

According to an embodiment, the information providing unit may generate and store an object dictionary for each emotion by arranging the objects in an order of increasing the degree of emotion change for a preset emotion item.

According to an embodiment, after receiving the reaction video collected from the image collecting unit, the emotional object analysis apparatus further includes a pre-processing unit, which extracts only a section in which a person included in the collected reaction video shows a reaction and delivers it to the frame extraction unit. may include

According to an embodiment, the emotional object analysis apparatus determines the similarity between the extracted frames based on the object appearing in each frame with respect to the frames extracted from the target video, and divides the similar frames into the same section according to the determination result. It may further include a section divider for merging and transferring the to the frame identification unit.

According to an embodiment, the emotion analysis unit divides the emotion into a plurality of emotion items, derives probability values of the plurality of emotion items according to the expression of the person in each frame, and a plurality of frames in the reaction video including each frame After calculating the average change amount for each emotion item of , the frame having a value greater than the average change amount for each emotion item is classified as a meaningful frame and the meaningful frame is transmitted to the object analysis unit.

According to an embodiment, the object analysis unit calculates an average change rate value for each emotion item of a person in a reaction video corresponding to the frames before and after the first frame in which each identified object appears, and then the average change amount for each emotion item and analyzing the emotion of the person with respect to the object based on probability values of the plurality of emotion items of the first frame.

According to an embodiment, when the first frame includes multiple objects, the object analysis unit sets the ranking of the objects on the emotion value based on the presence or absence of objects in the frames before and after the first frame for each object. can be one

According to an embodiment, when setting the rank of multiple objects, the object analysis unit gives the highest priority to the rank of the object in which the corresponding object does not appear in the front and rear frames, and ranks the objects in which the corresponding object appears in only one of the front and rear frames. can be done with

A method for analyzing an emotional object by an apparatus according to another embodiment of the present invention includes: when information on a target video is received, collecting a reaction video captured by a person watching the target video; extracting each frame included in the collected reaction video and the target video; identifying a frame that induces a change in emotion of the person from among the frames included in the target video based on the facial expression of the person included in each frame extracted from the collected reaction video; determining an object inducing a change in emotion of the person from among frames included in the target video based on the degree of change in emotion, and quantifying the degree of change in emotion of the person in the determined object; and storing the determined object and the degree of emotional change of the person corresponding to the determined object, and outputting the stored content.

According to an embodiment, the method for analyzing emotion objects may further include arranging objects in an order of increasing degree of emotion change for a preset emotion item to generate and store an object dictionary for each emotion.

According to an embodiment, the step of extracting each frame may be a step of extracting only a section in which the person included in the collected reaction video shows a reaction after receiving the collected reaction video.

According to an embodiment, the extracting of each frame may include determining a similarity between the extracted frames based on an object appearing in each frame with respect to the frames extracted from the target video, and according to the determination result, similar It may include merging frames.

According to an embodiment, the step of identifying a frame inducing a change in emotion of a person may include dividing the emotion into a plurality of emotion items, and then deriving a probability value of the plurality of emotion items according to the expression of the person in the first frame to do; calculating an average change amount for each emotion item of a plurality of frames in the reaction video including the first frame; and classifying a frame having a value greater than the average change amount for each emotion item as a meaningful frame.

According to an embodiment, the step of quantifying the degree of emotional change of the person of the determined object may include an average change rate value for each emotion item of a person in a reaction video corresponding to the front and rear frames of the first frame in which each identified object appears. After calculating , the emotion of the person with respect to the object may be analyzed based on the average change amount for each emotion item and the probability values of the plurality of emotion items of the first frame.

According to an embodiment, quantifying the degree of emotional change of the person of the determined object may include, when the first frame includes multiple objects, the presence or absence of objects in frames before and after the first frame for each object It may include the step of setting the ranking of the object on the emotion value based on the.

According to one embodiment, in the step of setting the rank of the object, the priority of the object in which the corresponding object does not appear in the front and rear frames is given priority, and the ranking of the object in which the corresponding object appears only in one of the front and rear frames is the next priority. may be doing

The present invention provides a computer-readable storage medium storing a computer program in which the above-described method is performed when the computer program is executed by a processor.

According to an embodiment disclosed in the present invention, each object in the target video according to the analysis result of the emotional object derived by the present invention without conducting individual research on the emotions felt by viewers who have watched the content in the fields related to advertisement and publicity It has the effect of understanding the relationship between and the viewer's emotions.

In addition, according to another embodiment disclosed in the present invention, when content creators want to elicit a desired emotion from viewers, they can produce content based on the emotional object analysis result derived by the present invention. In this case, it is possible for content creators to produce content that can more easily elicit a desired audience response.

1 is a diagram for explaining a target video and a reaction video.
2 is a block diagram of an apparatus for analyzing an emotion object for analyzing the emotion of a person in a reaction video to an object included in the target video according to an embodiment of the present invention.
3 is a block diagram of an emotion object analysis apparatus for analyzing a person's emotion in a reaction video to an object included in a target video according to another embodiment of the present invention.
4 is a view for explaining the operation of the section dividing unit according to an embodiment of the present invention.
5 is a diagram for explaining a method of deriving a probability value of each emotion item by an emotion analysis unit according to an embodiment of the present invention.
6 is a time series graph of emotion values of a target video analyzed by an emotion analysis unit according to an embodiment of the present invention.
7 is a diagram illustrating an example of a capital code of an algorithm for analyzing the emotional influence of an object according to an embodiment of the present invention.
8 to 10 are diagrams for explaining experimental results of the apparatus for analyzing an emotional object according to an embodiment of the present invention.
11 is a flowchart illustrating a method for analyzing an emotion object according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a target video and a reaction video.

The target video means a video viewed by the viewer, and the reaction video means a video captured by the viewer watching the target video. Conventional viewers express their emotions toward the target video in the form of text, such as comments, but recently, the number of viewers expressing their emotions toward the target video in the form of a video is increasing.

Prior to the description of the present invention, the emotional object referred to herein means an object that induces a specific emotion with respect to the corresponding object.

1 is a diagram illustrating an example of a reaction video 110 in which a target video 120 and a viewer 115 watching the target video 120 are captured. Referring to FIG. 1 , when the target video 120 is a music video of the female singer 125 , the viewer 115 watching the target video 120 may express their pleasant emotions through the reaction video 110 . In this case, the singer 125 appearing in the target video 120 can be viewed as one object included in the target video.

On the basis of the emotion of the person in the reaction video watching the target video according to an embodiment of the present invention, the method and apparatus for analyzing an emotion object collects the reaction video for the target video, and then measures the change in the emotion of the person in the reaction video , from among a plurality of objects included in the target video according to the measurement result, a specific object that induces a specific emotion from the viewer may be determined, and the relationship between each emotion item for each object may be analyzed. In addition, it is possible to build an object dictionary for each emotion defined based on the relationship between emotion items for each object.

2 is a block diagram of an apparatus for analyzing an emotion object for analyzing the emotion of a person in a reaction video to an object included in the target video according to an embodiment of the present invention.

The emotional object analyzing apparatus 200 according to an embodiment may include an image collecting unit 210 , a frame extracting unit 220 , an emotion analyzing unit 230 , an object analyzing unit 240 , and an information providing unit 250 . can Components included in the emotional object analysis apparatus 200 are divided into functions for convenience of explanation, but some components may be merged into one component, and the emotional object analysis apparatus 200 may have other components. It is apparent to those skilled in the art that it may further include elements.

When information on the target video is received, the image collection unit 210 may collect a reaction video in which a person watching the target video is captured. In this case, the image collection unit 210 may collect one or more reaction videos for the target video. Also, the information about the target video received by the image collection unit 210 may be the target video itself or a Uniform Resource Locator (URL) indicating the location of the target video.

The frame extraction unit 220 may extract a frame by separating at least one of a target video and a reaction video at a predetermined time interval. For example, the frame extractor 220 may extract a jpg format frame from each of the target video and the reaction video at 0.1 second intervals.

The emotion analyzer 230 analyzes the emotion of the person based on the expression of the person included in each frame extracted from the collected reaction video and analyzes the degree of emotional change.

In an embodiment, the emotion analysis unit 230 may include an expression analysis module 232 and an emotion variation measurement module 234 .

The expression analysis module 232 may analyze the emotion of the person based on the expression of the person included in the first frame extracted from the collected reaction video.

The expression analysis module 232 according to an embodiment divides the emotion into a plurality of emotion items, and then derives probability values of the plurality of emotion items according to the expression of the person in the first frame.

The emotion change amount measurement module 234 generates a time series graph based on the derived probability value of emotion and measures the emotion change amount. The average amount of change for each emotion is derived, and a section (or frame) with a greater amount of change is determined to be significant and selected. All frame images of the selected section (or frame) are transmitted to the object analysis unit 240 .

The object analysis unit 240 determines an object that induces a change in emotion of the person from among the frames included in the target video based on the degree of change in emotion, and quantifies the degree of change in emotion of the person in the determined object.

The object analyzer 240 may include an object determination module 242 and an object emotion value extraction module 244 .

The object determination module 242 according to an embodiment extracts an object included in the meaningful frame selected by the emotion variation measurement module 234 . The object determination module 242 may perform an operation based on a deep learning algorithm.

The object emotion value extraction module 244 according to an embodiment quantifies the degree of change in emotion of the person of the determined object by analyzing the influence of the object on emotion.

The information providing unit 250 stores the object determined by the object analysis unit 240 and the degree of emotional change of the person corresponding to the determined object, and outputs the stored content.

The information providing unit 250 according to an exemplary embodiment creates and stores an object dictionary for each emotion by arranging objects in the order of the greatest degree of emotion change for a preset emotion item.

The object dictionary for each emotion includes a category into which a target video is classified, an object name for each emotion item, an emotion change rate of each object, an object rank, and the like. The ranking of objects will be described later with reference to FIG. 9 .

3 is a view for explaining an emotion object analysis apparatus for analyzing the emotion of a person in a reaction video to an object included in the target video according to another embodiment of the present invention.

The emotional object analysis apparatus 300 according to another embodiment of the present invention includes an image collecting unit 310 , a preprocessing unit 315 , a frame extracting unit 320 , a section dividing unit 325 , an emotion analyzing unit 330 , It may include an object analysis unit 340 and an information providing unit 270 . Components included in the emotional object analysis apparatus 300 are divided into functions for convenience of explanation, but some components may be merged into one component, and the emotional object analysis apparatus 200 may have other components. It is apparent to those skilled in the art that it may further include elements.

Referring to FIG. 3 , when information on a target video to be analyzed is received, the image collection unit 310 of the emotional object analysis apparatus 300 may collect a target video and a corresponding reaction video.

At this time, the collected reaction video may be transmitted to the pre-processing unit 315, and the pre-processing unit 315 extracts a portion of the reaction video including a figure in which a person in the reaction video reacts to the target video to extract a frame It can be passed to (320). Accordingly, since the emotional object analysis apparatus 300 can perform emotion analysis on only a part of the reaction video including the person responding to the target video, there is an effect that can reduce the consumption amount.

The frame extraction unit 320 may extract a frame by separating at least one of a target video and a reaction video at a predetermined time interval. For example, the frame extraction unit 320 may extract a jpg format frame from each of the target video and the reaction video at 0.1 second intervals.

Thereafter, the section separator 325 receives each frame of the target video from the frame extractor 320 and merges the similar frames into the same section based on the degree of similarity between frames included in the target video to perform the emotion analysis unit 330 . can be forwarded to When similar frames of the target video are merged into the same section through the section separator 325 , the number of frames required for analysis in the target video is reduced, thereby reducing the amount of resources consumed.

Meanwhile, a criterion for determining the similarity between frames of a target video will be described in detail below.

4 is a view for explaining the operation of the section dividing unit according to an embodiment of the present invention.

The section dividing unit (325 in FIG. 3) according to an embodiment of the present invention may be performed based on the SIFT algorithm of OpenCV, but the algorithm that can be used in the section dividing unit (325 in FIG. 3) is not limited thereto. is apparent to those skilled in the art.

If the section dividing unit (325 in FIG. 3) is based on the SIFT algorithm of OpenCV, objects included in the front and rear frames of the target video extracted by the frame extracting unit (320 in FIG. 3) are the same size in the background. only changed or rotated, the corresponding front and rear frames are determined to be similar frames and may be merged into the same section.

Referring to FIG. 4 , since the object 410 of the previous frame and the object 420 of the next frame included in the target video hardly change on the same background, the Frames before and after are merged into the same section and transmitted to the emotion analyzer ( 340 of FIG. 3 ).

Referring back to FIG. 3 , the emotion analysis unit 330 may derive a probability value of each emotion item of the person in each frame by dividing the expression of the person appearing in the reaction video into eight emotion items. Meanwhile, a method of deriving a probability value of each emotion item by dividing the expression of a person into a plurality of emotion items will be described in detail below.

5 is a diagram for explaining a method of deriving a probability value of each emotion item by an emotion analysis unit according to an embodiment of the present invention.

The emotion analysis unit (330 in FIG. 3) according to an embodiment of the present invention may utilize the Face API of Microsoft Azure to analyze the emotion of a person in the reaction video, but the emotion analysis unit (330 in FIG. 3) may use it It will be apparent to those skilled in the art that the available algorithm is not limited thereto.

For example, if the probability value of the emotion item of the person included in the frame of FIG. 5(a) is derived using the Face API algorithm, the same result as that of FIG. 5(b) can be obtained. Referring to Figure 5 (b), the emotion items derived through the Face API algorithm are anger, pleasure, sadness, disgust, contempt, fear, It may include surprise and neutral emotion, and the probability value of each emotion item may be expressed as an integer value between 0 and 1. However, the number of the above-described emotion items and specific content of the emotion items are not limited thereto.

The emotional object analysis apparatus (300 in FIG. 3) according to an embodiment of the present invention sequentially inputs each frame of the reaction video extracted by the frame extraction unit (320 in FIG. 3) to the Face API algorithm, so that each frame Probability values of 8 emotions can be obtained with respect to the included person. Thereafter, an average emotion probability value for each section may be calculated based on the separated sections of the target video received from the section separator ( 325 of FIG. 3 ).

Referring back to FIG. 3 , the emotion analysis unit 330 may generate a time series graph based on the probability value of each emotion item of a person included in each frame of the reaction video and measure the change in emotion.

In addition, by deriving the average amount of change for each emotion item with respect to a plurality of frames of the reaction video, when the probability value of the emotion item for each frame has a larger amount of change than the average amount of change, the corresponding frame of the target video induces emotional change identified by frame.

Thereafter, the emotional object analyzing apparatus 300 transmits the frame of the target video identified as inducing the emotional change to the object analyzing unit 340 .

6 is a time series graph of emotion values of a target video analyzed by an emotion analysis unit according to an embodiment of the present invention. In the graph of FIG. 6 , the x-axis represents an image section, and the y-axis represents the degree of emotion (quantified emotion value).

Meanwhile, the color corresponding to each emotion item of the graph shown in FIG. 6 is shown in Table 1.

emotion colour emotion colour anger blue happiness purple contempt green neutral light green disgust Red sadness Black color fear Sky blue surprise Orange

In (a) of FIG. 6 , it can be seen that the happy emotion is dominant except for the neutral emotion. In addition, in Fig. 6(a), while happy emotions are large at the beginning, as time goes by, the values of happy decrease, the emotions of anger and surprise greatly increase, and then it is possible to observe the flow of emotions in which the values of happy increase again. can In (b) of FIG. 6 , it can be observed that the values of all emotions largely fluctuate. In addition, in (b) of FIG. 6 , it can be seen that the target video has a generally gloomy atmosphere, as the value of sadness is widely distributed throughout.

Referring again to FIG. 3 , the object analysis unit 340 extracts an object appearing in a meaningful frame. In this case, the object analyzer 340 according to an embodiment may apply the YOLO algorithm to detect an object in a corresponding frame of the target video, but an algorithm that can be used for object detection is not limited thereto.

In addition, the object analysis unit 340 according to an embodiment analyzes the influence of the object on the emotion and quantifies the degree of change in the person's emotion of the determined object.

The object analysis unit 340 must analyze the rate of change of emotion of a person (viewer) of the reaction video corresponding to the front and rear sections (or frames) in order to analyze the influence of the object appearing in the meaningful section on emotion.

7 is a diagram illustrating an example of a capital code of an algorithm for analyzing the emotional influence of an object according to an embodiment of the present invention.

As can be seen with reference to FIG. 7 , the emotion influence analysis algorithm analyzes the emotion value of the object based on the presence or absence of the object in the front and rear sections. More specifically, the emotion influence analysis algorithm compares the number of objects in the front and rear sections, classifies them into three categories: presence of front and rear, only one of the front or back, and neither exists, and then extracts the emotion value in the classified section. .

First, the input data is a number of meaningful sections and character string information for all sections. For all sections, the object array is created by composing a two-dimensional array with a set of one-dimensional arrays of objects for each section. After that, if the object exists for all significant sections, the front and back sections are checked. Objects detected in the relevant section are classified into three categories: “AllSection” when they appear in all sections before and after, “FrontOrRearSection” when they appear in only one section before and after, and “NotAnySection” when they do not appear in either section before or after. do. If the object in the corresponding section is classified as “AllSection”, it is determined that the meaningful rate of change of emotion in the corresponding section is due to the detected object, and the object is removed. When the object in the corresponding section is classified as “FrontOrRearSection”, the emotion change rate value with the section in which the object does not appear among the front and rear sections is calculated and applied to the emotion value extractor of the object. When the object in the corresponding section is classified as “NotAnySection”, the emotion change rate values of both the front and rear sections are calculated, averaged, and applied to the emotion value extractor of the object. In addition, in the case of multiple objects with one or more objects, the highest values are applied in the order of "NotAnySection", "FrontOrRearSection", and "AllSection" according to the classified category to rank.

3, the information providing unit 350 stores the object determined by the object analysis unit 340 and the degree of emotional change of the person corresponding to the determined object, and outputs the stored content as necessary. do.

In addition, the information providing unit 350 according to an embodiment sorts the objects in the order of the greatest degree of emotional change for a preset emotion item, creates and stores an object dictionary for each emotion, and outputs the stored content as necessary. .

On the other hand, if the analyzed result is stored in the form of a table as in the above-described example, it is possible to list and provide recommendations for each emotion item corresponding to each object in the content to the user.

8 to 10 are diagrams for explaining experimental results of the apparatus for analyzing an emotional object according to an embodiment of the present invention.

In order to derive the results of FIGS. 8 to 10 , one deep learning server including an object analysis unit was used. The specifications of the server are Intel Xeon E5-2630V3, octa-core, 20MB of cache, and the speed is 2.4GHz. The software used for the experiment was Python 3.5, YOLO v2, CUDA 8.0, and cuDNN v5.1 were used for deep learning of the object analysis part, and OpenCV version 3.2 was used as the software for implementing the section division part.

In the experiment, the target videos were 'IU - Palette' music video (hereinafter, 'first target video'), 'Taeyeon - something new' music video (hereinafter 'second target video'), and 'BTS - Dope' music video (hereinafter referred to as 'second target video'). '3rd target video'), 'Billie Eilish - Bury a friend' music video (hereinafter 'fourth target video'), and 'marshmello - happier' music video (hereinafter 'fifth target video').

8 shows the average change for each of the 8 emotion items that the person in each of the 10 reaction videos shows according to the time sequence of each target video for 5 target videos, and averaged again for each section divided by the SIFT algorithm. This is a graph of time-series sentiment values for each section. Since some frames were merged into the same section according to the similarity of the frames, as Table 2 shows the number of meaningful sections according to the target video, the first target video has 57 sections, the second target video has 13 sections, and the second target video has 13 sections. The 3 target video is divided into 29 sections, the fourth target video is divided into 91 sections, and the fifth target video is divided into 90 sections.

Content significant interval Korean singer IU - palette (1st video) 57 Taeyeon - something new (2nd video) 13 BTS - dope (3rd video) 29 foreign singer Bilie Eilish - Bury a friend (4th video) 91 marshmello - happy (5th video) 90

Fig. 8 (a) is a bar graph of a significant section of the 'IU - Palette' music video. As shown in Table 2, the first video has 57 significant sections. It can be seen that happiness and sadness values are large overall in 57 significant sections. In particular, it can be seen that the happiness value increases rapidly after frame 1300, and this section seems to be because the face part of the singer occupies most of it.

8(b) is a histogram of a significant section of the music video 'Taeyeon - something new'. As shown in Table 2, the music video for Taeyeon - something new' has 57 meaningful sections. In 57 meaningful sections, happiness, sadness, and surprise occupy a large part overall. There are a total of 5 meaningful sections of the content, and it is estimated that there are fewer meaningful sections compared to other content. Among the significant sections, especially, surprise in section 136~151, the first section, shows the largest value.

Fig. 8 (c) is a bar graph of meaningful sections of the music video for 'BTS - Dope'. As shown in Table 2, the third video has 29 significant sections. It can be seen that overall happiness, sadness, and surprise emotions were large in 29 meaningful sections. Since the face of the singer occupies most of the third video unlike other target videos, it is estimated that the change in emotion of a person (viewer) in the reaction video viewing the third video depends on the viewer's preference for singers.

FIG. 8(d) is a histogram of significant sections of Billie Eilish - Bury a friend' music video. As shown in Table 2, the fourth video has 91 significant sections. It can be seen that the fourth video has a large number of meaningful sections, indicating a large change in the viewer's emotions. In addition, it can be seen that in the fourth video, the viewer's emotional value also appears in a variety of ways compared to other target content.

8(e) is a histogram of a significant section of the music video 'marshmello - happier'. As shown in Table 2, the fourth video has 91 significant sections. In FIG. 8(e), it can be seen that this graph also has a large number of meaningful sections, indicating a large change in the viewer's emotions. Overall, it can be seen that the emotions of happiness, sadness, and surprise are large. In particular, the happiness value in the early part shows a large value, which can be inferred because the corresponding section is a birthday party scene.

9 is a view showing a representative image of a first section among significant sections of a first target video and a fourth target video of the present invention, and FIG. 10 is a continuous view of the first section of the first video and fourth video of the present invention. This is a bar graph of emotion values in the front and back sections.

Figure 9 (a) is a view showing representative images of 835 ~ 841, which is one of the significant sections of the first target video of the present invention, Figure 9 (b) is one of the significant sections of the fourth target video of the present invention It is a drawing showing representative images of people 177~196.

Figure 10 (a) is a bar graph of the emotion value of the continuous front and rear sections of the first section of Fig. 9 (a), Fig. 10 (b) is a continuous section of the first section of Fig. 9 (b) This is a bar graph of emotion values in the front and back sections.

Referring to FIGS. 9A and 10A , when the emotion change rate of the first section of the first target video is calculated, negative numbers are obtained from all emotions except natural. Therefore, although the first section of the first target video is selected as a meaningful section, it is determined that there is no object having a meaningful emotion.

Through this, it can be seen that even when a meaningful section is selected in the target video, an object having a meaningful emotion may not exist.

Referring to FIGS. 9B and 10B , the first section of the fourth target video is a multi-object section in which a total of four objects appear. In the case of flower, balloon, and cone_hat, it has a rank of 1/4 because it does not appear in any section before and after, and in the case of ribbon, it has a rank of 4/4 because it appears in the front frame. Also, compared with the previous section, the happiness emotion value increased by 15.494 (%), which can be predicted as the change rate of the emotion value of the newly appeared objects of flower, balloon, and cone_hat.

Through this, when multiple objects are included in the first meaningful section of the target video, priority is given according to the presence or absence of each object in the front and rear sections continuous to the first meaningful section. At this time, compared with the front and rear sections, the newly appeared object has the highest priority, and compared with the front and rear sections, the object appearing only in one of the front and rear sections has the second priority.

11 is a flowchart illustrating a method for analyzing an emotion object according to an embodiment of the present invention.

In operation 810, when information on the target video is received, the image collection unit of the emotional object analysis apparatus may collect a reaction video in which a person watching the target video is captured.

In operation 820, the frame extraction unit of the emotional object analysis apparatus may extract each frame included in the collected reaction video and the target video.

Here, in step 820, after receiving the collected reaction video, each frame included in the collected reaction video and the target video is extracted.

Also, step 820 may include extracting only a section in which a person included in the collected reaction video reacts.

In addition, operation 820 may include determining a degree of similarity between the extracted frames based on an object appearing in each frame with respect to the frames extracted from the target video, and merging similar frames according to the determination result.

In operation 830, a frame that induces a change in emotion of the person may be identified from among the frames included in the target video based on the facial expression of the person included in each frame extracted from the collected reaction video.

In one embodiment, in operation 830, the emotion analyzer of the emotional object analysis apparatus may analyze the emotion of the person based on the expression of the person included in each frame extracted from the collected reaction video. In addition, in step 830, after dividing the emotion into a plurality of emotion items, deriving the probability values of the plurality of emotion items according to the expression of the person in each frame, the average for each emotion item of the plurality of frames in the reaction video including each frame Calculating the change amount, analyzing the emotion of the person based on the average change amount for each emotion item and the probability value of the plurality of emotion items in each frame, and the emotion of the person among the frames included in the target video based on the analysis result It may include identifying the frame that induces the change. Here, as the result of analyzing the person's emotion, when there is an emotion item having a value greater than the average change amount for each emotion item among the probability values of a plurality of emotion items in each frame, it may be determined that the emotion of the person is an emotion item.

In operation 840, the object extraction unit of the emotional object analysis apparatus may store the degree of change in emotion of the person corresponding to the identified object, and output the stored content.

In operation 850, the information providing unit of the emotional object analysis apparatus may store the degree of change in emotion of the person corresponding to the identified object, and output the stored content. Also, in step 580, the information providing unit may generate and store an object dictionary for each emotion by arranging the objects in the order of increasing the degree of emotion change for the preset emotion item.

The emotion-specific object dictionary may be provided in the form of a table listing recommendations for each emotion item corresponding to each object in the content.

Table 3 shows a part of the object dictionary for each emotion in which the emotional influence of objects appearing in the first to fifth target videos according to an embodiment of the present invention is analyzed.

emotion object Emotion change rate (%) ranking
(Rank) Korean singer contempt neon_sign 2.30 1/1 disgust magnetic_tape 2.73 1/1 neon_sign 1.79 1/1 happiness umbrella 19.30 1/3 camera 19.30 1/3 orange 14.67 1/1 sadness neon_sign 14.74 1/1 lip 3.80 1/2 lipstick 3.80 1/2 surprise gun 7.86 1/1 phone 1.31 1/1 foreign singer anger hand 0.79 1/1 shoes 0.45 1/1 contempt shoes 0.81 1/1 fear bed 0.74 1/3 happiness picture 11.65 1/3 cake 11.30 1/3 cone_hat 9.99 1/3 sadness syringe 2.9 1/1 soccer_ball 1.96 1/2 bed 1.87 1/3 surprise soccer_ball 5.69 1/2 bed 5.22 1/3 dog 4.18 1/1

The object dictionary for each emotion according to an embodiment was created by arranging objects in the order of the highest emotion change rate for 7 emotion values for two categories, a Korean singer and a foreign singer.

Objects appearing in the first significant section are extracted, and a rank is derived according to whether objects appear before or after the first section meaningful. When an object with the same emotion of the object appears in several sections, the average of the emotion change rate is obtained.

As can be seen from Table 3, in the case of the happiness emotion with the largest average rate of change of emotion, 19.30%, 19.3%, and 14.67% of the emotions of the umbrella, camera, and orange objects of the Korean singer category, the picture of the foreign singer category, and the picture of the foreign singer category, respectively. For the emotions of cake and cone_hat objects, 11.65%, 11.30%, and 9.99% of emotion change rates are shown, respectively. In the case of sadness, with the second largest average change in emotion rate, 14.74%, 3.80, and 3.80% for the Korean singer category’s neon_sign, lip, and lipstick object, respectively, and the foreign singer category’s syringe, soccer_ball, and bed object’s emotions. They show emotional change rates of 2.9%, 1.96%, and 1.87%, respectively.

As such, Korean target videos showed high reactivity and foreign target videos showed relatively low reactivity. In other words, it is expected that the use of the object emotion dictionary of Korean videos will be more effective when producing content using objects.

For example, referring to the object dictionary for each emotion, as shown in Table 1, when producing an image for Koreans, when expressing emotions about happiness in an image, use an umbrella, a camera, an orange, etc. It can be recommended to add to the video.

For another example, if you are making a video for foreigners, if you want to express your feelings about happiness in the video, it may be recommended to add pictures, cakes, hats, etc. to the video.

The above description is merely illustrative of the technical idea of the present invention, and various modifications and variations may be made by those of ordinary skill in the art to which the present invention pertains without departing from the essential characteristics of the present invention. Accordingly, the embodiments implemented in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (17)

an image collecting unit for collecting a reaction video in which a person viewing the target video is captured when information on a target video including one or more objects is received;
a frame extraction unit for extracting each frame included in the collected reaction video and the target video;
an emotion analysis unit for identifying a frame that induces a change in emotion of the person from among the frames included in the target video based on the facial expression of the person included in each frame extracted from the collected reaction video; and
an object analysis unit for determining an object inducing a change in emotion of the person from among the frames included in the target video based on the degree of change in emotion, and quantifying the degree of change in emotion of the person in the determined object;
Information providing unit for storing the determined object and the degree of emotional change of the person corresponding to the determined object and outputting the stored content
Emotion object analysis device comprising a.
According to claim 1,
The apparatus for analyzing an emotion object, characterized in that the information providing unit generates and stores an object dictionary for each emotion by arranging the objects in the order of the greatest degree of emotion change for a preset emotion item.
According to claim 1,
After receiving the reaction video collected from the image collecting unit, extracting only the section in which the person included in the collected reaction video shows a reaction and delivering it to the frame extraction unit, characterized in that it further comprises a pre-processing unit, Emotional object analysis device.
According to claim 1,
With respect to the frames extracted from the target video, the degree of similarity between the extracted frames is determined based on the object appearing in each frame, and the similar frames are merged into the same section according to the determination result and delivered to the emotion analysis unit , An emotional object analysis device, characterized in that it further comprises a section separation unit.
According to claim 1,
The emotion analysis unit
After dividing the emotion into a plurality of emotion items, the probability value of the plurality of emotion items is derived according to the expression of the person in each frame, and the average amount of change for each emotion item of the plurality of frames in the reaction video including each frame is calculated Then, the emotion object analysis apparatus, characterized in that the frame having a value greater than the average change amount for each emotion item is classified as a meaningful frame and the meaningful frame is transmitted to the object analysis unit.
According to claim 1,
The object analysis unit,
After calculating the average change rate value for each emotion item of the person in the reaction video corresponding to the front and rear frames of the first frame in which each identified object appears, the average change amount for each emotion item and the plurality of emotion items in the first frame An emotion object analysis apparatus, characterized in that the emotion of the person for the object is analyzed based on the probability value of .
7. The method of claim 6,
The object analysis unit is characterized in that when the first frame includes multiple objects, for each object, the ranking of the objects affecting the emotion value is set based on the presence or absence of objects in the frames before and after the first frame, Emotional object analysis device.
8. The method of claim 7,
The object analysis unit gives the highest priority to the priority of the object in which the corresponding object does not appear in the front and rear frames when setting the ranking of multiple objects, and the ranking of the object in which the corresponding object appears only in one of the front and rear frames is the next priority, characterized in that which is an emotional object analysis device.
A method for analyzing an emotion object by a device, comprising:
collecting a reaction video captured by a person watching the target video when information on the target video is received;
extracting each frame included in the collected reaction video and the target video;
identifying a frame that induces a change in emotion of the person from among the frames included in the target video based on the facial expression of the person included in each frame extracted from the collected reaction video;
determining an object inducing a change in emotion of the person from among the frames included in the target video based on the degree of change in emotion, and quantifying the degree of change in emotion of the person in the determined object; and
Storing the determined object and the degree of emotional change of the person corresponding to the determined object, and outputting the stored contents
Including, an emotional object analysis method.
10. The method of claim 9,
Creating and storing an object dictionary for each emotion by arranging the objects in the order of the greatest degree of emotion change for the preset emotion item
Including, an emotional object analysis method.
10. The method of claim 9,
The step of extracting each frame is
After receiving the collected reaction video, the emotional object analysis method, characterized in that the step of extracting only a section in which the person included in the collected reaction video shows a reaction.
10. The method of claim 9,
The step of extracting each frame is
With respect to the frames extracted from the target video, determining the degree of similarity between the extracted frames based on an object appearing in each frame, and merging similar frames according to the determination result. object analysis method.
10. The method of claim 9,
The step of identifying the frame that induces the emotional change of the person,
after dividing the emotion into a plurality of emotion items, deriving probability values of the plurality of emotion items according to the expression of the person in a first frame;
calculating an average change amount for each emotion item of a plurality of frames in the reaction video including the first frame; and
and classifying a frame having a value greater than the average change amount for each emotion item as a meaningful frame.
14. The method of claim 13,
The step of quantifying the degree of emotional change of the person of the determined object is,
After calculating the average change rate value for each emotion item of the person in the reaction video corresponding to the front and rear frames of the first frame in which each identified object appears, the average change amount for each emotion item and the plurality of emotions of the first frame An emotion object analysis method, characterized in that the emotion of the person for the object is analyzed based on the probability value of the item.
15. The method of claim 14,
The step of quantifying the degree of emotional change of the person of the determined object is,
When the first frame includes multiple objects, it characterized in that it comprises the step of setting the rank of the object on the emotion value based on the presence or absence of the object in the frame before and after the first frame for each object, Method of analyzing emotional objects.
16. The method of claim 15,
The step of setting the rank of the object is,
An emotion object analysis method, characterized in that the priority of the object in which the corresponding object does not appear in the front and rear frames is given the highest priority, and the ranking of the object in which the corresponding object appears only in one of the front and rear frames is the second priority.
17. A computer-readable storage medium storing a computer program in which the method according to any one of claims 9 to 16 is performed when the computer program is executed by a processor.

Images