KR102512468B1 - Apparatus and method for analyzing viewers' responses to video content - Google Patents

Abstract

The present invention provides a device and method for analyzing the reaction of a viewer regarding video content. The present invention is configured to: play video content and generate a reaction video by filming the face of a viewer watching the video content; apply each video frame of the reaction video to an artificial intelligence model trained to determine a viewer's emotion from a face image to calculate each probability value in which a face image of the viewer included in each video frame corresponds to pre-defined emotions. Then, after calculating the average of the probability values of each emotion of the viewer for each video frame, the average value of each emotion is used to calculate the viewer's immersion for each video frame. The immersion for each video frame obtained in this way is compared with the average immersion of the entire reaction video so as to select and provide a user with 1) a section with the highest instantaneous immersion and a section with the lowest instantaneous immersion during the entire playback time of the video content, 2) an immersion maintenance section where the immersion for the video frame remains higher than the average and an immersion decline section where the immersion for the video frame remains lower than the average, and 3) a section with the maximum probability value for each emotion and a section with the minimum probability value for each emotion. Accordingly, it is possible to analyze the emotions and reactions of viewers regarding video content in various ways and provide analysis results obtained thereby.

Description

Apparatus and method for analyzing viewers' responses to video content}

The present invention relates to an apparatus and method for analyzing a viewer's reaction, and more particularly, to an apparatus and method for analyzing a viewer's reaction to advertisements, movies, dramas, and other video content.

Recently, as OTT services are widely spread and the number of video content providers providing video contents using social media and internet channels such as YouTube and Instagram is rapidly increasing, the media environment is rapidly changing.

As video contents provided to consumers increase exponentially, patterns in which consumers consume video contents are also diversifying. In other words, as consumers' video content consumption patterns change, such as binge-watching, double-speed viewing, skipping scenes, and viewing only summary content, the concept of regular viewing, such as 'on-premise shooter', is fading, and advertisements, dramas, movies, and other videos are changing. In terms of content consumption, new viewing patterns are emerging, such as consumers selecting only the scenes they want, increasing playback speed faster with double speed and skip functions, and recognizing the content only with the summary rather than watching the entire drama.

In accordance with these changes in video consumption patterns, the need for video content producers to reflect the satisfaction and dissatisfaction factors of video content in production is increasing by analyzing which scenes of video content consumers are satisfied with and dissatisfied with.

In particular, dissatisfaction factors can be rather difficult to find than satisfaction factors. This is because viewers are willing to leave reviews for parts they feel satisfied with or for parts they feel greatly disappointed with, but they are more likely not to write reviews for parts that do not arouse their interest.

Therefore, in the production of video content, a method and apparatus capable of analyzing actual reactions of consumers in addition to a method of simply analyzing viewing reviews or comments are required.

An object to be solved by the present invention is to provide a reaction analysis device and method capable of analyzing viewers' reactions in units of video frames by detecting reactions expressed by viewers while watching video content.

A viewer reaction analysis method for video content according to a preferred embodiment of the present invention for solving the above problems is a method for analyzing viewer reaction to video content performed in a reaction analysis system including a processor and a memory, (b) extracting an image of a viewer's face region from each video frame included in a reaction video synchronized with the video content, in which images of viewers watching the video content are captured; (c) When each of the extracted face region images is applied to an emotion determination artificial intelligence model, the emotion determination artificial intelligence model, for each video frame of the reaction video for each viewer, each face region image has a predefined emotion determining and outputting probabilities for each emotion; (d) calculating the degree of immersion of viewers for each video frame, using probability values for each emotion, for each video frame corresponding to the same time among reaction videos of different viewers; and (f) selecting and providing a section of image frames in which the level of immersion of each image frame meets a predefined condition.

In addition, before the step (b), (a) providing the artificial intelligence model for determining the emotion learned by using facial images including facial expressions representing the corresponding emotion for each emotion; may further include .

In addition, in the step (a), a face region is extracted from an image including a human face, and the extracted face region is analyzed to determine a probability that each emotion of a person expresses predefined emotions for each emotion, and output the result. An artificial intelligence model for emotion determination may be provided by learning an artificial intelligence model so as to be able to do so.

In addition, the video content and the reaction videos for different viewers may be synchronized with each other with time stamps.

In addition, in the step (d), among the reaction videos of different viewers, for each video frame corresponding to the same time, the average of probability values for each emotion may be calculated to calculate the viewer's immersion for each video frame. .

Further, in the step (d), the degree of immersion of each image frame may be calculated by adding up all probability averages of emotions other than expressionless (neutral) for each image frame.

In addition, in the step (f), a frame with the maximum and minimum immersion is selected from the entire video frame, and the maximum instantaneous immersion section and the instantaneous immersion degree are selected for a predetermined time before and after the selected frame. It can be selected as the smallest interval.

In addition, between the steps (d) and the step (f), (e) leveling the degree of immersion in units of a predefined number of image frames may be further included.

In the step (f), the longest section in which the degree of immersion of the video frames is maintained higher than the average degree of immersion of all the image frames is selected as the immersion maintenance section, and the degree of immersion of the video frames is maintained to be less than the average degree of immersion. It is possible to select and provide the longest section as an immersion reduction section.

In addition, in the step (f), the overall average of the probability for each emotion is calculated for all video frames, and among the intervals in which the corresponding emotion probability value of each video frame is higher than the overall average, the interval with the longest period is sequentially selected and provided. In addition, among the sections in which the emotion probability value of each video frame is less than the overall average, a section having the longest period may be sequentially selected and provided.

On the other hand, the computer program according to a preferred embodiment of the present invention for solving the above problems is stored in a non-transitory storage medium, and is executed in a computer including a processor to perform the above-described viewer reaction analysis method for video content. do.

On the other hand, an apparatus for analyzing viewer reaction to video content according to a preferred embodiment of the present invention for solving the above problems is an apparatus for analyzing viewer reaction to video content including a processor and a memory for storing predetermined instructions. The processor executing the commands stored in the memory (b) extracting an image of the viewer's face region from each video frame included in the reaction video synchronized with the video content, in which the images of the viewers watching the video content are photographed. ; (c) When each of the extracted face region images is applied to an emotion determination artificial intelligence model, the emotion determination artificial intelligence model, for each video frame of the reaction video for each viewer, each face region image has a predefined emotion determining and outputting probabilities for each emotion; (d) calculating the degree of immersion of viewers for each video frame, using probability values for each emotion, for each video frame corresponding to the same time among reaction videos of different viewers; and (f) selecting and providing a section of image frames in which the level of immersion of each image frame meets a predefined condition.

In addition, the processor, prior to the step (b), (a) providing the artificial intelligence model for determining the emotion learned using facial images including facial expressions representing the corresponding emotion for each emotion; can be done

In addition, in the step (a), a face region is extracted from an image including a human face, and the extracted face region is analyzed to determine a probability that each emotion of a person expresses predefined emotions for each emotion, and output the result. An artificial intelligence model for emotion determination may be provided by learning an artificial intelligence model so as to be able to do so.

In addition, the video content and the reaction videos for different viewers may be synchronized with each other with time stamps.

In addition, in the step (d), among the reaction videos of different viewers, for each video frame corresponding to the same time, the average of probability values for each emotion may be calculated to calculate the viewer's immersion for each video frame. .

Further, in the step (d), the degree of immersion of each image frame may be calculated by adding up all probability averages of emotions other than expressionless (neutral) for each image frame.

In addition, in the step (f), a frame with the maximum and minimum immersion is selected from the entire video frame, and the maximum instantaneous immersion section and the instantaneous immersion degree are selected for a predetermined time before and after the selected frame. It can be selected as the smallest interval.

In addition, between the steps (d) and the step (f), (e) leveling the degree of immersion in units of a predefined number of image frames may be further performed.

In the step (f), the longest section in which the degree of immersion of the video frames is maintained higher than the average degree of immersion of all the image frames is selected as the immersion maintenance section, and the degree of immersion of the video frames is maintained to be less than the average degree of immersion. It is possible to select and provide the longest section as an immersion reduction section.

In addition, in the step (f), the overall average of the probability for each emotion is calculated for all video frames, and among the intervals in which the corresponding emotion probability value of each video frame is higher than the overall average, the interval with the longest period is sequentially selected and provided. In addition, among the sections in which the emotion probability value of each video frame is less than the overall average, a section having the longest period may be sequentially selected and provided.

The present invention generates a reaction video by photographing the face of a viewer who watches it while playing video content, and transfers each video frame of the reaction video to an artificial intelligence model trained to determine the viewer's emotion from the viewer's face image. By applying, probability values corresponding to predefined emotions of the viewer's face image included in each video frame are obtained. In addition, after obtaining an average of probability values of each emotion of viewers for each video frame, the degree of viewer immersion for each corresponding video frame is obtained using the average value of each emotion. By comparing the degree of immersion of each video frame obtained in this way with the average degree of immersion of all reaction videos, ① the section with the maximum and minimum immersion during the total playback time of the video content, and ② the immersion level of the video frame is maintained higher than the average. By selecting and providing users with an immersion maintenance section and an immersion deterioration section that is maintained below the average, and ③ the maximum section/minimum section of the probability value for each emotion, viewers can analyze and provide various emotions and reactions to video content. there is.

1 is a diagram showing the overall configuration of a viewer reaction analysis system for video content according to a preferred embodiment of the present invention.
2 is a diagram illustrating a method for analyzing a viewer's reaction to video content according to a preferred embodiment of the present invention.
3A to 3C are graphs in which immersion is scaled and displayed.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

Here, the above objects, features and advantages of the present invention will become more apparent through the following detailed description in conjunction with the accompanying drawings. However, the present invention can apply various changes and can have various embodiments, hereinafter, specific embodiments will be illustrated in the drawings and described in detail.

Like reference numerals designate essentially like elements throughout the specification. In addition, components having the same function within the scope of the same idea appearing in the drawings of each embodiment are described using the same reference numerals.

When it is said that a certain part "includes" a certain component throughout the specification, it means that it may further include other components without excluding other components unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

If it is determined that a detailed description of a known function or configuration related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. In addition, numbers (eg, 1st, 2nd, etc.) used in the description process of this specification are only identifiers for distinguishing one component from another component.

1 is a diagram showing the overall configuration of a viewer reaction analysis system for video content according to a preferred embodiment of the present invention.

Referring to FIG. 1, the viewer reaction analysis system for video content basically includes a reaction image collection device 230 and a reaction analysis device 250, and stores the viewer reaction images collected by the reaction image collection device 230. A reaction analysis DB 260 for storing the results analyzed by the reaction image DB 240 and the reaction analysis device 250 may be further included. In addition, the reaction analysis device 250 may include a processor 253, a communication module 251, and a memory 255.

Looking at the function of each component, the reaction image collection device 230 includes a camera 210 and a display 220 inside, or is connected to an external camera 210 and the display 220, and the display 220 At the same time as the video content is displayed to the viewer through, a reaction video is generated by photographing the viewer using the camera 210, and the reaction video is transmitted to the reaction analysis device 250 or stored in the reaction video DB 240. . At this time, a timestamp is set in each frame of the reaction video to be synchronized with the reproduction time of the video content.

In a preferred embodiment of the present invention, the reaction video is taken at a frame rate of 25 fps, but is not limited thereto.

The reaction analysis device 250 receives a reaction video for video content from the reaction video collecting device 230 or receives a reaction video for video content stored in the reaction video DB 240, and displays each video frame of the reaction video. For each frame included in the reaction video, the emotions expressed by each viewer while watching the video content are analyzed.

Then, the reaction analysis device 250 calculates the average value of each emotion expressed by all viewers for each frame, calculates the degree of immersion for the frame, obtains the average degree of immersion for the entire video content, and calculates the degree of immersion. Various statistical information can be generated and provided.

Statistical information provided according to a preferred embodiment of the present invention includes a section in which the degree of immersion of an image frame is higher than the average degree of immersion, a section in which the degree of immersion is maintained longer than the average degree of immersion, a section in which the degree of immersion of a frame is lower than the average degree of immersion, and It may include information about a section of 30 seconds or 1 minute before and after the section that is maintained below the average level of immersion for the longest time, the frame with the highest level of immersion, and the frame with the lowest level of immersion.

The reaction analysis device 250 according to a preferred embodiment of the present invention includes a processor 253, a memory 255, and a communication module 251.

The memory 255 according to a preferred embodiment of the present invention may store instructions executable by the processor 253 and programs executed by the processor 253, and may store input/output data. Examples of the memory 255 include a hard disk drive (HDD), a solid state drive (SSD), a flash memory 255, a read-only memory (ROM), and a random access RAM (RAM). memory), etc. The memory 255 may be alternatively operated as a web storage or cloud server that performs the function of a storage medium on the Internet.

The processor 253 according to a preferred embodiment of the present invention may be implemented as a central processing unit (CPU) or a similar device (eg, a micro processing unit (MPU), a micro control unit (MCU), etc.), and a memory 255 ), each step of the method for analyzing a viewer's reaction to video content, which will be described later with reference to FIG. 2, is performed.

The communication module 251 communicates with the reaction image collection device 230 in a wired or wireless communication method. As a wired communication method used by the communication module 251, LAN (Local Area Network) or USB (Universal Serial Bus) communication is a representative example, and other methods are also possible.

In addition, in the case of the wireless type, a WPAN (Wireless Personal Area Network)-based communication method such as Bluetooth or Zigbee can be used, and a WLAN (Wireless Local Area Network)-based communication method such as Wi-Fi can be used. It is also possible to use a communication method, a mobile communication network (LTE, 5G, etc.) or other known communication methods.

2 is a diagram illustrating a method for analyzing a viewer's reaction to video content according to a preferred embodiment of the present invention.

With further reference to FIG. 2 , a function of a viewer reaction analysis system for video content and a method for analyzing viewer reaction to video content according to a preferred embodiment of the present invention will be described.

First, the reaction analysis device 250 according to a preferred embodiment of the present invention learns an artificial intelligence model using facial images including facial expressions representing corresponding emotions for each emotion, and from images including human faces. An artificial intelligence model for emotion judgment is provided so that a face area is extracted and a person's emotion is determined by analyzing the extracted face area (S210).

To this end, in a preferred embodiment of the present invention, the processor 253 extracts only the face region from the face image for learning using a multi-task cascaded convolutional networks (MTCNN) algorithm in advance, and uses the extracted face region image for emotion determination. The artificial intelligence model for emotion determination was learned by performing supervised learning in a way that applied to the input end of the artificial intelligence model and applied the emotion represented by the corresponding face area image to the output end.

In a preferred embodiment of the present invention, facial regions were extracted from about 500,000 emotion-expressing facial images provided by the Korea Institute for Intelligent Information Society Promotion and used as facial images for learning, and the EfficientNet b3 algorithm was used as an artificial intelligence model for emotion determination. , Face images provided by other databases can be used as face images for training, and other well-known AI models can be used in the case of an AI model for emotion judgment, of course.

In addition, the learning images provided by the Korea Intelligence Information Society Agency classify and provide the user's facial expressions into seven emotions of joy, embarrassment, anger, anxiety, hurt, sadness, and neutral (expressionless), and are used for emotion judgment of the present invention The artificial intelligence model learns all of the facial images representing these seven emotions, and outputs the probability that the input face image corresponds to each of the seven emotions as a value for each emotion in an emotion determination process described later.

For example, if a face image called A is input, the emotion judgment artificial intelligence model, joy: 0.4, embarrassment: 0.1, anger: 0.1, anxiety: 0.1, hurt: 0.5, sadness: 1.5, neutral (expressionless): 1.0, The corresponding face image outputs a probability value corresponding to each emotion, and the sum of these probability values is 1.

Here, the present invention uses face images classified as 7 emotions as training images, but when face images classified as 4 emotions are used as training images, the emotion decision artificial intelligence model calculates the probability value belonging to each of the 4 emotions. will output

When the learning of the artificial intelligence model for emotion determination is completed, the reaction video collection device 230 plays the video content through the display 220, takes pictures of the viewers watching it with the camera 210, and captures the video content. When generating a reaction video synchronized with a time stamp and transmitting the generated reaction video to the reaction analysis device 250 or storing it in the reaction image DB 240, the reaction analysis device 250 communicates the module 251 Receives a reaction video from the reaction image collecting device 230 or receives a reaction video previously stored in the reaction image DB 240 from the reaction image DB 240 (S220).

In step S220 of the preferred embodiment of the present invention, the reaction video is generated at a frame rate of about 25 fps, but the frame rate can be varied, and a timestamp is set in each frame of the reaction video to be synchronized with the playback time of the video content has been

In addition, in a preferred embodiment of the present invention, it has been described that a reaction video is generated after the learning of the artificial intelligence model for emotion judgment is completed, but the reaction video collection device 230 generates a reaction video and the reaction video DB 240 After storing in , learning of an artificial intelligence model for emotion judgment may be performed.

In addition, in order to analyze viewers' satisfaction/dissatisfaction and emotional changes with respect to video content, face images of a plurality of viewers are required. Therefore, when a reaction video is generated for a viewer who watches video content alone, a plurality of reaction videos are required for each viewer for one video content.

If a plurality of viewers watch video content together to produce one reaction video, and the facial images of the plurality of viewers are included in the video frame of the reaction video, the analysis described below is possible with only one reaction video.

Meanwhile, after the reaction video is received, the processor 253 of the reaction analysis device 250 uses a multi-task cascaded convolutional networks (MTCNN) algorithm to generate an image of the viewer's face region from each video frame included in the reaction video. Extract (S230).

When the face region is extracted, the processor 253 determines the emotion represented by the face region of each frame by applying the face region image of each image frame to the emotion determination artificial intelligence model (S240).

Specifically, when a face image is input to the learned emotion determination artificial intelligence model, values representing the possibility that the face image corresponds to each emotion are output for each emotion.

As described above, when the emotion determination artificial intelligence model of the present invention is learned with seven emotions, the analysis result output in step S240 indicates that the corresponding face image is joy, embarrassment, anger, anxiety, hurt, sadness, neutral ( Expressionless) outputs a probability value corresponding to each of the 7 emotions. If analysis is performed using all of these seven emotions, the corresponding values are used as they are in a process described later.

However, as in a preferred embodiment of the present invention described later, when the number of emotions to be actually analyzed is smaller than the number of learned emotions, the probability value that the corresponding face image belongs to the emotions to be analyzed is recalculated only for the emotions to be analyzed. do.

For example, a preferred embodiment of the present invention described later analyzes whether the viewer is immersed in video content and which emotion among the most representative emotions, joy, sadness, and anger, is analyzed. Therefore, instead of using all of the above seven emotions, the probability that the face image belongs to the above four emotions is recalculated using the output values for the four emotions of joy, sadness, anger, and neutral (expressionless). In the present invention, the probabilities of belonging to any one of the above four emotions were calculated by applying the probability values belonging to the corresponding four emotions among the seven emotions to the Softmax function. Softmax is a function that normalizes all input values to output values between 0 and 1, and the sum of the output values is always 1. It is widely known in the deep learning technology field to which the present invention belongs. Since it is a function, a detailed description is omitted.

For convenience of description, as a simple example, assuming that a reaction video composed of three frames is generated and analyzed for two viewers, the above-described steps S230 and S240 for each viewer's reaction video. , emotion analysis results for each viewer can be obtained as shown in Table 1 below.

When the probability value that the face image belongs to each emotion is output for each viewer and for each video frame, the processor 253 selects each video frame for each video frame corresponding to the same time among the reaction videos of different viewers, for each emotion. An average of the probability values is calculated (S250).

Referring to the case of Table 1, the average of the probability values of each emotion in the first frame of viewer 1 and the corresponding emotion in the first frame of viewer 2 is calculated as shown in Table 2 below. In addition, the results for the second frame and the third frame are also shown in Table 2.

frame number pleasure sadness anger expressionless Sum One 0.45 0.2 0.1 0.25 One 2 0.55 0.1 0.15 0.2 One 3 0.15 0.2 0.2 0.45 One

After that, the processor 253 calculates the degree of immersion of the viewers by using the average of probability values for each emotion of the viewers for each video frame (S260). In the present invention, the degree of immersion is defined as the probability that viewers express a specific emotion while viewing a corresponding video frame. That is, in the examples described in Tables 1 and 2 above, if the viewer watching the video content feels joy, sadness, or anger, it means that he is immersed in the video content, and thus the viewer's immersion in the corresponding video frame. Degree can be expressed as the sum of the corresponding emotions, which is equivalent to subtracting the probability of expressionless expression showing no emotion from the total probability of 1. Therefore, in the example of Table 2 above, the degree of immersion of the viewers in the video frame 1 is 0.75 (0.45+0.2+0.1 or 1-0.25), the degree of immersion of the viewers in the video frame 2 is 0.8, and the degree of immersion of the viewers in the video frame 3 is The degree of immersion of viewers for is 0.55.

Then, the processor 253 equalizes the degree of immersion of each image frame by grouping them in units of predetermined image frames (S270). The present invention aims to provide useful information for producing video contents by providing statistical information such as a section in which viewers are immersed in video content, a section in which a particular emotion is felt, and a section for a predetermined time in which the viewer is most immersed. However, since the facial expressions of viewers can change instantaneously, there are cases in which the level of immersion changes rapidly, and the emotions of the viewers remain the same, but the emotions change rapidly due to errors caused by the artificial intelligence model not being perfect. It may also appear that

3A to 3C are graphs in which immersion is scaled by a value between 0 and 3.5.

Referring to FIG. 3A , the average immersion value of all frames is indicated by a white dotted line, and the immersion value of each frame is indicated by a blue color. As shown in FIG. 3A , if the immersion value changes rapidly for each frame, it is difficult to find a section in which the viewer continuously watches for a considerable period of time with a certain level of immersion.

Therefore, as shown in FIG. 3B, a preferred embodiment of the present invention calculates the average of the degree of immersion by grouping the image frames in units of 100 frames and outputs it as a graph, or groups the image frames in units of 500 frames and calculates the average degree of immersion. It can be calculated and displayed as a graph.

After that, the processor 253 selects an image frame section in which the probability value of immersion or emotion of each image frame meets a predefined condition and provides the selected image frame section to the user (S280).

For example, if the condition defined in advance is ① a section in which the instantaneous degree of immersion is the maximum and the minimum in the total playback time of the video content, the processor 253 determines the degree of immersion in the entire frame as shown in FIG. 3A. The maximum frame 301 and the minimum frame 302 are selected, respectively, and a section of about 30 seconds before and after the frame for a total of 1 minute is selected as the maximum section of moment immersion and the minimum section of moment immersion to analyze the reaction DB (260 ) and provide it to users.

If the predefined condition is ② immersion maintenance section / immersion decrease section, the processor 253 determines the longest section 303 in which the immersion level is maintained higher than the average immersion level and the immersion level is maintained lower than the average immersion level. The longest section 304 is selected and provided. At this time, as shown in FIGS. 3B and 3C , the processor 253 selects an immersion maintenance section/immersion lowering section from the leveled section, stores it in the response analysis DB 260, and provides it to users.

If the predefined condition is ③ the maximum/minimum range for each emotion, the processor 253 calculates the overall average of the probabilities for each emotion for all video frames in Table 2 above, and the corresponding emotion probability value for each frame. Among the sections above the average, the sections with the longest period are selected in order, stored in the response analysis DB 260 and provided to users, and among the sections with the corresponding emotion probability value of each frame below the average, the sections with the longest period are selected in order. It can be stored in the reaction analysis DB 260 and provided to users.

However, in selecting the maximum and minimum sections of immersion and emotion, a predetermined time section immediately after video content is played is excluded from the selection. This is because the intro of the video content is played immediately after the video content starts, and content that can evoke the expression of emotion from the viewer is not played. because it corresponds to the section.

In addition, in step S280, the processor 253 responds to the result of analysis through the above-described process, for example, the degree of immersion of each image frame, probability values for each emotion, and information about sections satisfying predefined conditions. It can be stored in the analysis DB (260).

The method for analyzing a viewer's reaction to video content according to a preferred embodiment of the present invention described so far may be implemented as a computer executable command and a computer program stored in a non-transitory storage medium.

The storage medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable storage media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable storage medium is distributed in computer systems connected through a network, so that computer-readable codes can be stored and executed in a distributed manner.

So far, the present invention has been looked at with respect to its preferred embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

100: viewers
210: camera
220: display
230: reaction image collection device
240: reaction image DB
250: reaction analysis device
251: communication module
253: processor
255: memory
260: reaction analysis DB

Claims (21)

A viewer reaction analysis method for video content performed in a reaction analysis system including a processor and memory,
(b) extracting an image of the viewer's face region from each video frame included in a reaction video synchronized with the video content, which captures the viewers watching the video content;
(c) When each of the extracted face region images is applied to an emotion determination artificial intelligence model, the emotion determination artificial intelligence model, for each video frame of the reaction video for each viewer, each face region image has a predefined emotion determining and outputting probabilities for each emotion;
(d) calculating the degree of immersion of viewers for each video frame, using probability values for each emotion, for each video frame corresponding to the same time among reaction videos of different viewers;
(e) leveling the degree of immersion in units of a predefined number of image frames; and
(f) selecting and providing a section of video frames in which the degree of immersion of each video frame meets a predefined condition;
The step (f) is
The longest section in which the degree of immersion of video frames is maintained higher than the average degree of immersion of all image frames is selected as the immersion maintenance section, and the longest section in which the degree of immersion of video frames is maintained smaller than the average degree of immersion is designated as the immersion reduction section. Select and provide
Calculate the overall average of probabilities for each emotion for all video frames,
Among the sections in which the corresponding emotion probability value of each video frame is higher than the overall average, the section with the longest period is selected and provided in order, and among the sections in which the corresponding emotion probability value of each video frame is lower than the overall average, the section with the longest period is selected and provided in order. A method for analyzing viewer reactions to video content, characterized in that for. The method of claim 1, before step (b)
(a) providing the emotion determination artificial intelligence model learned by using facial images including facial expressions representing the corresponding emotion for each emotion; a viewer reaction analysis method for video content, characterized in that it further comprises. The method of claim 2, wherein step (a)
An artificial intelligence model is learned to extract a face region from an image including a human face, analyze the extracted face region, determine the probability that each emotion represents a predefined emotion, and output the result. A method for analyzing a viewer's reaction to video content, characterized in that it comprises an emotion determination artificial intelligence model. According to claim 1,
The video content and the reaction videos for different viewers are synchronized with each other with a time stamp. The method of claim 1, wherein step (d) is
Among the reaction videos of different viewers, for each video frame corresponding to the same time, the average of probability values for each emotion is calculated to calculate the degree of immersion of viewers for each video frame. analysis method. The method of claim 5, wherein step (d)
A method for analyzing a viewer's reaction to video content, characterized in that for each video frame, the degree of immersion of each video frame is calculated by summing up all probability averages of the remaining emotions except expressionless (neutral). The method of claim 1, wherein step (f) is
Selecting frames with the maximum and minimum immersion in the entire video frame, respectively, and selecting a predetermined time before and after the selected frame as the maximum section for the maximum immersion and the minimum section for the instantaneous immersion A method for analyzing viewer reactions to video content. delete delete delete A computer program stored in a non-transitory storage medium and executed on a computer including a processor to perform the method of analyzing a viewer's reaction to any one of claims 1 to 7.
An apparatus for analyzing viewer reaction to video content including a processor and a memory storing predetermined instructions,
The processor executing the instructions stored in the memory
(b) extracting an image of the viewer's face region from each video frame included in a reaction video synchronized with the video content, which captures the viewers watching the video content;
(c) When each of the extracted face region images is applied to an emotion determination artificial intelligence model, the emotion determination artificial intelligence model, for each video frame of the reaction video for each viewer, each face region image has a predefined emotion determining and outputting probabilities for each emotion;
(d) calculating the degree of immersion of viewers for each video frame, using probability values for each emotion, for each video frame corresponding to the same time among reaction videos of different viewers;
(e) leveling the degree of immersion in units of a predefined number of image frames; and
(f) performing a step of selecting and providing a section of video frames in which the degree of immersion of each video frame meets a predefined condition;
The step (f) is
The longest section in which the degree of immersion of video frames is maintained higher than the average degree of immersion of all image frames is selected as the immersion maintenance section, and the longest section in which the degree of immersion of video frames is maintained smaller than the average degree of immersion is designated as the immersion reduction section. Select and provide
Calculate the overall average of probabilities for each emotion for all video frames,
Among the sections in which the corresponding emotion probability value of each video frame is higher than the overall average, the section with the longest period is selected and provided in order, and among the sections in which the corresponding emotion probability value of each video frame is lower than the overall average, the section with the longest period is selected and provided in order. Apparatus for analyzing viewer reaction to video content, characterized in that for doing. 13. The method of claim 12, wherein the processor
Before step (b),
(a) providing the emotion determination artificial intelligence model learned using facial images including facial expressions representing the corresponding emotion for each emotion; 14. The method of claim 13, wherein step (a)
An artificial intelligence model is learned to extract a face region from an image including a human face, analyze the extracted face region, determine the probability that each emotion represents a predefined emotion, and output the result. Apparatus for analyzing viewer reaction to video content, characterized in that it comprises an emotion determination artificial intelligence model. According to claim 12,
Apparatus for analyzing viewer reaction to video content, characterized in that the video content and the reaction videos for different viewers are synchronized with each other with a time stamp. 13. The method of claim 12, wherein step (d)
Among the reaction videos of different viewers, for each video frame corresponding to the same time, the average of probability values for each emotion is calculated to calculate the degree of immersion of viewers for each video frame. analysis device. 17. The method of claim 16, wherein step (d)
An apparatus for analyzing a viewer's reaction to video content, characterized in that for each video frame, the degree of immersion of each video frame is calculated by adding up all probability averages of the remaining emotions except expressionless (neutral). 13. The method of claim 12, wherein step (f)
Selecting frames with the maximum and minimum immersion in the entire video frame, respectively, and selecting a predetermined time before and after the selected frame as the maximum section for the maximum immersion and the minimum section for the instantaneous immersion A device for analyzing viewer reactions to video content. delete delete delete

Images