JP7398853B1 - Video viewing analysis system, video viewing analysis method, and video viewing analysis program - Google Patents

Abstract

[Problem] To provide detailed insights regarding videos from viewers' behavior when viewing videos based on their biological information. [Solution] A video viewing analysis system that analyzes viewer insight regarding a video acquires a plurality of gaze point data corresponding to each of a plurality of viewers, and based on at least some of the plurality of gaze point data. Then, a gaze point visualization map is generated that visualizes the gaze point when the viewer corresponding to the gaze point data views the video, and the gaze point visualization map is displayed superimposed on the video. [Selection diagram] Figure 9

Description

Article 30, Paragraph 2 of the Patent Act applies Website publication date: May 10, 2020, Website address: https://research. viagate. com/ Site publication date: May 25, 2020, publication site address: https://prtimes. jp/main/html/rd/p/000000002.000116494. html Site publication date: May 31, 2020, publication site address: https://prtimes. jp/main/html/rd/p/000000003.000116494. html Site publication date: March 27, 2020, publication site address: https://www. wantedly. com/companies/company_7088487/about Site publication date: June 5, 2020, publication site address: https://sogyotecho. jp/news/20230605viagate/

The present disclosure relates to a video viewing analysis system, a video viewing analysis method, and a video viewing analysis program.

Due to the effects of the coronavirus pandemic, most aspects of social life are now moving online (in other words, digital transformation (DX)) at an accelerating pace. In particular, in the field of marketing, digital marketing has become more mainstream than traditional marketing using paper media and the like. In the field of digital marketing, advertising videos on video platforms such as YouTube (registered trademark) are currently becoming mainstream.

Patent Document 1 discloses an advertisement evaluation system for evaluating the effectiveness of advertisement videos. The advertisement evaluation system includes a user terminal, an e-mail server, a video distribution server, and a management server. These elements are communicatively connected via a communications network. The advertising evaluation system determines whether or not the user has purchased the product introduced in the advertising video based on the user's transaction data, and then evaluates the effectiveness of the advertising video based on the determination result. .

Patent No. 7057606

In the advertisement evaluation system disclosed in Patent Document 1, the effectiveness of an advertisement video is evaluated according to the sales performance of whether a user actually purchased the product introduced in the advertisement video. Therefore, in this advertisement evaluation system, it is not possible to evaluate an advertisement video until there is no actual sales record of the advertised product. On the other hand, companies that conduct digital marketing for products have a need to obtain detailed insights about videos such as advertising videos before launching them. By “insights” we mean a deep understanding of a customer’s underlying needs, emotions, motivations, values, behavior patterns and trends, presented in a sophisticated form by analyzing customer data and feedback. do. In particular, on the corporate side, there is a need to obtain insights regarding advertising videos based on the biometric information (for example, gaze and facial expressions) of viewers who view the videos before launching them. As described above, there is room for consideration of an advertisement evaluation system that can meet the above needs.

The purpose of the present disclosure is to provide a video viewing analysis system, a video viewing analysis method, and a video viewing analysis program that can provide detailed insights regarding videos from the viewer's behavior when viewing videos based on biological information. shall be.

A video viewing analysis system according to an aspect of the present disclosure is a video viewing analysis system that analyzes viewer insights regarding videos, and acquires a plurality of gaze point data corresponding to each of a plurality of viewers, A gaze point visualization map that visualizes the gaze point when the viewer corresponding to the gaze point data views the video is generated based on at least some of the gaze point data, and the gaze point visualization map is transformed into the gaze point visualization map. Display superimposed on the video.

According to the above configuration, the point of interest visualization map is displayed superimposed on the video, so the company can understand the viewer's viewing behavior toward the video in detail by checking the point of interest visualization map. . For example, the company can grasp in detail the areas of interest in the video by viewers for each playback time. In this way, it is possible to provide detailed insights regarding videos to companies that desire video analysis.

A video viewing analysis system according to an aspect of the present disclosure is a video viewing analysis system that analyzes viewer insights regarding videos, and acquires a plurality of gaze point data corresponding to each of a plurality of viewers, Based on at least some of the point of interest data, the object of interest when the viewer corresponding to the point of interest data views the video is identified, and information regarding the object of interest is displayed.

According to the above configuration, the company can grasp information regarding at least one object of interest in the video. In this way, it is possible to provide detailed insights regarding videos to companies that desire video analysis.

A video viewing analysis system according to one aspect of the present disclosure is a video viewing analysis system that analyzes viewer insights regarding videos, acquires a plurality of facial expression data corresponding to each of a plurality of viewers, and acquires a plurality of facial expression data corresponding to each of the plurality of viewers. Based on at least some of the facial expression data, comprehensive facial expression information indicating a comprehensive facial expression when a viewer corresponding to the facial expression data views the video is generated, and the comprehensive facial expression information is displayed.

According to the above configuration, since the comprehensive facial expression information is displayed, the company can understand the viewer's viewing behavior toward the video in detail by checking the comprehensive facial expression information. For example, it becomes possible for companies to grasp in detail viewers' emotions toward videos (for example, happy, sad, etc.) for each playback time. In this way, it is possible to provide detailed insights regarding videos to companies that desire video analysis.

A video viewing analysis system according to one aspect of the present disclosure is a video viewing analysis system that analyzes viewer insights regarding videos, acquires a plurality of facial expression data corresponding to each of a plurality of viewers, and acquires a plurality of facial expression data corresponding to each of the plurality of viewers. Generate comprehensive interest level information indicating a comprehensive interest level when a viewer corresponding to the facial expression data views the video based on at least some of the facial expression data, and display the comprehensive interest level information. .

According to the above configuration, since the comprehensive interest level information is displayed, the company can understand the viewer's viewing behavior toward the video in detail by checking the comprehensive interest level information. For example, it becomes possible for companies to grasp in detail the interest level of viewers in videos for each playback time. In this way, it is possible to provide detailed insights regarding videos to companies that desire video analysis.

Further, the video viewing analysis system may input extraction conditions for the plurality of viewers, and extract at least some of the plurality of gaze point data based on the extraction conditions.

According to the above configuration, a gaze point visualization map is generated based on the extracted gaze point data. In this way, it is possible to understand in detail the viewing behavior of viewers who meet the extraction conditions, making it possible to provide companies with more detailed insights regarding videos.

Further, the extraction conditions include age information of the plurality of viewers, gender information of the plurality of viewers, address information of the plurality of viewers, occupation information of the plurality of viewers, and The information may be associated with at least one of household number information of viewers, marital information of the plurality of viewers, and educational background information of the plurality of viewers.

According to the above configuration, a viewer who matches the extraction condition associated with at least one of age information, gender information, address information, occupation information, household size information, marital information, and educational background information of multiple viewers Since it is possible to understand viewing behavior in detail, it is possible to provide companies with more detailed insights regarding videos.

Furthermore, the video may include advertising content.

According to the above, it is possible to provide companies with detailed insights regarding digital marketing.

Moreover, the gaze point visualization map may be a gaze point heat map.

According to the above, by checking the attention point heat map, the company can understand in detail the viewing behavior of viewers regarding videos. In particular, by checking the intensity distribution of the gaze point heat map, it is possible to more clearly understand the viewer's attention area on the video.

Furthermore, the point of gaze data may indicate a temporal change in a point of gaze corresponding to the viewer's line of sight on the display area of the moving image.

Furthermore, the video viewing analysis system may display the objects of interest in a ranking format.

Furthermore, the video viewing analysis system may display at least one of the attention level, appearance time, and average attention level of the attention object in association with the attention object.

Further, the comprehensive facial expression information may indicate a probability throughout the video for each of a plurality of facial expression elements.

According to the above, the company can understand the viewer's viewing behavior toward the video in detail by checking the probability throughout the video for each of the plurality of facial expression elements.

Further, the comprehensive facial expression information may indicate a probability transition corresponding to a time series of the video for each of a plurality of facial expression elements.

According to the above, the company can understand the viewer's viewing behavior toward the video in detail by checking the transition of the probability corresponding to the time series of the video for each of the plurality of facial expression elements.

Further, the comprehensive interest level information may indicate a change in the interest level corresponding to a time series of the video.

According to the above, the company can understand the viewer's viewing behavior toward the video in detail by checking the change in interest level corresponding to the character of the video.

Furthermore, the video viewing analysis system may calculate the positive degree or negative degree of the viewer with respect to the video based on the probability of each of a plurality of facial expression elements, and display the positive degree or the negative degree.

According to the above, by checking the positive degree or negative degree of the viewer, the company can understand the viewer's viewing behavior toward the video in detail.

A video viewing analysis method according to one aspect of the present disclosure is a video viewing analysis method for analyzing viewer insight regarding a video, the method comprising: acquiring a plurality of gaze point data corresponding to each of a plurality of viewers; generating a gaze point visualization map that visualizes the gaze point when a viewer corresponding to the gaze point data views the video, based on at least some of the plurality of gaze point data; and the gaze point The method includes a step of superimposing and displaying a visualization map on the moving image, and is executed by a computer.

A video viewing analysis method according to one aspect of the present disclosure is a video viewing analysis method for analyzing viewer insight regarding a video, the method comprising: acquiring a plurality of gaze point data corresponding to each of a plurality of viewers; a step of identifying, based on at least some of the plurality of points of interest data, an object of interest when a viewer corresponding to the point of interest data views the video; and a step of displaying information regarding the object of interest. , and executed by a computer.

A video viewing analysis method according to an aspect of the present disclosure is a video viewing analysis method for analyzing viewer insight regarding a video, the method comprising: acquiring a plurality of facial expression data corresponding to each of a plurality of viewers; a step of generating, based on at least some of the plurality of facial expression data, comprehensive facial expression information indicating a comprehensive facial expression when a viewer corresponding to the facial expression data views the video; and displaying the comprehensive facial expression information. and executed by a computer.

A video viewing analysis method according to an aspect of the present disclosure is a video viewing analysis method for analyzing viewer insight regarding a video, the method comprising: acquiring a plurality of facial expression data corresponding to each of a plurality of viewers; a step of generating, based on at least some of the plurality of facial expression data, comprehensive interest level information indicating a comprehensive interest level when a viewer corresponding to the facial expression data views the video; and displaying the information, the method being executed by the computer.

Further, a video viewing analysis program that causes a computer to execute the video viewing analysis method may be provided.

According to the present disclosure, it is possible to provide a video viewing analysis system, a video viewing analysis method, and a video viewing analysis program that can provide detailed insights regarding videos from the viewer's behavior during video viewing based on biological information. I can do it.

1 is a diagram illustrating an example of a configuration of a video viewing analysis system according to an embodiment of the present disclosure (hereinafter referred to as the present embodiment). It is a diagram showing an example of the hardware configuration of a viewer terminal. It is a diagram showing an example of the hardware configuration of a server. FIG. 2 is a diagram illustrating an example of the hardware configuration of a company-side terminal. FIG. 2 is a sequence diagram for explaining a series of processes executed between a viewer terminal and a server in the video viewing analysis system according to the present embodiment. FIG. 3 is a diagram for explaining a viewer's gaze point on a display area of a moving image. FIG. 3 is a diagram for explaining facial feature points of a viewer. FIG. 2 is a sequence diagram for explaining a series of processes executed between a server and a company terminal in the video viewing analysis system according to the present embodiment. It is a figure showing an example of a video viewing analysis screen. (a) is a diagram showing an example of a moving image. (b) is a diagram showing an example of a gaze point heat map. (c) is a diagram showing an example of a composite video in which a gaze point heat map is superimposed on a video. FIG. 3 is a diagram schematically showing each frame making up a video and each frame making up a gaze point heat map, and (a) shows a state where the frame rate of the video and the frame rate of the gaze point heat map match. (b) is a diagram showing a state where the frame rate of the moving image and the frame rate of the gaze point heat map do not match. It is a figure showing an example of an extraction condition display screen. It is a figure which shows an example of a video viewing analysis screen after extraction conditions are specified.

Hereinafter, a video viewing analysis system 1 according to the present embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of a video viewing analysis system 1 according to the present embodiment. As shown in FIG. 1, the video viewing analysis system 1 includes viewer terminals 2a and 2b, a server 3, and a company terminal 4. These are connected to a communication network 8. Each of the viewer terminals 2a and 2b is communicably connected to the server 3 via a communication network 8. The company terminal 4 is communicably connected to the server 3 via the communication network 8 . The communication network 8 is configured by at least one of a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a wireless core network.

The viewer terminal 2a is a terminal associated with the viewer Va, and is operated by the viewer Va. The viewer terminal 2b is a terminal associated with the viewer Vb, and is operated by the viewer Vb. Note that in this embodiment, for convenience of explanation, the viewer terminals 2a and 2b may be collectively referred to as the viewer terminal 2. Similarly, viewers Va and Vb may be collectively referred to as viewer V. Further, in this embodiment, a large number of viewer terminals 2 associated with a large number of viewers are provided in the video viewing analysis system 1, but for convenience of explanation, only two of the large number of viewer terminals Only the user terminals 2a and 2b are shown in FIG. The type of viewer terminal 2 is not particularly limited, and the viewer terminal 2 may be, for example, a smartphone, a personal computer, a tablet, or a wearable device (for example, a head-mounted display or an AR display).

Next, with reference to FIG. 2, the hardware configuration of the viewer terminal 2 will be described below. FIG. 2 is a diagram showing an example of the hardware configuration of the viewer terminal 2. As shown in FIG. As shown in FIG. 2, the viewer terminal 2 includes a control section 20, a storage device 21, an imaging section 22, a communication section 23, an input operation section 24, a display section 25, a speaker 26, and an RTC ( Real Time Clock) 28. These elements making up the viewer terminal 2 are connected to a communication bus 29.

The control unit 20 includes a memory and a processor. The memory is configured to store computer readable instructions (programs). For example, the memory includes a ROM (Read Only Memory) in which various programs are stored, a RAM (Random Access Memory) having a plurality of work areas in which various programs executed by a processor are stored, and the like. The processor includes, for example, at least one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). The CPU may be composed of multiple CPU cores. A GPU may be configured by multiple GPU cores. The processor may be configured to load a designated program from various programs incorporated in the storage device 21 or the ROM onto the RAM, and execute various processes in cooperation with the RAM.

The storage device 21 is, for example, a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory, and is configured to store programs and various data. The imaging unit 22 is configured to acquire video data showing the surrounding environment of the viewer terminal 2 through photography. In particular, the imaging unit 22 is a camera configured to generate image data or video data showing the surrounding environment of the viewer terminal 2 through photography, and includes an image sensor (for example, a CCD sensor or a CMOS sensor), and an image sensor drive processing circuit. In this embodiment, the control unit 20 functions as a line-of-sight tracking unit that detects changes in the line of sight of the viewer V based on the video data acquired by the imaging unit 22. Furthermore, the control unit 20 functions as a facial expression tracking unit that detects changes in the facial expression of the viewer V based on the video data.

The communication unit 23 includes a wireless communication module and/or a wired communication module for communicating with an external device connected to the communication network 8. The wireless communication module is configured to wirelessly communicate with an external device such as a base station or a wireless LAN router, and includes a transmitting/receiving antenna and a wireless transmitting/receiving circuit. The wireless communication module may be a wireless communication module compatible with short-range wireless communication standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or may be an X generation wireless communication module that uses a SIM (Subscriber Identity Module). It may be a wireless communication module compatible with a mobile communication system (for example, a fourth generation mobile communication system such as LTE).

The input operation unit 24 is, for example, a touch panel, a mouse, and/or a keyboard arranged over the video display of the display unit 25, and receives input operations from the viewer V and performs operations corresponding to the input operations. configured to generate a signal. The display unit 25 includes, for example, a video display and a video display circuit that drives and controls the video display. The display unit 25 has a display screen 27 on which a moving image is displayed (see FIG. 6).

The speaker 26 is configured to output the audio of the video to the outside based on the audio data included in the video. The RTC 28 is configured to obtain information indicating the current time.

Returning to FIG. 1, the server 3 is communicably connected to the viewer terminal 2 and the company terminal 4 via the communication network 8. The server 3 transmits video data to each of the plurality of viewer terminals 2 via the communication network 8, and also transmits a composite video (described later) in which a gaze point heat map is superimposed on the video to the company terminal 4. . The server 3 may be configured by a plurality of servers. The server 3 functions as a web server configured to provide a video viewing analysis application as a web application. In this regard, the server 3 is configured to send data (for example, HTML files, CSS files, image and video files, program files, etc.) for displaying the video viewing analysis screen on the web browser of the company terminal 4. has been done. In this way, the server 3 functions as a server for providing SaaS (System as a Service). The server 3 may be constructed on-premises or may be a cloud server. Further, the server 3 functions as a data management server that manages a plurality of video data, gaze point data of each viewer V, and facial expression data of each viewer V.

The hardware configuration of the server 3 will be described below with reference to FIG. 3. FIG. 3 is a diagram showing an example of the hardware configuration of the server 3. As shown in FIG. As shown in FIG. 3, the server 3 includes a control section 30, a storage device 31, an input/output interface 32, a communication section 33, an input operation section 34, and a display section 35. These elements making up the server 3 are connected to a communication bus 36.

The control unit 30 includes a memory and a processor. The memory is configured to store computer readable instructions. In particular, the memory may store a video viewing analysis program for causing the processor to execute a series of processes (video viewing analysis method) executed by the server 3 in FIGS. 5 and 8. The memory is composed of ROM and RAM. The processor includes at least one of a CPU, an MPU, and a GPU.

The storage device 31 is, for example, a storage device such as an HDD, SSD, or flash memory, and is configured to store programs and various data. The storage device 31 stores a plurality of moving image data, gaze point data of each viewer V, and facial expression data of each viewer V. The storage device 31 also stores a viewer information table related to information of each viewer V and a user information table related to each user U who uses the video viewing analysis application. The viewer information table includes attribute information of each viewer V. For example, the viewer information table may include at least one of identification information, gender information, age information, household number information, address information, and occupation information of each viewer V. The user information table may include identification information, attribute information, login information, etc. of each user U.

The input/output interface 32 is an interface that enables connection between an external device and the server 3, and includes an interface that conforms to a predetermined communication standard such as the USB standard or the HDMI (registered trademark) standard. The communication unit 33 may include various wired communication modules for communicating with external terminals on the communication network 8. The input operation unit 34 is, for example, a touch panel, a mouse, and/or a keyboard, and is configured to receive an input operation from an operator and to generate an operation signal according to the input operation from the operator. The display unit 35 includes, for example, a video display and a video display circuit.

The company terminal 4 is a terminal operated by a user U who uses a video viewing analysis application provided by the server 3. In this embodiment, a plurality of company terminals 4 are provided in the video viewing analysis system 1 (in other words, in this embodiment, a plurality of users U are using the video viewing analysis application). For convenience of explanation, only one company terminal 4 is shown in FIG. 1.

Referring to FIG. 4, the hardware configuration of the company terminal 4 will be described below. FIG. 4 is a diagram showing an example of the hardware configuration of the company terminal 4. As shown in FIG. As shown in FIG. 4, the company terminal 4 may be, for example, a personal computer, a smartphone, a tablet, or a wearable device attached to the user U. The company terminal 4 has a WEB browser. It is assumed that the video viewing analysis application operates on the web browser of the company terminal 4. The company terminal 4 includes a control section 40, a storage device 41, an input/output interface 42, a communication section 43, an input operation section 44, and a display section 45. These elements are connected to a communication bus 46.

The control unit 40 includes a memory and a processor. The memory is configured to store computer readable instructions (programs). For example, the memory is composed of ROM and RAM. The processor includes, for example, at least one of a CPU, an MPU, and a GPU.

The storage device 41 is, for example, a storage device such as an HDD, SSD, or flash memory, and is configured to store programs and various data. The input/output interface 42 is an interface (eg, USB, HDMI, etc.) that enables connection between an external device and the company terminal 4. The communication unit 43 is configured to connect the company terminal 4 to the communication network 8, and includes a wireless communication module and/or a wired communication module. The input operation unit 44 is, for example, a touch panel, a mouse, and/or a keyboard, and is configured to receive input operations from the user U and to generate operation signals in accordance with the input operations performed by the user U. The display section 45 includes, for example, a video display and a video display circuit. In this embodiment, a video viewing analysis screen 70 (see FIG. 9) is displayed on the display unit 45.

Next, a series of processes executed between each of the plurality of viewer terminals 2 and the server 3 in the video viewing analysis system 1 according to the present embodiment will be described with reference to FIG. 5. FIG. 5 is a sequence diagram for explaining a series of processes executed between the viewer terminal 2 and the server 3 in the video viewing analysis system 1. Note that for convenience of explanation, a series of processes executed between a single viewer terminal 2 and the server 3 is illustrated in FIG. 5, but in this embodiment, each of the plurality of viewer terminals 2 is It is assumed that a series of processes shown in 5 are executed. In addition, in the series of processes shown in FIG. 5, the viewer V provides the server 3 with point-of-gaze data and facial expression data of the viewer V regarding the predetermined video by viewing the predetermined video provided by the server 3. shall be taken as a thing. In this regard, an operating company that operates the server 3 and provides a video viewing analysis application may provide the viewer V with an incentive for viewing the video. For example, the company may award viewer V with predetermined points or electronic money for each viewing of a video.

As shown in FIG. 5, in step S1, the viewer terminal 2 sends a video viewing request to the server 3 to view the video in response to the viewer V's input operation on the input operation unit 24 (for example, a touch panel). Send. Next, after receiving the video viewing request from the viewer terminal 2, the server 3 transmits the video stored in the storage device 31 to the viewer terminal 2 (step S2). The video may be transmitted in a streaming format or may be transmitted to the viewer terminal 2 in a download format. The video transmitted to the viewer terminal 2 (that is, the video subject to viewing analysis) may include advertising content. For example, the video may be an advertising video played on a video platform such as YouTube (registered trademark).

Next, the viewer terminal 2 starts playing the video (step S3). As shown in FIG. 6, the viewer terminal 2 displays the moving image on the display screen 125 of the display unit 25. Furthermore, the viewer terminal 2 starts acquiring gaze point data and facial expression data of the viewer V at the same time as starting the reproduction of the video (step S4). As shown in step S5, the viewer terminal 2 continues to acquire the gaze point data and facial expression data of the viewer V until the reproduction of the video ends. On the other hand, the viewer terminal 2 finishes acquiring the gaze point data and facial expression data of the viewer V when the playback of the video ends, and then transmits the gaze point data and facial expression data to the server 3 (step S6 ).

<Method for acquiring gaze point data>
The viewer terminal 2 acquires video data indicating the viewer V through photography using the imaging unit 22, and then acquires point-of-gaze data of the viewer V based on the video data. In this respect, the control unit 20 of the viewer terminal 2 functions as a line-of-sight tracking unit that detects changes in the line of sight of the viewer V based on the video data acquired by the imaging unit 22. As an example of a method for acquiring gaze point data, the control unit 20 identifies the viewer V and the eyes of the viewer V shown in the predetermined frame based on a predetermined frame (still image) of the video data, and then Then, position information indicating the position of the viewer V's eyes with respect to the imaging unit 22 is acquired. Next, the control unit 20 specifies the rotation angle of the iris of the eyeball of the viewer V based on the predetermined frame, and then transmits line-of-sight information indicating the direction of the line-of-sight L (see FIG. 6) of the viewer V. get. Further, the control unit 20 acquires face orientation information indicating the direction of the viewer V's face based on the predetermined frame. Note that the control unit 20 may acquire line-of-sight information based on a three-dimensional eyeball model (passive method), or may acquire line-of-sight information based on a corneal reflex method (active method). When line-of-sight information is acquired based on the corneal reflection method, the viewer terminal 2 further includes an infrared light emitting element (for example, an infrared LED) configured to emit infrared rays and an infrared camera (infrared image sensor). Good too.

Next, the control unit 20 determines the point of gaze P, which is the point where the line of sight L of the viewer V and the display screen 125 of the viewer terminal 2 intersect, based on the position information, line of sight information, and face orientation information. Get the position (X, Y) (XY coordinates). Here, the position of the gaze point P where the line of sight L and the display screen 125 intersect may be acquired as a position on the XY coordinate space set on the display screen 125. In this way, the control unit 20 can acquire the position of the gaze point P where the line of sight L and the display screen 125 intersect from a predetermined frame of video data. The control unit 20 may acquire the position of the point of interest P at an acquisition rate corresponding to the frame rate of the video data, or may acquire the position of the point of interest P at a rate different from the frame rate. For example, when the frame rate of video data is 30 fps, the acquisition rate of the gaze point P may be 30 Hz, or may be an acquisition rate lower than 30 Hz (for example, 15 Hz).

Next, the control unit 20 converts the position of the gaze point P of the viewer V from the position in the XY coordinate space set on the display screen 125 to the position on the XY coordinate space set in the display area S of the video. . In this regard, if the video is not displayed on the display screen 125 in full screen mode, the XY coordinate space set in the display area S of the video does not match the XY coordinate space set on the display screen 125. do not have. Therefore, the position of the gaze point P is converted from the position on the XY coordinate space set on the display screen 125 to the position on the XY coordinate space set on the display area S of the moving image. Note that when the video is displayed on the display screen 125 in full screen mode, the XY coordinate space set in the display area S of the video matches the XY coordinate space set on the display screen 125. It is not necessary to execute the coordinate transformation process.

Furthermore, the control unit 20 acquires the position of the point of interest P on the display area S as well as the playback time of the video at the time the position of the point of interest P was acquired. For example, if the playback time of the video when the position (X1, Y1) of the gaze point P1 is acquired is t1, the position (X1, Y1) of the gaze point P1 on the display area S and the playback time t1 are The position (X1, Y1) of the point of interest P1 and the playback time t1 are stored in association with each other. In this manner, the control unit 20 can acquire point-of-gaze data indicating temporal changes in the position of the point of interest P on the display area S. For example, if the point of interest acquisition rate is 30 Hz and the playback time of the video is 30 seconds, the number of points of interest P included in the point of interest data is 900.

<Facial expression data acquisition method>
The viewer terminal 2 acquires video data showing the viewer V through photography using the imaging unit 22, and then acquires facial expression data of the viewer V based on the video data. In this respect, the control unit 20 of the viewer terminal 2 functions as a facial expression tracking unit that detects changes in the facial expression of the viewer V based on the video data acquired by the imaging unit 22. Here, the facial expression data is data indicating temporal changes in facial expression information of the viewer V. More specifically, the facial expression data indicates temporal changes in probability information of viewer V's facial expression elements (in this example, five types of facial expression elements: happy, sad, neutral (no expression), angry, and surprised). It is data.

As an example of a method for acquiring facial expression data, the control unit 20 identifies the face of the viewer V shown in the predetermined frame based on the predetermined frame (still image) of the video data M, and then identifies the face of the viewer V shown in the predetermined frame. A plurality of facial feature points K included in V's face are acquired (see FIG. 7). As shown in FIG. 7, in this example, 24 facial feature points K related to each of the eyebrows, eyes, nose, and mouth are acquired. Next, the control unit 20 calculates a plurality of facial feature amounts based on the plurality of acquired facial feature points K, and then combines the calculated facial feature amounts with an expression recognition model (learning model). Based on this, probability information of each facial expression element of the viewer V is obtained. In this facial expression recognition model, the value of each unit in the input layer indicates a corresponding one of multiple facial features, and the value of each unit in the output layer indicates the probability of a corresponding one of multiple facial features. Show information. The probability information for each facial expression element may be expressed as a percentage. In particular, the probability of the expression element ``happy,'' the probability of the expression element ``sad,'' the probability of the expression element ``neutral,'' the probability of the expression element ``angry,'' and the probability of the expression element ``surprised.'' Expressed as percentage %. In this way, the control unit 20 can acquire probability information of each facial expression element of the viewer V from a predetermined frame.

The control unit 20 may acquire the facial expression information of the viewer V at an acquisition rate corresponding to the frame rate of the video data, or may acquire the facial expression information of the viewer V at a rate different from the frame rate. . For example, when the frame rate of video data is 30 fps, the acquisition rate of facial expression information may be 30 Hz, or may be an acquisition rate lower than 30 Hz (for example, 15 Hz).

The control unit 20 also acquires the facial expression information as well as the playback time of the video at the time the facial expression information was acquired. For example, if the playback time of the video when the probability information of each facial expression element is acquired is t1, the probability information and playback time t1 of each facial expression element are associated with each other, and the probability information and the playback time t1 are be remembered. In this way, the control unit 20 can acquire facial expression data indicating temporal changes in facial expression information of the viewer V.

Returning to FIG. 5, in step S7, the server 3 receives the gaze point data and facial expression data of the viewer V regarding the video from each viewer terminal 2, and then receives the gaze point data and facial expression data of each viewer V. The gaze point data and facial expression data of each viewer V are stored in the storage device 31 while being associated with the video to be viewed. Note that in the example shown in FIG. 5, one type of video is transmitted to each viewer terminal 2 for convenience of explanation, but multiple types of video may be transmitted to each viewer terminal 2. In this case, the gaze point data and the facial expression data may be associated with both the viewer's identification information and the video's identification information. For example, the gaze point data and facial expression data of the viewer Va transmitted from the viewer terminal 2a may be associated with the identification information of the viewer Va and the identification information of the video to be viewed. Similarly, the gaze point data and facial expression data of the viewer Vb transmitted from the viewer terminal 2b may be associated with the identification information of the viewer Vb and the identification information of the video to be viewed. In this way, through the series of processes shown in FIG. 5, it becomes possible to accumulate the gaze point data and facial expression data of each viewer V in the server 3.

Next, with reference to FIG. 8, a series of processes executed between the server 3 and the company terminal 4 in the video viewing analysis system 1 according to the present embodiment will be described below. FIG. 8 is a sequence diagram for explaining a series of processes executed between the server 3 and the company terminal 4 in the video viewing analysis system 1. In this example, the user U who operates the company terminal 4 is engaged in digital marketing work related to videos, and the user U who operates the company terminal 4 is engaged in digital marketing work related to videos, and the user U who operates the company terminal 4 is engaged in digital marketing work related to videos. They want to understand the insights. The video viewed by each viewer V in the series of processes shown in FIG. 5 is a video related to the digital marketing work of the user U (for example, an advertising video displayed on a video platform, etc.).

As shown in FIG. 8, in step S10, the company terminal 4 transmits user U's login information (eg, login ID and login password) to the server 3 through user U's input operation. In step S11, the server 3 authenticates the user U based on the login information sent from the company terminal 4. Note that the authentication method for user U is not limited to login authentication. After authenticating the user U, the server 3 identifies the video associated with the user U by referring to the user information table stored in the storage device 31.

In step S12, the server 3 transmits data for displaying the video viewing analysis screen 70 shown in FIG. 9 (hereinafter referred to as video viewing analysis screen data) to the company terminal 4 via the communication network 8. The video viewing analysis screen data may be a file (for example, an HTML file, a CSS file, a program file, etc.) for displaying the video viewing analysis screen 70 on the web browser of the company terminal 4. Here, the program file may be a program file executable on a web browser such as JavaScript.

As shown in FIG. 9, the video viewing analysis screen 70 includes a video display area 71, an object of interest display area 76, a facial expression information display area 72, and a facial expression/interest level information display area 73. Further, the video viewing analysis screen 70 includes a viewer selection button 74 and a video selection button 75. In the video display area 71, a composite video 63, which will be described later, is displayed. Information regarding the object of interest within the video is displayed in the object of interest display area 76. In the facial expression information display area 72, comprehensive facial expression information in a radar chart format is displayed. The facial expression/interest level information display area 73 displays comprehensive facial expression information of the time series graph and comprehensive interest level information of the time series graph.

In step S13, the company terminal 4 selects the selected video and the extraction conditions for the viewer V displayed on the video viewing analysis screen 70 in response to the user U's input operation on the video viewing analysis screen 70. Specifically, the company terminal 4 selects the selected video displayed in the video display area 71 in response to the user U's input operation on the video selection button 75. The selected video that can be selected through the video selection button 75 is the video associated with the user U. In response to the user's input operation on the viewer selection button 74, extraction conditions for viewer V are selected. More specifically, as shown in FIG. 12, the company terminal 4 displays an extraction condition display screen 80 for specifying extraction conditions for the viewer V in response to the user U's input operation on the viewer selection button 74. is displayed on the video viewing analysis screen 70.

In the extraction condition display screen 80 shown in FIG. 12, the extraction condition for viewer V is associated with viewer V's gender information, viewer V's age information, and viewer V's household number information. For example, if a viewer V in his or her 40s is selected as the extraction condition, the server 3 acquires the gaze point data of the viewer V in his or her 40s in step S15. Further, the user U can specify an extraction condition associated with at least one of gender information, age information, and household information through the extraction condition display screen 80. For example, as shown in FIG. 12, when the user U specifies the extraction condition (item name: target) regarding a woman in her 20s living in a one-person household, in step S15, the server 3 determines that Viewer V's gaze point data is acquired.

Note that although FIG. 12 shows three examples of extraction conditions for the viewer V: gender information, age information, and household number information, the present embodiment should not be limited to these. In this regard, viewer V's address information, occupation information, marriage information, and educational background information may be further associated with viewer V's extraction conditions. In this way, the extraction condition for viewer V may be associated with at least one of viewer V's age information, gender information, address information, occupation information, household number information, marital information, and educational background information. .

Furthermore, the extraction conditions for the viewer V may be further associated with at least one of the viewer V's questionnaire information, survey information, and past behavior history information (for example, frequency of going to the movie theater, etc.).

Next, in step S14, the company terminal 4 requests the server 3 to transmit a composite video in response to the user U's input operation. In this regard, in response to the user U's input operation on the playback button for a composite video provided on the video viewing analysis screen 70, the company terminal 4 selects a composite video associated with the selected video selected through the video selection button 75. A video transmission request is sent to the server 3. Further, the company terminal 4 transmits information regarding the extraction conditions of the viewer V to the server 3 along with a request to transmit the composite video.

In step S15, the server 3 receives the synthetic video transmission request and the information regarding the extraction conditions of the viewer V from the company terminal 4, and then stores the gaze point data of the viewer V that matches the extraction conditions in the storage device 31. Extract from. In this respect, the server 3 extracts the identification information of the viewer V that matches the extraction conditions by referring to the viewer information table related to the information of each viewer V. After that, the server 3 extracts the point of gaze data associated with both the identification information of the selected video and the extracted identification information of the viewer V. For example, as shown in FIG. 13, if the extraction conditions for viewer V are age: 20s, gender: female, and number of people in the household: one person, the server 3 selects the following: age: 20s, gender: female, In addition, after identifying the identification information of the viewer V who satisfies all the conditions of household size: one person household, the identification information of the viewer V associated with both the identification information of the specified viewer V and the identification information of the selected video. Extract gaze point data. On the other hand, as shown in FIG. 9, when the extraction condition for viewer V is all viewers, the gaze point data of viewer V associated with the identification information of the selected video is extracted.

In step S16, the server 3 generates a gaze point heat map (gazing point visualization An example of a map) is generated. In the gaze point heat map, temporal changes in a collection of gaze points P of a plurality of viewers V are visualized. The frame rate of the gaze point heat map may be the same as the acquisition rate of the gaze point data. For example, if the gaze points of viewers V1 to Vn (n is an integer greater than or equal to 2) at the playback time t1 of the video are P1 to Pn, the gaze points P1 to Pn are visualized in the gaze point heat map at the playback time t1. There is. In the gaze point heat map, if there are a large number of gaze points that overlap a predetermined area (that is, if the predetermined area is an area that is being gazed at by multiple viewers V), the predetermined area is colored in a warm color such as red. May be colored with. On the other hand, if the number of gaze points that overlap a predetermined area is small (that is, if the predetermined area is an area that is not often watched by multiple viewers V), the predetermined area is colored in a cool color such as blue. May be colored. In this way, as the number of gaze points overlapping a predetermined region increases, the coloring of the predetermined region may gradually change from a cold color to a warm color. Furthermore, if there are no gaze points included in a predetermined area (that is, if the predetermined area is an area that is not gazed at by a plurality of viewers V at all), the predetermined area does not need to be colored. In this respect, a region containing only a gaze point treated as noise (outlier) may not be colored at all.

In this embodiment, a gaze point heat map is described as an example of a gaze point visualization map that visualizes temporal changes in the gaze points of a plurality of viewers V, but gaze point visualization maps are limited to gaze point heat maps. It's not something you can do.

Furthermore, in this embodiment, when the extraction condition for the viewer V is all viewers, a gaze point heat map that visualizes temporal changes in the positions of the gaze points P of all the viewers V is generated. . In this way, the gaze point heat map that is generated changes according to the extraction conditions of the viewer V (in other words, the attributes of the viewer V).

In step S17, the server 3 generates a composite video 63 in which the gaze point heat map 62 is superimposed on the video 60 (see FIG. 10). In particular, the server 3 generates a composite video 63 in which the gaze point heat map 62 and the video 60 are superimposed on the video 60 in a state where the gaze point heat map 62 and the video 60 are temporally synchronized. In this respect, if the frame rate of the video 60 and the frame rate of the gaze point heat map 62 are the same, each frame Fb1 to Fb6 constituting the gaze point heat map 62 is different from the video 60, as shown in FIG. is superimposed on a corresponding one of the plurality of frames Fa1 to Fa6 that make up the frame. For example, frame Fb3 of the gaze point heat map 62 corresponding to playback time t3 of video 60 is superimposed on frame Fa3 of video 60 corresponding to playback time t3. On the other hand, when the frame rate of the gaze point heat map 62 is half the frame rate of the video 60, each of the plurality of frames Fb1 to Fb3 configuring the gaze point heat map 62 is a video image, as shown in FIG. The frame 60 is superimposed on at least one corresponding one (two frames in this example) of the plurality of frames Fa1 to Fa6 constituting the frame 60. In the example shown in FIG. 11(b), frame Fb1 of the gaze point heat map 62 is superimposed on frame Fa1 and frame Fa2 of the video 60. As described above, in this embodiment, "a state in which the gaze point heat map 62 and the moving image 60 are temporally synchronized" includes not only the state shown in FIG. 11(a) but also the state shown in FIG. 11(b). shall be held.

In step S18, the server 3 transmits the composite video 63 to the company terminal 4. The composite video 63 may be transmitted in a streaming format or in a download format. After receiving the composite video 63 from the server 3, the company terminal 4 displays the composite video 63 in the video display area 71 of the video viewing analysis screen 70. In this way, the composite video 63 is played back on the company terminal 4 (step S19).

In step S20, the server 3 acquires information regarding the object of interest (area of interest) in the video 60, and then transmits the information regarding the object of interest to the company terminal 4. In this regard, based on the gaze point data of the viewer V extracted in step S15, the server 3 selects an object of interest (area of interest) in the video 60 that overlaps a lot with the gaze points of a plurality of viewers V that match the extraction conditions. ), and then obtains information regarding the object of interest. As for the transmission timing of the information, the information may be transmitted after the composite video 63 is played back. As shown in FIG. 9, information regarding the object of interest within the video 60 is displayed in the object of interest display area 76 of the video viewing analysis screen 70 (step S21).

Information regarding the object of interest includes information indicating the specified object of interest (area of interest), attention degree information regarding the degree of attention of the object of interest, time of appearance information regarding the appearance time of the object of interest, and average information regarding the average attention degree of the object of interest. It may also include attention level information. Information indicating the identified object of interest may be displayed as a trimmed image 176 indicating a partial area of the moving image 60, as shown in FIG. For example, the server 3 identifies the top three objects of interest in the video 60 that overlap many of the gaze points of the multiple viewers V, and then displays the trimmed images 176 of the three objects of interest as the objects of interest in the video 60 that overlap many of the gaze points of the multiple viewers V. It may also be displayed within the object display area 76. In this case, the object of interest that overlaps the viewer V's gaze point the most becomes the object of interest that ranks first in the attention ranking.

The server 3 also determines the degree of attention of each object of interest based on the number of points of interest of the viewer V that overlap with the objects of interest (in this example, the top three objects of interest in the attention ranking) in each frame of the video 60. You may. The degree of attention of the object of interest may be displayed as a percentage, or as a grade (A, B, C, etc.).

The server 3 may determine whether or not an object of interest (in this example, the top three objects of interest in the attention ranking) exists in each frame of the video 60, and then determine the appearance time of the object of interest. For example, if the video has a frame rate of 60 fps and a playback time of 30 seconds, the total number of frames in the video is 1800. Here, if the attention object O1, which ranks first in the attention ranking, exists within 600 frames, the appearance time of the attention object O1 is 30 seconds x (600 frames/1800 frames) = 10 seconds.

The server 3 specifies the number of viewer V's gaze points that overlap with the attention object (in this example, the top three attention objects in the attention ranking) in each frame of the video 60, and then determines the number of viewer V's gaze points that overlap with the attention object. Determine the average number of fixation points. Thereafter, the server 3 may determine the average degree of attention of the object of interest based on the average number of gaze points of the viewer V that overlap the object of interest. The average attention level of the attention object may be displayed as a percentage, or as a grade (A, B, C, etc.).

Next, in step S22, the server 3 extracts facial expression data of the viewer V that matches the extraction conditions from the storage device 31. In this respect, the server 3 extracts the identification information of the viewer V that matches the extraction conditions by referring to the viewer information table related to the information of each viewer V. Thereafter, the server 3 extracts facial expression data associated with both the identification information of the selected video and the extracted identification information of the viewer V. For example, as shown in FIG. 13, if the extraction conditions for viewer V are age: 20s, gender: female, and number of people in the household: one person, the server 3 selects the following: age: 20s, gender: female, In addition, after identifying the identification information of the viewer V who satisfies all the conditions of household size: one person household, the identification information of the viewer V associated with both the identification information of the specified viewer V and the identification information of the selected video. Extract facial expression data. On the other hand, as shown in FIG. 9, when the extraction condition for viewer V is all viewers, the facial expression data of viewer V associated with the identification information of the selected video is extracted.

In step S23, the server 3 generates comprehensive facial expression information and comprehensive interest level information based on the facial expression data of the viewer V extracted in step S22. Thereafter, the server 3 transmits the generated comprehensive facial expression information and comprehensive interest level information to the company terminal 4. As for the transmission timing of these pieces of information, these pieces of information may be sent after the composite video 63 is played back. As shown in FIG. 9, the company terminal 4 displays comprehensive facial expression information displayed in the form of a radar chart in the facial expression information display area 72 of the video viewing analysis screen 70, and also displays comprehensive facial information displayed in the form of a time series graph. The facial expression information is displayed in the facial expression/interest level information display area 73 of the video viewing analysis screen 70 (step S24). Further, the company terminal 4 displays comprehensive interest level information displayed as a time series graph in the facial expression/interest level information display area 73.

The comprehensive facial expression information indicates comprehensive facial expression information of the viewer V that matches the extraction conditions, and includes comprehensive facial expression information in a radar chart format and comprehensive facial expression information in a time series graph format. The comprehensive facial expression information in the form of a radar chart indicates average probability information (percentage %) of each comprehensive facial expression element (happy, sad, neutral, angry, surprised) of the viewer V that meets the extraction conditions. For example, comprehensive facial expression information in the form of a radar chart includes the average probability of the facial expression element "happy" for multiple viewers V that matches the extraction conditions, the average probability of the facial expression element "sad" for the multiple viewers V, and The average probability of the facial expression element "neutral" of the plurality of viewers V, the average probability of the facial expression element "angry" of the plurality of viewers V, the average probability of the facial expression element "surprise" of the plurality of viewers V, May include.

For example, when the extracted viewers V include viewers Va, Vb, and Vc, the server 3 analyzes each facial expression element of the viewer Va based on facial expression data indicating temporal changes in each facial expression element of the viewer Va. Determine the average probability of Similarly, the server 3 determines the average probability of each facial expression element of the viewer Vb based on facial expression data indicating temporal changes in each facial expression element of the viewer Vb, and also determines the average probability of each facial expression element of the viewer Vc. The average probability of each facial expression element of the viewer Vc is determined based on the facial expression data showing the change in the target. Thereafter, the server 3 determines the overall average probability of each facial expression element of the viewers Va to Vc based on the average probability of each facial expression element of the viewers Va to Vc. In this regard, if the average probabilities of the facial expression element "happy" for viewers Va, Vb, and Vc are Ra1, Rb1, and Rc1, then the average probability of the overall facial expression element "happy" for viewers Va to Vc is Rave1. is Rave1=(Ra1+Rb1+Rc1)/3. Similarly, if the average probabilities of the facial expression element "sad" for viewers Va, Vb, and Vc are Ra2, Rb2, and Rc2, the average probability Rave2 of the overall facial expression element "sad" for viewers Va to Vc is , Rave2=(Ra2+Rb2+Rc2)/3.

Further, the server 3 may calculate the positive degree and negative degree of the extracted viewer V with respect to the video 60 as a percentage based on the average probability of each comprehensive facial expression element of the extracted viewer V. . Furthermore, the server 3 may transmit information indicating the positive degree and the negative degree to the company terminal 4. As shown in FIG. 9, information indicating the degree of positivity and degree of negativity may be displayed within the facial expression information display area 72. For example, when the average probability of the facial expression element "happy" is high, the value of the positivity level may be large. On the other hand, when the average probability of the facial expression element "anger" is high, the value of the degree of negativity may become large. The server 3 uses a relational expression or a learning model in which the overall average probability of each facial expression element is set as an input value, and the positive degree and negative degree values are set as output values. The positive degree and the negative degree may be calculated from the average probability of facial expression elements.

The comprehensive facial expression information in the form of a time-series graph indicates temporal changes in comprehensive facial expression elements (happy, sad, neutral, angry, surprised) of the viewer V that match the extraction conditions. For example, the comprehensive facial expression information in a time-series graph format includes information indicating the temporal change in the average probability of the facial expression element "happy" of a plurality of viewers V that matches the extraction conditions, and information indicating the temporal change in the average probability of the facial expression element "happy" of the plurality of viewers V who match the extraction conditions. information indicating a temporal change in the average probability of "sad", information indicating a temporal change in the average probability of the facial expression element "neutral" of the plurality of viewers V, and information indicating a temporal change of the average probability of the facial expression element "angry" of the plurality of viewers V. , and information indicating a temporal change in the average probability of the facial expression element "surprise" of the plurality of viewers V.

For example, when the extracted viewers V include viewers Va, Vb, and Vc, the server 3 stores information indicating the temporal change in the average probability of the facial expression element "happy" of viewers Va to Vc, and - Information indicating the temporal change in the average probability of the facial expression element "sad" of ~Vc, information indicating the temporal change of the average probability of the facial expression element "neutral" of the viewers Va~Vc, and facial expressions of the viewers Va~Vc After calculating the information indicating the temporal change in the average probability of the element "anger" and the information indicating the temporal change in the average probability of the facial expression element "surprise" of viewers Va to Vc, these pieces of information are combined in chronological order. It is sent to the company terminal 4 as comprehensive facial expression information in graph format.

Further, the server 3 may calculate comprehensive interest level information in a time series graph format based on the comprehensive facial expression information in a time series graph format. The comprehensive interest level information indicates a temporal change in the overall interest level (percentage %) of the viewer V who meets the extraction conditions in the video 60. For example, if the overall average probability of each facial expression element at playback time t1 is R1 to R5, the overall interest level I at playback time t1 may be calculated based on the average probabilities R1 to R5. In this regard, the server 3 uses a relational expression or a learning model in which the overall average probabilities R1 to R5 of each facial expression element are set as input values, and the overall interest level I is set as an output value. Then, the overall interest level I may be calculated from the overall average probabilities R1 to R5 of each facial expression element.

In addition, in this embodiment, when the extraction condition for the viewer V is all the viewers V, the comprehensive facial expression information indicates the comprehensive facial expression information of all the viewers V, and the comprehensive interest level information indicates all the viewers V. 3 shows a temporal change in the overall interest level of viewer V in the video 60. In this way, the comprehensive expression information and the comprehensive interest level information change depending on the extraction conditions of the viewer V (in other words, the attributes of the viewer V to be extracted).

Furthermore, in the present embodiment, the comprehensive interest level information of a plurality of viewers V is calculated based on the comprehensive facial expression information (particularly, the temporal change in the average probability of each facial expression element of the viewer V). The degree information may be calculated based on the gaze point data of each viewer V. Specifically, the comprehensive interest level information is calculated based on the interest level data of each viewer V, and the interest level data of the viewer V is calculated based on the viewer V's gaze point data and the viewer V's blinking. It may be calculated based on information regarding the viewer V and drowsiness information of the viewer V. The information regarding the viewer V's blinking may be information indicating the number of times the viewer V blinks or information indicating whether or not the viewer V blinks. Information regarding viewer V's blinking may be specified based on each frame of viewer V's video data. The drowsiness information of the viewer V may be information indicating whether the viewer V is drowsy. Viewer V's drowsiness information may be specified based on each frame of viewer V's video data. For example, the interest level data of the viewer V includes information regarding whether the position of the gaze point P of the viewer V exists within the display area S of the video, information regarding the presence or absence of sleepiness of the viewer V, and information regarding the presence or absence of sleepiness of the viewer V. It may be calculated based on information regarding the presence or absence of blinking. Based on the interest level data of each viewer V, it becomes possible to calculate comprehensive interest level information of a plurality of viewers V.

Furthermore, in this embodiment, the information indicating the positive degree and the negative degree of the plurality of viewers V is calculated based on the facial expression data of each viewer V, but this information is calculated based on the gaze point of each viewer V. It may be calculated based on data.

Note that in this embodiment, the order of each process shown in FIG. 8 is not particularly limited. In addition, in the examples shown in FIGS. 9 and 13, a composite video 63 in which the gaze point heat map 62 is superimposed on the video 60 is displayed in the video display area 71; An alternative video 60 may be displayed. Even in this case, information regarding the object of interest within the moving image 60 may be displayed within the object of interest display area 76. Further, comprehensive facial expression information in a radar chart format is displayed in the facial expression information display area 72, and comprehensive facial expression information and comprehensive interest level information in a time series graph format are displayed in the facial expression/interest level information display area 73. good.

In addition, in this example, two pieces of information, comprehensive facial expression information and comprehensive interest level information, are displayed on the video viewing analysis screen 70. Only the information may be displayed on the video viewing analysis screen 70. Further, in this example, five types of facial expression elements, happy, sad, neutral, angry, and surprised, are employed as facial expression information, but the types of facial expression elements are not limited to these. For example, fear, happiness, enjoyment, disgust, etc. may be additionally employed as facial expression elements other than these.

According to the present embodiment, the composite video 63 in which the gaze point heat map 62 and the video 60 are superimposed on the video 60 is presented to the company terminal 4 in a state where the gaze point heat map 62 and the video 60 are temporally synchronized. By checking the composite video 63 displayed on the video viewing analysis screen 70, U can understand in detail the viewing behavior of the multiple viewers V with respect to the video 60. In particular, the user U can grasp in detail objects of interest (areas of interest) in the video 60 by a plurality of viewers V for each playback time of the video 60. In this way, it is possible to provide detailed insights regarding the video 60 (more specifically, detailed insights regarding digital marketing of video advertisements) to companies that desire video analysis.

Further, since information regarding the object of interest in the video 60 is displayed on the video viewing analysis screen 70, the user U can view information regarding the object of interest (in particular, a cropped image of the object of interest, information on the degree of attention of the object of interest, or average attention By checking the time information, appearance time of the object of interest, etc.), it is possible to obtain more detailed insight regarding the object of interest that exists in the video 60.

In particular, in this embodiment, based on the gaze point data of the viewers V that match the extraction conditions associated with the age information, gender information, household size information, etc. of the plurality of viewers V, viewing that satisfies these attribute conditions is performed. A gaze point heat map that visualizes temporal changes in the position of the gaze point of person V is generated. In this way, it is possible to provide the company with more detailed insight regarding the video 60 according to the viewer's attributes.

Furthermore, according to the present embodiment, comprehensive facial expression information and comprehensive interest level information in a radar chart format and a time series graph format are presented to the company terminal 4, so that the user U can see the information displayed on the video viewing analysis screen 70. By checking this information, it is possible to understand in detail the viewing behavior of the plurality of viewers V with respect to the video 60. In particular, the user U can grasp in detail the facial expressions (happy, sad, neutral, angry, surprised, etc.) and interest level of the viewer V toward the video 60 for each playback time. In this way, it is possible to provide detailed insights regarding videos (more specifically, detailed insights regarding digital marketing of video advertisements) to companies that desire video analysis.

In particular, in this embodiment, based on facial expression data of viewers V who match extraction conditions associated with age information, gender information, household size information, etc. of a plurality of viewers V, viewers who meet these attribute conditions are selected. Comprehensive facial expression information indicating the probability information of each comprehensive facial expression element of V is generated, and comprehensive interest level information indicating the overall interest level of the viewer V who meets the extraction conditions in the video 60 is generated. In this way, it is possible to understand in detail the viewing behavior of the viewer V who meets the extraction conditions through this information, so it is possible to provide the company with more detailed insight regarding the video 60.

Although the embodiments of the present invention have been described above, the technical scope of the present invention should not be interpreted to be limited by the description of the embodiments. This embodiment is an example, and those skilled in the art will understand that various changes can be made within the scope of the invention as set forth in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the scope of equivalents thereof.

1: Video viewing analysis system, 2, 2a, 2b: viewer terminal, 3: server, 4: company terminal, 8: communication network, 20: control unit, 21: storage device, 22: imaging unit, 23: communication section, 24: input operation section, 25: display section, 26: speaker, RTC: 28, 30: control section, 31: storage device, 32: input/output interface, 33: communication section, 34: input operation section, 35: Display unit, 40: Control unit, 41: Storage device, 42: Input/output interface, 43: Communication unit, 44: Input operation unit, 45: Display unit, 60: Video, 62: Gaze point heat map, 63: Composite video , 70: Video viewing analysis screen, 71: Video display area, 72: Facial expression information display area, 73: Interest level information display area, 74: Viewer selection button, 75: Video selection button, 76: Attention object display area, 80 : Extraction condition display screen, 125: Display screen, 176: Trimmed image, K: Facial feature point, L: Line of sight, M: Video data, P: Gaze point, S: Display area, U: User, V, Va, Vb :Audience

Claims (11)

A video viewing analysis system that analyzes insights of multiple viewers regarding videos,
a plurality of viewer terminals, each of which is associated with a corresponding one of the plurality of viewers and includes a display unit;
a server communicably connected to the plurality of viewer terminals via a network;
a company-side terminal communicably connected to the server via the network and equipped with a display unit;
The video viewing analysis system includes:
providing the video to each of the plurality of viewer terminals;
Displaying the video on the display section of each of the plurality of viewer terminals,
obtaining a plurality of facial expression data each indicating a temporal change in the probability of a plurality of facial expression elements of a corresponding one of the plurality of viewers for the video;
receiving extraction conditions for the plurality of viewers from the company terminal;
Extracting some of the plurality of facial expression data based on the extraction conditions,
After integrating the extracted several facial expression data, a comprehensive facial expression indicating a comprehensive facial expression in the time axis when the plurality of viewers viewed the video based on the combined facial facial expression data. generate information;
displaying the comprehensive facial expression information on a display section of the company terminal;
Video viewing analysis system. A video viewing analysis system that analyzes insights of multiple viewers regarding videos,
a plurality of viewer terminals, each of which is associated with a corresponding one of the plurality of viewers and includes a display unit;
a server communicably connected to the plurality of viewer terminals via a network;
a company-side terminal communicatively connected to the server via the network and equipped with a display unit;
The video viewing analysis system includes:
providing the video to each of the plurality of viewer terminals;
Displaying the video on the display section of each of the plurality of viewer terminals,
obtaining a plurality of facial expression data each indicating a temporal change in the probability of a plurality of facial expression elements of a corresponding one of the plurality of viewers for the video;
receiving extraction conditions for the plurality of viewers from the company terminal;
Extracting some of the plurality of facial expression data based on the extraction conditions,
After integrating the extracted facial expression data, a composite indicating the overall interest level in the time axis when the plurality of viewers watched the video based on the integrated facial expression data. Generate interest level information,
displaying the comprehensive interest level information on a display section of the company terminal;
Video viewing analysis system. The extraction conditions are:
Age information of the plurality of viewers;
Gender information of the plurality of viewers;
Address information of the plurality of viewers;
Occupational information of the plurality of viewers;
Household number information of the plurality of viewers;
Marriage information of the plurality of viewers;
educational background information of the plurality of viewers;
associated with at least one of
The video viewing analysis system according to claim 1 or 2 . the video includes advertising content;
The video viewing analysis system according to claim 1 or 2 . 2. The video viewing analysis system according to claim 1 , wherein the comprehensive facial expression information indicates an average probability of each facial expression element of the plurality of viewers throughout the video. 2. The video viewing analysis system according to claim 1 , wherein the comprehensive facial expression information indicates a temporal change in the average probability of each facial expression element of the plurality of viewers along the time series of the video. 3. The video viewing analysis system according to claim 2 , wherein the comprehensive interest level information indicates a temporal change in the comprehensive interest level of the plurality of viewers along the time series of the video. Calculating the positivity or negativity of the plurality of viewers with respect to the video based on the probability of each facial expression element of the plurality of viewers,
displaying the positive degree or the negative degree;
The video viewing analysis system according to claim 1 . A video viewing analysis method executed by a video viewing analysis system by analyzing insights of multiple viewers for a video, the method comprising:
The video viewing analysis system includes:
a plurality of viewer terminals, each of which is associated with a corresponding one of the plurality of viewers and includes a display unit;
a server communicably connected to the plurality of viewer terminals via a network;
a company-side terminal communicably connected to the server via the network and equipped with a display unit;
The video viewing analysis method is
providing the video to each of the plurality of viewer terminals;
Displaying the video on a display unit of each of the plurality of viewer terminals;
obtaining a plurality of facial expression data each indicating a temporal change in the probability of a plurality of facial expression elements of a corresponding one of the plurality of viewers for the video;
receiving extraction conditions for the plurality of viewers from the company terminal;
extracting some of the plurality of facial expression data based on the extraction conditions;
After integrating the extracted several facial expression data, a comprehensive facial expression indicating a comprehensive facial expression in the time axis when the plurality of viewers viewed the video based on the combined facial facial expression data. a step of generating information;
displaying the comprehensive facial expression information on a display section of the company terminal;
Video viewing analysis methods, including: A video viewing analysis method executed by a video viewing analysis system by analyzing insights of multiple viewers for a video, the method comprising:
The video viewing analysis system includes:
a plurality of viewer terminals, each of which is associated with a corresponding one of the plurality of viewers and includes a display unit;
a server communicably connected to the plurality of viewer terminals via a network;
a company-side terminal communicably connected to the server via the network and equipped with a display unit;
The video viewing analysis method is
providing the video to each of the plurality of viewer terminals;
Displaying the video on a display unit of each of the plurality of viewer terminals;
obtaining a plurality of facial expression data each indicating a temporal change in the probability of a plurality of facial expression elements of a corresponding one of the plurality of viewers for the video;
receiving extraction conditions for the plurality of viewers from the company terminal;
extracting some of the plurality of facial expression data based on the extraction conditions;
After integrating the extracted facial expression data, a composite indicating the overall interest level in the time axis when the plurality of viewers watched the video based on the integrated facial expression data. generating interest level information;
displaying the comprehensive interest level information on a display section of the company terminal;
Video viewing analysis methods, including: A video viewing analysis program that causes a computer to execute the video viewing analysis method according to claim 9 or 10 .

Images