JP2020126645A - Information processing method, terminal device and information processing device - Google Patents

Abstract

To provide an information processing method and the like capable of determining the emotion of a viewer who is watching a content, and providing an emotion report of the viewer to the content by using a determination result.SOLUTION: A terminal device displays contents and obtains face image data of the face of a user visually recognizing the displayed contents. The terminal device generates hemoglobin image data by extracting a pigment component corresponding to hemoglobin from the face image data, generates heartbeat data indicating heartbeat of the user on the basis of the hemoglobin image data, and performs frequency conversion on the heartbeat data to generate heartbeat variability data that indicates the user's heartbeat variability. For example, a server generates emotion data indicating the emotion of a user by analyzing the emotion of the user on the basis of the heartbeat variability data, and generates an emotion report to the content on the basis of the emotion data of the plurality of users.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to an information processing method, a terminal device, and an information processing device.

2. Description of the Related Art In recent years, a technique has been developed for determining a person's emotion of a subject from a captured image obtained by capturing a human face. Patent Document 1 discloses a technique for creating a heart rate variability spectrogram showing the heart rate variability of a subject person based on a face image captured using an RGB camera, and determining a stress state of the subject person based on the heart rate variability spectrogram. Is proposed. In the technique disclosed in Patent Document 1, a hemoglobin image is created from a face image by pigment component separation, and a heart rate variability spectrogram is created based on the hemoglobin image.

JP, 2017-29318, A

The heartbeat variability spectrogram created by the technique disclosed in Patent Document 1 can be used not only for determining the stress state of the person who is the subject but also for determining the emotion of the person. Therefore, the technique disclosed in Patent Document 1 is expected to be used in various systems that perform various processes by using the result of determining the emotion of a person.

The present disclosure has been made in view of such circumstances, and an object thereof is to determine the emotion of a viewer who views content, and to use the determination result to generate emotion data of the viewer for the content. Providing information processing method.

An information processing method according to an aspect of the present disclosure displays content on a display unit, acquires face image data of a face of a user who visually recognizes the displayed content, and extracts a pigment component corresponding to hemoglobin from the face image data. Extracting to generate hemoglobin image data, generating heartbeat data indicating the heartbeat of the user based on the hemoglobin image data, performing frequency conversion on the heartbeat data, and heartbeat fluctuation indicating heartbeat fluctuation of the user. The computer executes a process of generating data, analyzing the emotion of the user based on the heart rate variability data, and generating emotion data indicating the emotion of the user.

According to an aspect of the present disclosure, the emotion of the user who is visually recognizing can be sequentially analyzed based on the face image of the user who visually recognizes (views) the content, and thus the emotion of the user at each reproduction timing of the content can be acquired. In addition, a viewer (viewer) emotion report for the content can be generated based on the emotion analysis results of a plurality of users. Therefore, the viewer's emotional report can be provided to the content creator, and the content can be evaluated based on the viewer's emotional report.

It is a schematic diagram which shows the structural example of an emotion analysis system. It is a block diagram which shows the structural example of an emotion analysis system. It is a schematic diagram which shows the structural example of DB memorize|stored in a server and a terminal device. It is a block diagram which shows the function implement|achieved by the control part of a terminal device. It is explanatory drawing of the process by a terminal device. It is a block diagram which shows the function implement|achieved by the control part of a server. It is a schematic diagram which shows an example of a screen. It is a flow chart which shows a procedure of acquisition processing of heartbeat variability data by a terminal unit. It is a flowchart which shows the procedure of the emotion analysis process by a server. It is a block diagram which shows the structural example of the emotion analysis system of Embodiment 2. 9 is a flowchart showing a procedure of heartbeat fluctuation data acquisition processing by the terminal device of the second embodiment. 11 is a flowchart showing a procedure of heartbeat fluctuation data acquisition processing by the terminal device of the third embodiment. 11 is a flowchart showing a procedure of heartbeat fluctuation data acquisition processing by the terminal device of the third embodiment. It is a schematic diagram which shows an example of a screen. 9 is a flowchart showing a procedure of emotion analysis processing by the server of the third embodiment. 16 is a flowchart showing a procedure of health condition determination processing by the terminal device of the fourth embodiment.

Hereinafter, the information processing method, the terminal device, and the information processing device of the present disclosure will be specifically described based on the drawings showing the embodiment applied to the emotion analysis system that analyzes the emotion of the user who visually recognizes (views) the content. .. In the following embodiments, the content may be content including only images, or may be content including images and audio. Therefore, in the following, viewing the content includes not only the case of viewing the content including the image and the sound but also the case of visually recognizing the content including only the image. In addition, the viewer includes not only a user who views content including images and sounds, but also a user (viewer) who views content including only images.

(Embodiment 1)
FIG. 1 is a schematic diagram showing a configuration example of an emotion analysis system. The emotion analysis system according to the present embodiment includes a server (information processing device) 10 and a plurality of terminal devices 20, and the server 10 and each terminal device 20 perform information via a network N such as the Internet or a LAN (Local Area Network). Send and receive. The server 10 is a server computer, a personal computer, or the like, and a plurality of servers may be provided, a plurality of virtual machines provided in one server device, or a cloud server. May be done. The server 10 acquires and stores data regarding heartbeat variability of the user (viewer) who is viewing the content from each terminal device 20, processes to analyze emotions of each viewer based on the stored data, and analyzes. Various information processing is performed, such as processing for generating an emotion report for each content based on the result. The terminal device 20 is a tablet terminal, a personal computer, or the like, and is preferably a dedicated terminal used in the emotion analysis system. The terminal device 20 performs various types of information processing such as content reproduction processing (display processing) and processing of collecting data regarding heartbeat fluctuation of a user (viewer) who views the content being reproduced. The terminal device 20 is, for example, rented out to a user who has ordered food at a restaurant in a food court provided in a shopping mall or the like, and a notification terminal for notifying that the ordered food is completed until the ordered food is completed. A device configured to play interim content can be used. The terminal device 20 may be a terminal that is rented to a visitor at a museum or the like and reproduces the commentary content for the exhibited work.

FIG. 2 is a block diagram showing a configuration example of the emotion analysis system. The terminal device 20 includes a control unit 21, a storage unit 22, a communication unit 23, a display unit 24, an input unit 25, a camera 26, and the like, and these units are mutually connected via a bus. The control unit 21 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit) or a GPU (Graphics Processing Unit). The control unit 21 appropriately executes the control program 22P stored in the storage unit 22 to perform various types of information processing, control processing, and the like that the terminal device 20 should perform.

The storage unit 22 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 22 stores in advance a control program 22P executed by the control unit 21 and various data necessary for executing the control program 22P. The storage unit 22 also temporarily stores data and the like generated when the control unit 21 executes the control program 22P. Further, the storage unit 22 stores a content DB (database) 22a and a heart rate variability DB 22b described later. The content DB 22a and the heart rate variability DB 22b may be stored in an external storage device connected to the terminal device 20, or may be stored in a storage device communicable with the terminal device 20 via the network N.

The communication unit 23 is an interface for connecting to the network N by wireless communication, for example, and transmits/receives information to/from an external device via the network N. The communication unit 23 may be connected to the network N by wire communication. The display unit 24 is a liquid crystal display, an organic EL display, or the like, and displays various kinds of information according to an instruction from the control unit 21. The input unit 25 receives an operation input by the user and sends a control signal corresponding to the operation content to the control unit 21. The display unit 24 and the input unit 25 may be a touch panel integrally configured.

The camera 26 is an image pickup device having a lens, an image pickup element, and the like, and photoelectrically converts light incident through the lens by an image pickup element corresponding to each color of R (red), G (green), and B (blue). Then, the RGB color image data is acquired. The camera 26 captures images according to an instruction from the control unit 21, and sequentially sends the acquired image data (captured image) to the storage unit 22 for storage. The camera 26 and the display unit 24 are provided in the terminal device 20 so that the shooting direction of the camera 26 is on the display screen side of the display unit 24 as shown in FIG. The face of the user to be visually recognized is arranged so that the camera 26 can photograph it.

In the terminal device 20, the control program and data stored in advance in the storage unit 22 may be downloaded from the external device via the network N via the communication unit 23 by the control unit 21 and stored in the storage unit 22. When the terminal device 20 has a reading unit that reads information stored in the portable storage medium, the control unit 21 stores the control program and data stored in advance in the storage unit 22 via the reading unit. It may be read from and stored in the storage unit 22.

The server 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, and the like, and these units are mutually connected via a bus. The control unit 11 includes one or more processors such as a CPU, MPU or GPU. By appropriately executing the control program 12P stored in the storage unit 12, the control unit 11 causes the server 10 to perform various types of information processing, control processing, and the like that the information processing apparatus of the present disclosure should perform.

The storage unit 12 includes a RAM, a flash memory, a hard disk, an SSD and the like. The storage unit 12 stores in advance the control program 12P executed by the control unit 11 and various data necessary for executing the control program 12P. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the control program 12P. Further, the storage unit 12 stores a content DB 12a, a heart rate variability DB 12b, and an emotion DB 12c, which will be described later. The content DB 12a, the heart rate variability DB 12b, and the emotion DB 12c may be stored in an external storage device connected to the server 10 or may be stored in a storage device that can communicate with the server 10 via the network N.

The communication unit 13 is an interface for connecting to the network N by wire communication or wireless communication, and transmits/receives information to/from an external device via the network N. The input unit 14 includes a mouse, a keyboard, and the like, receives an operation input by the user, and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various kinds of information according to an instruction from the control unit 11.

The reading unit 16 reads information stored in a portable storage medium 1a including a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, or a USB (Universal Serial Bus) memory. The control program and data stored in advance in the storage unit 12 may be read from the portable storage medium 1 a by the control unit 11 via the reading unit 16 and stored in the storage unit 12. The control program and data stored in advance in the storage unit 12 may be downloaded from the external device via the network N via the communication unit 13 by the control unit 11 and stored in the storage unit 12. Further, the control unit 11 may read the control program and data from the semiconductor memory 1b.

FIG. 3 is a schematic diagram showing a configuration example of the DBs 12a to 12c stored in the server 10 and the DBs 22a to 22b stored in the terminal device 20. The content DB 12a stored in the server 10 and the content DB 22a stored in the terminal device 20 have the same configuration, and FIG. 3A shows the content DBs 12a and 22a. The heart rate variability DB 12b stored in the server 10 and the heart rate variability DB 22b stored in the terminal device 20 have the same configuration, and FIG. 3B shows the heart rate variability DBs 12b and 22b. FIG. 3C shows the emotion DB 12c.

The content DBs 12a and 22a store content reproduced by the terminal device 20. The content DBs 12a and 22a shown in FIG. 3A include a content ID sequence, a content data sequence, and the like. The content ID column stores identification information assigned in advance to each content. The content data string stores reproduction data of each content. The content may be moving image content or still image content, and the reproduction data is display data for displaying an image related to the content. The reproduction data may include audio output data for outputting audio related to the content. The content reproduction data may be stored in predetermined areas of the storage units 12 and 22 or an external storage device in addition to being stored in the content DBs 12a and 22a. In this case, the content data string is used for content reproduction. Information for reading data (for example, a file name indicating a storage location of data) is stored. The storage content of the content DB 12a of the server 10 is added, changed, or deleted by the control unit 11 each time an instruction for addition, change, or deletion is acquired via the communication unit 13 or the input unit 14. The storage content of the content DB 22a of the terminal device 20 is updated by the control unit 21 when, for example, update data is received from the server 10 via the network N. The storage contents of the content DBs 12a and 22a are not limited to the example shown in FIG. 3A, and for example, the content name given to the content, the reproduction time of the content, etc. may be stored. Further, the content DBs 12a and 22a may have different configurations, and for example, the reproduction order of each content may be registered in the content DB 22a of the terminal device 20.

The heart rate variability DBs 12b and 22b store information of viewers who have viewed the content using the terminal device 20. The heart rate variability DBs 12b and 22b shown in FIG. 3B include a content ID sequence, a viewing ID sequence, an age sequence, a gender sequence, a heart rate variability data sequence, and the like. The content ID column stores identification information assigned in advance to each content. The viewing ID column stores identification information assigned to each viewing process (reproduction process) for each content. The viewing ID is assigned to each viewing process (viewer) of each content, for example. The age column and the sex column each store the age and sex of the viewer who has performed the viewing process. The heart rate variability data string stores data indicating the heart rate variability of the viewer during the content viewing process. The data indicating the heart rate variability may be stored in the heart rate variability DBs 12b and 22b, or may be stored in a predetermined area of the storage units 12 and 22 or an external storage device. In this case, the heart rate variability data string is the heart rate variability. The information for reading the data indicating (for example, the file name indicating the storage location of the data) is stored.

The content ID and the viewing ID stored in the heart rate variability DB 22b of the terminal device 20 are stored by the control unit 21 when the control unit 21 starts the content reproduction process. The age, sex, and heart rate variability data stored in the heart rate variability DB 22b of the terminal device 20 is stored by the control unit 21 each time the control unit 21 detects the content reproduction process. The terminal device 20 collectively transmits the data stored in the heart rate variability DB 22b to the server 10 for each content or each viewing process. The storage content of the heart rate variability DB 12b of the server 10 is stored by the control unit 11 each time the control unit 11 acquires each data from the terminal device 20 via the network N, for example. The stored contents of the heartbeat variability DBs 12b and 22b are not limited to the example shown in FIG. 3B, and the heartbeat variability DBs 12b and 22b may have different configurations. For example, the identification information of the terminal device 20 that is the transmission source of the data may be registered in the heart rate variability DB 12b of the server 10. It is preferable that the heart rate variability DBs 12b and 22b store data in a format that cannot identify an individual viewer who has viewed the content.

The emotion DB 12c stores information regarding the emotion of the viewer who has viewed the content using the terminal device 20. The emotion DB 12c illustrated in FIG. 3C includes a content ID column, a viewing ID column, an age column, a gender column, an emotion data column, and the like. The content ID column stores identification information assigned in advance to each content, the viewing ID column stores identification information assigned to each viewing process (reproduction process), and the age column and the gender column respectively The age and sex of the viewer who has performed the viewing process are stored. The emotion data sequence stores data indicating the emotion of the viewer while viewing the content, and specifically includes a pleasure sequence, an anger sequence, a sadness sequence, a surprise sequence, a dislike sequence, a relaxation sequence, and the like. The pleasure row, anger row, sadness row, surprise row, disgust row, and relax row each indicate how the viewer may have felt about their feelings of joy, anger, sadness, surprise, disgust, or relaxation while viewing content. Memorize the numerical value. The numerical value of each emotion is stored in time series in association with, for example, the elapsed time (reproduction time) from the start of reproduction of the content, and indicates a change in the emotion of the viewer while viewing the content. The numerical value indicating each emotion is calculated by analyzing the heartbeat variability data of the viewer in the viewing process of each content stored in the heartbeat variability DB 12b. Each data other than the emotion data stored in the emotion DB 12c is stored by the control unit 11 when the control unit 11 starts the viewer emotion analysis process based on the heart rate variability data stored in the heart rate variability DB 12b. It The emotion data stored in the emotion DB 12c is stored by the control unit 11 each time the control unit 11 obtains the numerical value of each emotion by the emotion analysis process. The storage content of the emotion DB 12c is not limited to the example illustrated in FIG. 3C, and for example, the identification information of the terminal device 20 that has performed each viewing process (the terminal device 20 that has collected heartbeat variability data) may be registered. Further, each emotion stored in the emotion DB 12c is not limited to the example shown in FIG. 3C.

Next, a function realized by the control unit 21 of the terminal device 20 executing the control program 22P will be described. FIG. 4 is a block diagram showing functions realized by the control unit 21 of the terminal device 20, and FIG. 5 is an explanatory diagram of processing by the terminal device 20. When the control unit 21 of the terminal device 20 executes the control program 22P stored in the storage unit 22, the content reproduction unit 201, the imaging unit 202, the attribute determination unit 203, the hemoglobin image generation unit 204, and the heartbeat data generation unit 205. The respective functions of the heart rate variability data generation unit 206 and the output unit 207 are realized. In the present embodiment, each of these functions is realized by the control unit 21 executing the control program 22P, but a part of these functions may be realized by a dedicated hardware circuit.

It is assumed that the storage unit 22 of the terminal device 20 stores a content DB 22a as shown in FIG. 3A and a heartbeat fluctuation DB 22b in which only the content ID is stored in the configuration shown in FIG. 3B. The heart rate variability DB 22b may be in a state in which no content ID is stored.

The content reproduction unit 201 reproduces the content based on the reproduction data of the content stored in the content DB 22a (displays the image related to the content on the display unit 24). When the reproduction data includes audio output data for outputting the audio related to the content, the content reproduction unit 201 outputs the audio related to the content from the speaker (not shown) based on the audio output data. To do. The content reproduction unit 201 may reproduce the respective contents in the order stored in the content DB 22a. If the reproduction order is registered in the content DB 22a, the contents reproduction unit 201 reproduces the respective contents in the registered reproduction order. Good. When the content reproduction unit 201 starts the content reproduction process, the content reproduction unit 201 stores the viewing ID in the heart rate variability DB 22b in association with the content ID of this content. Note that the viewing ID may be any information as long as it is an ID that does not overlap between the viewing processes of each content.

When the content reproduction unit 201 starts the reproduction processing of the content, the imaging unit (acquisition unit) 202 starts shooting by the camera 26, and the face image of the face of the viewer who views the content displayed on the display unit 24 ( (Color face image data) is acquired. The image capturing unit 202 acquires, for example, 30 face images (30 frames) per second, and sequentially stores the acquired face images in the storage unit 22, for example.

The attribute determining unit 203 determines the attribute of the person who is the subject of the face image based on the face image (face image data) acquired by the image capturing unit 202. The attribute determination unit 203 determines the attribute of the subject in the face image using, for example, the template matching technique. Specifically, for each attribute of a person including age (age group) and sex, a template based on an image of a general face of a person having each attribute or each organ of the face is stored in the storage unit 22 in advance, The attribute determining unit 203 detects whether or not the face image has a region that matches any of the templates, and when a region that matches the template is detected, the attribute associated with the matching template is set as the object of the subject here. Judge as an attribute. When determining the attribute of the subject, the attribute determining unit 203 associates the determined attribute (age and gender) with the content ID of the content being reproduced and the viewing ID of the reproduction process (viewing process) being executed. It is stored in the fluctuation DB 22b.

The hemoglobin image generation unit (extraction unit) 204 performs pigment component separation processing on the face image (face image data) acquired by the imaging unit 202, extracts the pigment component corresponding to hemoglobin, and extracts the hemoglobin image (of the hemoglobin component). Face image data) is generated. Note that the hemoglobin image generation unit 204 may detect the face area of the subject from the face image acquired by the imaging unit 202, perform pigment component separation only on the detected face area, and generate the hemoglobin image. In this case, the hemoglobin image generation unit 204 may set all the pixel values of each pixel in the area (background area) other than the face area to 0, for example.

The heartbeat data generation unit 205 generates heartbeat data (pulse wave data) indicating the heartbeat of the subject person based on the hemoglobin image generated by the hemoglobin image generation unit 204. Specifically, the heartbeat data generation unit 205 calculates the average value (average pixel value) of the pixel values (component amounts of the hemoglobin component) of each pixel in the face area of the subject in the sequentially generated hemoglobin images, The time change of the average pixel value in each hemoglobin image is generated as heartbeat data. Since the component amount of the hemoglobin component represents the blood flow amount, the heartbeat can be estimated based on the component amount of the hemoglobin component. FIG. 5A shows an example of heartbeat data. In the heartbeat data shown in FIG. 5A, the horizontal axis represents the elapsed time from the start of content reproduction, and the vertical axis represents the average pixel value in the hemoglobin image. The heartbeat data generation unit 205 calculates heartbeat data by calculating an average pixel value of pixels included in a part of the face area instead of the average pixel value of all pixels included in the face area of the subject in the hemoglobin image. Good. For example, the control unit 21 detects organs such as the eyes, nose, and mouth of the human subject, based on the face image acquired by the imaging unit 202, and specifies a predetermined area such as the forehead, cheek, and mouth area. .. Then, the heartbeat data generation unit 205 may generate the heartbeat data by calculating an average pixel value of pixels included in a predetermined region such as the specified forehead, cheek, and mouth area in the hemoglobin image. As described above, by using the average pixel value in the predetermined area instead of the average pixel value in the entire face area, it is possible to suppress the influence of the imaging conditions and generate accurate heartbeat data.

The heartbeat data generation unit 205 performs a filtering process on the heartbeat data as shown in FIG. 5A (temporal change of the average pixel value in the hemoglobin image) to generate heartbeat data as shown in FIG. 5B. In the heartbeat data shown in FIG. 5B, the horizontal axis represents the elapsed time from the start of content reproduction, and the vertical axis represents the average pixel value after filtering. The heartbeat data generation unit 205 performs filtering processing such as slope removal processing and bandpass filtering processing, but is not limited to these processing. The heartbeat data generation unit 205 may perform various processes to obtain heartbeat data such that the heartbeat fluctuation data generation unit 206 in the subsequent stage can generate highly accurate heartbeat fluctuation data.

The heartbeat fluctuation data generation unit 206 performs frequency conversion processing on the heartbeat data generated by the heartbeat data generation unit 205, and generates heartbeat fluctuation data indicating the heartbeat fluctuation of the person who is the subject. The heartbeat variability data generation unit 206 generates heartbeat variability spectrogram data in the frequency domain by, for example, performing Fourier power spectrum conversion on the time-series heartbeat data (temporal change of the average pixel value in the hemoglobin image). The heartbeat variability spectrogram data is, for example, data indicating a temporal change in the heartbeat variability frequency, and FIG. 5C shows an example of the heartbeat variability spectrogram data. In the heart rate variability data shown in FIG. 5C, the horizontal axis represents the elapsed time from the start of content reproduction, and the vertical axis represents the fluctuation frequency. In the Fourier power spectrum conversion, the maximum power spectrum at 0.75 Hz to 3 Hz (heart rate of 45 bpm to 180 bpm) is determined as a pulse. The heart rate variability data generation unit 206 stores the generated heart rate variability data (heart rate variability spectrogram data) in the heart rate variability DB 22b in association with the content ID of the content being reproduced and the viewing ID of the viewing process being executed. ..

The hemoglobin image generation unit 204, the heartbeat data generation unit 205, and the heartbeat fluctuation data generation unit 206 each perform the above-described processing on the acquired face image data every time the imaging unit 202 acquires the face image data of the subject (viewer). I do. Therefore, when the content reproduction unit 201 performs the reproduction process of one content once, the heart rate variability data corresponding to the reproduced content is generated and stored in the heart rate variability DB 22b.

The output unit 207 transmits the heartbeat variability data corresponding to each content accumulated in the heartbeat variability DB 22b from the communication unit 23 to the server 10. Specifically, the output unit 207, for each reproduction process of each content, the content ID of the reproduced content, the viewing ID of the executed reproduction process (viewing process), the viewer's age, sex, and heartbeat. The fluctuation data is read from the heart rate fluctuation DB 22b, and is collectively transmitted to the server 10. Note that, for example, when the output request of the heart rate variability data is received from the server 10, the output unit 207 transmits the heart rate variability data of each content accumulated in the heart rate variability DB 22b up to that time to the server 10. The server 10 receives the content ID, the viewing ID, the age, the sex, and the heart rate variability data transmitted by the output unit 207, and stores the data in association with each other in the heart rate variability DB 12b. Thereby, the heartbeat variation data of the viewer acquired by the terminal device 20 is accumulated in the server 10. The output unit 207 may transmit the heartbeat fluctuation data to the server 10 together with the information for specifying the content (content information) and the information about the attribute of the viewer.

In the terminal device 20 of the present embodiment, the image capturing unit 202 sequentially stores the acquired face image data in the storage unit 22, and the hemoglobin image generation unit 204 reads the face image data stored in the storage unit 22 to obtain the hemoglobin image. Is generated. Since the face image data for which the hemoglobin image is generated is not necessary in the subsequent processing, the control unit (erasing unit) 21 deletes the face image data from the storage unit 22 after the hemoglobin image generating unit 204 generates the hemoglobin image ( It may be deleted). Note that the control unit 21 may have a configuration in which, for example, after the output unit 207 transmits the heartbeat fluctuation data to the server 10, the face image data used when generating the heartbeat fluctuation data is deleted from the storage unit 22. In this way, by deleting the face image data that can identify the individual viewer from the storage unit 22, it is possible to prevent the personal information from remaining in the terminal device 20.

Next, a function realized by the control unit 11 of the server 10 executing the control program 12P will be described. FIG. 6 is a block diagram showing functions realized by the control unit 11 of the server 10, and FIG. 7 is a schematic diagram showing an example of a screen. When the control program 11P stored in the storage unit 12 is executed, the control unit 11 of the server 10 acquires the heartbeat variability data acquisition unit 101, the heartbeat variability data storage unit 102, the emotion analysis unit 103, the report generation unit 104, the output unit. Each function of 105 is realized. A part of each of these functions may be realized by a dedicated hardware circuit.

The heartbeat variability data acquisition unit (data acquisition unit) 101 acquires the heartbeat variability data transmitted by the terminal device 20 (output unit 207) at the communication unit 13. Specifically, the heart rate variability data acquisition unit 101, for each reproduction process of each content from the terminal device 20, the content ID of the reproduced content, the viewing ID of the executed reproduction process, the age of the viewer, Collect gender and heart rate variability data together. The heartbeat variability data storage unit 102 stores the content ID, the viewing ID, the age, the sex, and the heartbeat variability data acquired by the heartbeat variability data acquisition unit 101 in the heartbeat variability DB 12b in association with each other. The heartbeat variability data storage unit 102 stores the acquired heartbeat variability data in the heartbeat variability DB 12b each time the heartbeat variability data acquisition unit 101 acquires heartbeat variability data from any of the terminal devices 20. Thereby, the heartbeat variation data of the viewer acquired (collected) during the reproduction of the content in each of the plurality of terminal devices 20 can be accumulated in the server 10.

The emotion analysis unit 103 analyzes the emotion of each viewer based on the heartbeat variation data stored in the heartbeat variation DB 12b, and generates emotion data indicating the emotion of each viewer. For example, the emotion analysis unit 103 obtains (calculates) a numerical value (score, analysis value) indicating the degree of each emotion (emotion parameter) of joy, anger, sadness, surprise, disgust, and relaxation, for example, with a maximum of 10 points. To do. The higher the degree of each emotion, the larger the value. The emotion analysis unit 103 determines the numerical value of each corresponding emotion in the heart rate variability data based on the variation frequency at a predetermined time (for example, 1/30 second) from the start of reproduction of the content, and each emotion during viewing of the content is determined. The emotion data indicating the time change of is generated. Specifically, for example, a numerical value indicating the degree of each emotion is set in advance for each frequency band in the fluctuating frequency, and the emotion analysis unit 103 uses the heartbeat variability data to change the fluctuating frequency for each predetermined time from the start of playing the content. The numerical value of each emotion set in the frequency band corresponding to the value of is determined as the numerical value indicating the degree of emotion at this point. For example, for a feeling of relaxation, a numerical value (for example, a value close to 0) indicating that the degree of relaxation is low is set in the frequency band of 0.05 Hz to 0.15 Hz, and the frequency band of 0.15 Hz to 0.4 Hz, A numerical value (for example, a value close to 10) indicating that the degree of relaxation is high may be set. A numerical value that increases or decreases in stages may be set in each frequency band. Further, even if the frequency band corresponding to the numerical value indicating the degree of each emotion is set for each emotion, the emotion analysis unit 103 similarly similarly determines each emotion corresponding to the fluctuating frequency for each predetermined time from the start of reproducing the content. A numerical value indicating the degree can be specified.

When performing the emotion analysis process, the emotion analysis unit 103 reads the content ID, the viewing ID, the age, the sex, and the heart rate variability data from the heart rate variability DB 12b, and associates the read content ID, the viewing ID, the age, and the gender with the emotion. It is stored in the DB 12c. Then, the emotion analysis unit 103 analyzes the emotion for each predetermined time from the start of the reproduction of the content based on the heart rate variability data, and the emotion data of each emotion obtained as a result of the analysis is stored in the previously stored content ID, the viewing It is stored in the emotion DB 12c in association with the ID, age, and sex. As a result, emotion data indicating temporal changes in each emotion of the viewer during reproduction (viewing) of the content is generated and stored in the emotion DB 12c. The emotion analysis unit 103 generates heartbeat data (pulse wave data) from the heartbeat fluctuation data by inverse conversion processing, analyzes the viewer's emotion based on the heartbeat data, and generates emotion data indicating the viewer's emotion. You may. For example, the emotion analysis unit 103 may detect the peak value of the heart rate, the time between peaks, and the like in the heartbeat data, and determine the degree of each emotion based on the detection results. Also in this case, the emotion analysis unit 103 sets the numerical value indicating the degree of each emotion to the peak value of the heart rate in the heartbeat data, the time between the peaks, and the like in advance, so that the emotion analysis unit 103 can reproduce the content at predetermined time intervals. It is possible to specify the numerical value of each emotion corresponding to the peak value of the heart rate and the time between peaks in. In addition, by setting the peak value of the general heart rate, the time between peaks, etc. in advance in association with the numerical value indicating the degree of each emotion for each emotion, the emotion analysis unit 103 starts the reproduction of the content. From this, it is possible to specify a numerical value indicating the degree of each emotion corresponding to the peak value of the heart rate at each predetermined time, the time between peaks, and the like.

The report generation unit 104 generates a viewer emotion report for each content based on the emotion data accumulated in the emotion DB 12c. For example, the report generation unit 104 reads out the content ID and content data (reproduction data) of the content for which an emotional report should be generated from the content DB 12a, and reads out all emotional data for this content from the emotional DB 12c. The report generation unit 104 generates display data for displaying an emotion report as shown in FIGS. 7A and 7B, based on each information read from the DBs 12a and 12c. 7A and 7B each show an example of a display screen based on an emotion report. The screen shown in FIG. 7A displays the content ID, content name, and playback time of the content to be analyzed. The content name and the reproduction time are stored in, for example, the content DB 12a, and the report generation unit 104 reads them from the content DB 12a. The screen shown in FIG. 7A has a content area R1, a button area R2, and a report area R3, and the content data read from the content DB 12a is reproduced (displayed) in the content area R1. In the button area R2, buttons for accepting operations such as a reproduction instruction, a stop instruction, a fast forward instruction, and a fast reverse instruction for the content reproduced in the content area R1 are displayed. Further, in the button area R2, an indicator showing the elapsed time from the start of reproduction of the content in the image (scene) being displayed in the content area R1 is displayed. Below the indicator, from the reproduction start in the scene being displayed. The elapsed time of is displayed. It should be noted that the indicator can receive an arbitrary time point between the reproduction start and the reproduction end of the content as the time point when the reproduction is started (restarted) in the content region R1, and when the indicator receives the arbitrary time point, The content is reproduced in the content area R1 from the time when the indicator is received.

In the report area R3, a report of each emotion regarding the content reproduced in the content area R1 is displayed. When the emotion report shown in FIG. 7A is generated, the report generation unit 104 calculates the average value of the numerical values of each emotion for each predetermined time from the start of reproducing the content, based on all the emotion data for the content read from the emotion DB 12c. Then, with respect to each emotion, an emotion report showing the time change (analysis value) of the average value (average degree) of the viewers who viewed the content is generated. The emotion report shown in FIG. 7A shows the degree of each emotion by an emotion gauge with a vertically long rectangle, the lower end of the emotion gauge shows the lowest value, the upper end shows the highest value, and the color region from the lower end (FIG. 7A Then, the degree of emotion is shown by the size of the hatching area). In the screen shown in FIG. 7A, for example, the reproduction timing of the content reproduced in the content area R1 (elapsed time from the start of reproduction) and the appearance timing of each emotion displayed in the report area R3 (the viewer holds each emotion) (Timing) is displayed in synchronization with. That is, the analysis value for each emotion and the content are displayed in synchronization with the elapsed time from the start of reproducing the content. Therefore, it is possible to easily grasp the degree of each emotion at each content reproduction timing. That is, the viewer's emotions and changes in emotions in each scene of the content can be easily grasped. The report generation unit 104 generates the emotion report of each emotion based on the average value of the emotion data (numerical value of each emotion) of all the viewers read from the emotion DB 12c, and also the attribute of the viewer for each emotion. An emotion report for each (age and sex) may be generated. In this case, the report generation unit 104 reads the emotion data of the viewer having a predetermined attribute from the emotion DB 12c and generates an emotion report of each emotion based on the average value of the read emotion data (numerical value of each emotion).

The screen shown in FIG. 7B displays the content ID, the content name, and the reproduction time of the content to be analyzed, and the emotion report for each attribute (age and sex) for each emotion with respect to the content. When the emotion report shown in FIG. 7B is generated, the report generation unit 104 reads emotion data for the content accumulated in the emotion DB 12c for each attribute. Attributes include, for example, T layer indicating males and females in their teens, M1 layer indicating males aged 20 to 34, M2 layer indicating males aged 35 to 49, M3 layer indicating males aged 50 and over, 20 years old aged It can be classified into an F1 layer indicating a female of 34 years old, an F2 layer indicating a female of 35 to 49 years old, and an F3 layer indicating a female of 50 years old or older, but not limited to these divisions. The report generation unit 104 reads, for each attribute, emotion data of the viewer of each attribute from the emotion DB 12c. The report generation unit 104 extracts the maximum value of each emotion for each attribute from the emotion data read for each attribute. For each emotion, the report generation unit 104 generates a graph (emotion report) that displays the maximum value extracted for each attribute in association with each attribute. In the emotion report shown in FIG. 7B, for example, the solid line indicates the maximum value for each attribute for the feeling of joy, and the broken line indicates the maximum value for each attribute for the feeling of sadness.

The report generation unit 104 may generate an emotion report indicating the maximum value for each attribute for each emotion, and may generate an emotion report indicating the average value for each attribute for each emotion. In this case, the report generation unit 104 calculates, for example, the average value of the numerical values of the emotions of each viewer based on the emotional data read for each attribute, and further calculates the average value of the numerical values of each emotion of all the viewers. To calculate. Thereby, for each attribute, the average value (average degree) of all viewers in each emotion for the content can be obtained. The report generation unit 104 generates a graph (emotion report) that displays the average value calculated for each attribute for each emotion in association with each attribute. Also in this case, the average value of each emotion for each attribute can be displayed in the graph as shown in FIG. 7B. Further, on the screen shown in FIG. 7B, instead of the emotion report indicating the maximum value or average value of emotions in the entire content, an emotion report indicating the maximum value or average value of emotions in each elapsed time period from the start of reproducing the content is displayed. It may be displayed. Further, as in the case of FIG. 7A, the screen shown in FIG. 7B has a content area in which the content is reproduced, and buttons for accepting operations such as a reproduction instruction, a stop instruction, a fast forward instruction, and a fast rewind instruction for the reproduced content. A displayed button area or the like may be provided.

The report generation unit 104 may generate the emotion report shown in FIG. 7A and the emotion report shown in FIG. 7B separately, or display the emotion report shown in FIG. 7A and the emotion report shown in FIG. 7B on one screen. You may generate the display data for doing. In addition to the emotion report described above, the report generation unit 104 may, for example, display an emotion report indicating the elapsed time from the start of reproduction at the time when the maximum value is obtained for each emotion, or the reproduction at the time when the value becomes a predetermined value or more for each emotion. You may generate the emotion report etc. which show the elapsed time from the start. Further, the report generation unit 104 may generate, for example, an emotion report indicating the total time when a value is equal to or larger than a predetermined value or the total time when the value is smaller than the predetermined value for each emotion. These emotion reports may also be generated for each attribute.

The output unit 105 transmits the emotion report (display data) generated by the report generation unit 104 from the communication unit 13 to a predetermined device. Note that, for example, when the server 10 receives a request to generate an emotional report for a content from an external device, the report generation unit 104 generates an emotional report for the requested content, and the output unit 105 outputs the requested external device. Output emotion report to. In addition, the report generation unit 104 generates an emotion report for each content at a predetermined timing and stores the emotion report in the storage unit 12, and when the server 10 is requested by the external device to send an emotion report for a content, the output unit The 105 may read the requested emotion report from the storage unit 12 and output it to the external device that has made the request. The external device receiving the emotion report output by the output unit 105 displays the screens shown in FIGS. 7A and 7B on the display unit based on the received emotion report. As a result, it is possible to notify the viewer of changes in the emotion of each content.

The processing performed by each device in the emotion analysis system having the above configuration will be described below. FIG. 8 is a flowchart showing a procedure of a process of acquiring heartbeat variability data by the terminal device 20. The following processing is executed by the control unit 21 according to the control program 22P stored in the storage unit 22 of the terminal device 20.

The control unit 21 of the terminal device 20 starts the content reproduction process based on the content reproduction data stored in the content DB 22a (S11). The timing of starting the reproduction of the content is, for example, when a reproduction instruction is input through the input unit 25, when a reproduction instruction is acquired from an external device through the communication unit 23, or the like. The control unit 21 stores in the heart rate variability DB 22b the content ID of the content that has started to be reproduced and the viewing ID issued for this reproduction process (viewing process) in association with each other.

The control unit 21 starts shooting with the camera 26, and acquires the face image of the viewer who views the content being reproduced (S12). Here, the control unit 21 acquires a face image which is one still image. The control unit 21 determines the attributes (age and sex) of the viewer who is the subject in the face image based on the acquired face image (face image data) (S13). The control unit 21 determines the attribute of the subject by, for example, the template matching technique, and stores the determined attribute in the heart rate variability DB 22b in association with the content ID of the content being reproduced and the viewing ID of the reproduction process being executed.

The control unit 21 performs a separation process of extracting a pigment component corresponding to hemoglobin on the face image acquired in step S12 to generate a hemoglobin image (S14), and based on the hemoglobin image, the subject (viewer) Heartbeat data indicating a heartbeat is generated (S15). Here, the control unit 21 calculates the average value (average pixel value) of the pixel values (the component amount of the hemoglobin component) of each pixel in the face area of the subject in the hemoglobin image, and captures this face image as heartbeat data. It is stored in the storage unit 22 in association with the elapsed time (reproduction timing) from the start of reproduction at the time. When the face image is acquired after the reproduction of the content is started, the control unit 21 specifies the elapsed time from the start of the reproduction of the content at the acquisition time.

The control unit 21 determines whether or not the content reproduction processing has been completed by reproducing the content to the end (S16). If it is determined that the content reproduction processing has not been completed (S16: NO), the face is determined in step S12. It is determined whether a predetermined time has elapsed since the image was acquired (S17). For example, when acquiring 30 face images in 1 second, the control unit 21 determines whether 1/30 second has elapsed. When it is determined that the predetermined time has not elapsed (S17: NO), the control unit 21 repeats the processing of steps S16 to S17, and when it is determined that the predetermined time has elapsed (S17: YES), the processing of step S12. Return to. Then, the control unit 21 acquires the face image of the viewer (S12), and repeats the processing of steps S13 to S15 based on the acquired face image. As a result, based on the face image of the viewer that is acquired at predetermined time intervals during the reproduction of the content, heartbeat data indicating the heartbeat of the viewer at predetermined time intervals is generated and stored in the storage unit 22.

When it is determined that the content reproduction processing has ended (S16: YES), the control unit 21 performs filter processing on the heartbeat data (temporal change in average pixel value in the hemoglobin image) generated by the above-described processing (S18). ). Then, the control unit 21 performs a frequency conversion process on the filtered heartbeat data (S19) to generate heartbeat variability data (heartbeat variability spectrogram in the frequency domain), and uses the generated heartbeat variability data as the content ID and the viewing ID. It is stored in the heart rate variability DB 22b in association with (S20). As a result, heartbeat variability data indicating the heartbeat variability of the viewer while viewing the content is generated and stored in the heartbeat variability DB 22b. Note that, when the reproduction processing of different contents is continuously performed, the control unit 21 finishes the reproduction processing of one content and then performs the processing of steps S18 to S20 for this content, while performing the processing of the next content. Then, the processing from step S11 is executed.

FIG. 9 is a flowchart showing a procedure of emotion analysis processing by the server 10. The following processing is executed by the control unit 11 according to the control program 12P stored in the storage unit 12 of the server 10. The control unit 11 of the server 10 acquires the heartbeat fluctuation data transmitted by the terminal device 20 by the communication unit 13 (S31), and stores the acquired heartbeat fluctuation data in the heartbeat fluctuation DB 12b (S32). The control unit 11 collectively acquires the content ID, the viewing ID, the age, the sex, and the heartbeat variation data from the terminal device 20 for each one-time reproduction process of each content. The timing at which the heart rate variability data is transmitted from the terminal device 20 to the server 10 is, for example, a predetermined time once a day, a predetermined time once a week, after the expiration of the reproducible period set in the content, or the like. You can Further, the terminal device 20 may transmit the heartbeat variability data to the server 10 when the transmission timing arrives, or may transmit the heartbeat variability data in response to a transmission request from the server 10.

The control unit 11 analyzes the emotion of each viewer for each content based on the heart rate variability data for each content stored in the heart rate variability DB 12b (S33). For example, the control unit 11 determines the degree of each emotional parameter such as joy, anger, sadness, surprise, disgust, and relaxation according to the value of each variation frequency in the heart rate variability data (heart rate variability spectrogram). The numerical value (analysis value) indicating is determined. For each emotion, the control unit 11 determines a numerical value at a sequential timing (for example, every 1/30 seconds) from the start of reproduction of the content, so that each emotion of the viewer from the start of reproduction of the content to the end of reproduction thereof. The emotion data indicating the time change of is generated. The control unit 11 stores the generated emotion data of each emotion in the emotion DB 12c in association with the content ID read from the heart rate variability DB 12b, the viewing ID, the age, and the sex (S34).

The control unit 11 generates an emotion report that comprehensively evaluates the emotions of a plurality of viewers for each content based on the emotion data stored in the emotion DB 12c for each content (S35). The control unit 11 reads all emotion data for the content for which an emotion report should be generated from the emotion DB 12c, and generates an emotion report as shown in FIGS. 7A and 7B based on the read emotion data. It should be noted that the emotion report generated by the control unit 11 is, for example, the time change of the average value of a plurality of viewers, the average value for each predetermined time period or each attribute, for each emotion between the reproduction start and the reproduction end of the content. The time change, the maximum value or the average value for a plurality of viewers, the maximum value or the average value for each predetermined time period or each attribute, and the like can be notified of a report that can evaluate the viewer's feelings about the content. The control unit 11 stores the generated emotion report in the storage unit 12 in association with, for example, the content ID, and, for example, when a request for an emotion report for predetermined content is received from an external device, the emotion report requested. To the requesting external device.

In the present embodiment, it is possible to acquire the change in emotion (time change) of the viewer during viewing from the face image of the viewer who views the content. Therefore, the emotion of the viewer at each time point from the start of reproducing the content can be grasped based on the temporal change of the emotion of the viewer. Further, in the present embodiment, the terminal device 20 transmits the heartbeat fluctuation data to the server 10, so that only the information (heartbeat fluctuation data) that cannot identify an individual viewer is transmitted and received between the terminal device 20 and the server 10. That is, the leakage of personal information can be prevented by not transmitting/receiving data that can identify the individual viewer, such as a face image. The heartbeat data before frequency conversion may be transmitted/received, and in this case, the server 10 receiving the heartbeat data performs a process of generating heartbeat variation data from the heartbeat data. Further, in the terminal device 20, it is possible to prevent the personal information from remaining in the terminal device 20 by deleting the data such as the face image that can identify the individual viewer from the storage unit 22 after it is no longer needed. When not only the face image data but also the hemoglobin image data is once stored in the storage unit 22 after being generated, it may be deleted from the storage unit 22 after it is no longer needed. Further, the terminal device 20 may be configured to directly transmit the collected (accumulated) heartbeat variability data to the server 10. In this case, the communication between the terminal device 20 and the server 10 may be wired communication or wireless communication. It may be communication.

(Embodiment 2)
A sentiment analysis system applied to a notification system in which a terminal device is lent to a user who ordered food at a restaurant in a food court provided in a shopping mall or the like and the terminal device notifies the user that the ordered food is completed will be described. .. FIG. 10 is a block diagram showing a configuration example of the emotion analysis system according to the second exemplary embodiment. The emotion analysis system of the present embodiment includes a notification device 30 in addition to the server 10 and the terminal device 20, and the notification device 30 is configured to perform wireless communication with each of the plurality of terminal devices 20. In this embodiment, the notification device 30 and each terminal device 20 are configured to directly communicate with each other, but may be configured to communicate via the network N. In the present embodiment, for example, one notification device 30 and a plurality of terminal devices 20 are prepared in each store, and the notification device 30 provided in each store is between the plurality of terminal devices 20 prepared for each store. Communicate with. The server 10 of this embodiment has the same configuration as that of the first embodiment and performs the same processing.

The terminal device 20 has a wireless communication unit 27 and a notification unit 28 in addition to the same configuration as the first embodiment. The wireless communication unit 27 is an interface for communicating with an external device by, for example, specific low power wireless communication, and communicates with the notification device 30 in the present embodiment. The wireless communication unit 27 may be configured to perform wireless communication such as Bluetooth (registered trademark) or Wi-Fi. The notification unit 28 includes, for example, a vibrator, a lamp, a buzzer, etc., and in accordance with an instruction from the control unit 21, a user rented out the terminal device 20 by vibrating the vibrator, lighting or blinking the lamp, or ringing the buzzer. Notify the specified situation (the ordered food is completed). The notification unit 28 may be configured to notify by outputting a predetermined message by voice from a speaker. Further, instead of the notification unit 28, the display unit 24 may display a predetermined message to notify.

The notification device 30 includes a control unit 31, a storage unit 32, an input unit 33, a display unit 34, a wireless communication unit 35, and the like, and these units are mutually connected via a bus. Each unit of the notification device 30 has the same configuration as each unit of the terminal device 20, and detailed description of the same configuration will be omitted. The wireless communication unit 35 of the notification device 30 communicates with the wireless communication unit 27 of the terminal device 20. A terminal DB 32a is stored in the storage unit 32 of the notification device 30. In the terminal DB 32a, identification information (for example, terminal ID) for identifying each of the plurality of terminal devices 20 with which the notification device 30 communicates, and destination information for performing wireless communication with each terminal device 20. And are stored in association with each other. In addition to the above-described configuration, the notification device 30 may be provided with an interface or the like for connecting to the network N.

FIG. 11 is a flowchart showing a procedure of a process of acquiring heartbeat fluctuation data by the terminal device 20 of the second embodiment. In the process shown in FIG. 11, step S51 is added in place of step S16 in the process shown in FIG. 8, steps S52 and S53 are added before step S18, and steps S41 and S42 are added as processes performed by the notification device 30. It was done. Description of steps similar to those in FIG. 8 is omitted.

In the present embodiment, for example, an employee of a restaurant lends one terminal device 20 to a user who has ordered food and completed payment of a charge. Then, when the food is completed, the employee inputs the terminal ID of the terminal device 20 into the input unit 33 of the notification device 30 so that the terminal device 20 lent to the user who ordered the food can perform the notification process. To do. The control unit 31 of the notification device 30 determines whether or not the terminal ID has been received via the input unit 33 (S41), and when determining that the terminal ID has not been received (S41: NO), waits until the reception. When determining that the terminal ID has been received (S41: YES), the control unit 31 reads the destination information corresponding to the received terminal ID from the terminal DB 32a, and transmits a notification signal by the wireless communication unit 35 based on the read destination information. (S42). Each time the control unit 31 of the notification device 30 receives the terminal ID, the control unit 31 transmits a notification signal to the terminal device 20 having the received terminal ID.

The control unit 21 of the terminal device 20 determines whether or not a notification signal has been received from the notification device 30 while performing the processing of steps S11 to S15 as in the first embodiment (S51), and has not received the notification signal. If it is determined (S51: NO), the process proceeds to step S17. By such processing, the control unit 21 generates the heartbeat data of the viewer based on the face image of the viewer who is viewing the content, and stores the heartbeat data in the storage unit 22. When it is determined that the notification signal is received from the notification device 30 (S51: YES), the control unit 21 performs the notification process by the notification unit 28 (S52). The notification unit 28 starts, for example, vibration by a vibrator, lighting or blinking of a lamp, and ringing of a buzzer to notify the user that the ordered food is completed. Then, the control unit 21 ends the content reproduction processing (S53), and performs the processing of steps S18 to S20 on the heartbeat data (temporal change of the average pixel value in the hemoglobin image) generated by the above-described processing.

By the above-described processing, the user can view and enjoy the content in the waiting time until the ordered dish is completed, and can collect the heartbeat variability data indicating the heartbeat variability while the user is viewing the content. In the above-described process, when the control unit 21 performs the notification process of notifying that the dish is completed, the control unit 21 ends the content reproduction process after receiving an instruction to end the notification process via the input unit 25, for example. Good. For example, when an operation such as a vibration stop instruction by a vibrator, a lamp off instruction, or a buzzer mute instruction is performed via the input unit 25, the content reproduction process may be ended. Further, the content reproduction process may be terminated when a content reproduction end instruction is issued via the input unit 25 separately from the notification process end instruction. In this case, the user who is viewing the content can view the content up to the desired scene, and it can be expected that the viewing time of the content becomes long.

In the present embodiment, the server 10 performs the same process as that shown in FIG. 9, and can generate an emotion report for each content based on the heartbeat variability data of a large number of viewers collected from a plurality of terminal devices 20. In this embodiment, the same effect as that of the first embodiment can be obtained. That is, it is possible to acquire the change in emotion (time change) of the viewer during viewing from the face image of the viewer who views the content, and based on the time change in the emotion of the plurality of viewers, the viewer's emotion report to the content is reported. Can be provided. Further, in the present embodiment, for example, a user who has ordered food at a restaurant such as a food court can enjoy the waiting time by viewing the content during the waiting time until the food is completed. Note that the content reproduced by the terminal device 20 of the present embodiment is, for example, a content for notifying information of advertisements, events, campaigns, coupons, etc. of stores in shopping malls provided with food courts, shopping malls and stores. Content and the like for accepting answers to questionnaires and complaints can be used. It should be noted that the terminal device 20 can accept a questionnaire response, a complaint input, and the like via the input unit 25.

The emotion analysis system described above can also be applied to, for example, a system in which a terminal device is lent to a visitor at a museum or the like and a commentary about an exhibited work is reproduced by reproducing the commentary content for the exhibited work on the terminal device. When applied to such a system, not only can the explanation of the exhibited work be performed by the terminal device 20, but by analyzing the emotions of the visitors who watch the explanation contents, the visitors can respond to the exhibited works and the explanation about the exhibited works. It is possible to grasp the presence or absence of interest.

In the emotion analysis system according to the first and second embodiments, the content reproduced by the terminal device 20 may be any content, and an emotion report of the viewer who is viewing the content can be generated. For example, the content is not limited to moving image content, and may be an electronic book that can be page forwarded by operating the input unit 25. In this case, the user's emotion report for each page can be generated by analyzing the user's emotion in association with the page number being read by the user. Further, contents such as a quiz, a game, and a questionnaire for selecting any one of a plurality of options may be used. In this case, the terminal device 20 can accept the selection of the options via the input unit 25, and can analyze the emotion of the user when the options are accepted. Further, when the questionnaire has a plurality of questions, the emotions of the respondents to each question can be analyzed based on the face image data of the respondent (user) who answers each question, and the emotion data for each question can be generated. Further, by using the emotion data of a plurality of respondents, it is possible to generate an emotion report of all the respondents to each question.

Further, for example, employee questionnaires such as employee satisfaction questionnaires, stress checks, and 360 degree evaluation surveys can be used as contents. For example, when the questionnaire includes questions regarding the stress check, the control unit 21 of the terminal device 20 displays each question on the display unit 24, accepts the answer of the respondent to each question through the input unit 25, and displays the face of the respondent. The emotion at the time of answering is analyzed based on the image to generate emotion data. In this case, the terminal device 20 can acquire the content that the respondent intentionally answered to the question and the emotion at the time of answering the question that the respondent unintentionally analyzed from the face image. Therefore, the terminal device 20 can acquire the response content under the consciousness of the respondent and the true meaning of the respondent that can be read from the emotion that the respondent does not intend, and the respondent responds based on these pieces of information. The authenticity of the content can be objectively determined, and the stress check for the respondents can be accurately performed. Further, the terminal device 20 can output the response content under the consciousness of the respondent and the true intention not intended by the respondent to the predetermined device in association with each other. The predetermined device here is, for example, a device for collecting response information of the stress check. In this way, for each question, not only the content of the answer by the respondent, but also the unintended intention (feeling) of the respondent at the time of answering can be collected. , It will be possible to promote work style reform.

In the emotion analysis system according to the first and second embodiments, the process performed by the terminal device 20 may be configured to be performed by a digital signage installed in a store or the like. For example, the digital signage can be configured to display advertisement content related to a store, a product, or the like, and acquire a change in emotion of the user (emotion report) from a face image of the user who is viewing the advertisement content. With such a configuration, the degree of interest of the user with respect to the displayed product or the like can be measured. Therefore, by switching the advertisement content to be displayed in accordance with the degree of interest of the user, it is possible to display the advertisement of the product or the like that the user is interested in, and the product can be efficiently advertised. Further, by displaying, for example, a product name or a package image of the product on the digital signage and the terminal device 20 and acquiring the emotion of the user who views the product, the user's awareness of the product being displayed can be measured. In this case, by accumulating the awareness of a plurality of users, the awareness of the product or the advertising effect of the advertisement can be measured.

In the emotion analysis system of the first and second embodiments, the content played back by the terminal device 20 may be game content in addition to movie content such as movies, dramas, and animations. In this case, a user emotion report for each unit can be generated by analyzing the user's emotion in association with each unit such as the stage, level, course, and scene in the game. In addition, by analyzing the emotion of the user in association with each timing such as the start time of the game, the reproduction time of the tutorial, the occurrence time of each event in the game, the billing time, the end time, etc., the emotion of the user at each timing is analyzed. Can generate reports. By analyzing changes in the emotions of the user during the game in this way, the timing at which the user enjoys the game and the place in the game, and the timing at which the user does not enjoy the game or ends the game and Can determine the location. Therefore, the reason why the user finishes the game can be analyzed, and by using the analysis result, it is possible to produce a game that the user does not get tired of.

(Embodiment 3)
A sentiment analysis system applied to a system in which a terminal device is lent to a visitor at a museum or the like and a commentary content for the exhibited work is reproduced on the terminal device will be described. The emotion analysis system of the present exemplary embodiment can be realized by the same devices as the respective devices of the first exemplary embodiment, so description of the configuration will be omitted.

12 and 13 are flowcharts showing the procedure of the process of acquiring heartbeat variability data by the terminal device 20 of the third embodiment, and FIG. 14 is a schematic diagram showing an example of a screen. In the process shown in FIGS. 12 and 13, steps S61 to S62 are added before step S11 in the process shown in FIG. 8, steps S63 to S65 are added instead of step S16, and step S66 is added after step S20. It was done. Description of steps similar to those in FIG. 8 is omitted.

In the present embodiment, the terminal device 20 is rented, for example, when a person who enters the museum enters. The terminal device 20 stores the commentary content for each of the exhibits exhibited in the museum in the content DB 22a, and displays the content list as shown in FIG. 14A on the display unit 24. The content ID in this embodiment is identification information assigned to the commentary content, that is, identification information assigned to each exhibit. The content list displays thumbnail images of each exhibit in the museum, and a "play commentary" button is provided in association with each thumbnail image to accept a playback instruction of commentary content related to each exhibit. ing. Therefore, when there is an exhibit item for which the commentary is desired to be heard, the visitor operates the "play commentary" button corresponding to the thumbnail image of the exhibit item to instruct the reproduction of the commentary content.

Therefore, in the present embodiment, the control unit 21 of the terminal device 20 displays a content list including thumbnail images of exhibits in the museum (S61), and reproduces any commentary content via the content list. It is determined whether an instruction has been accepted (S62). When it is determined that the reproduction instruction has not been received (S62: NO), the control unit 21 waits until it receives it. When it is determined that the instruction to reproduce any of the commentary contents has been received (S62: YES), the control unit 21 starts the reproduction processing of the commentary content for which the reproduction instruction has been accepted (S11). Specifically, the control unit 21 reads out the reproduction data of the commentary content instructed to be reproduced from the content DB 22a, and performs the reproduction process based on the read reproduction data. FIG. 14B shows an example of a reproduction screen of the explanation content, and the reproduction screen displays the explanation about the exhibit. The commentary content may be moving image data or still image data, and may include audio data. Further, on the reproduction screen shown in FIG. 14B, an input field for inputting impressions about the exhibit, a like button for notifying that the exhibit is liked, and an instruction to end display of the explanation content being displayed And an end button to do so. When the commentary content includes audio data, the control unit 21 also performs a reproduction process of the audio data via a speaker (not shown). When the reproduction process of the commentary content is started, the control unit 21 stores the content ID of the commentary content that has started to be reproduced and the viewing ID issued for this reproduction process in the heart rate variability DB 22b in association with each other.

After that, the control unit 21 performs the same processing as steps S12 to S15 in FIG. 8, and based on the face image of the user (viewer) who views the commentary content being reproduced, the attributes (age and gender) of the user. Is determined and heartbeat data indicating the user's heartbeat is generated. It should be noted that the user views the explanation of the exhibit on the reproduction screen, inputs an impression in the input field via the input unit 25 as necessary, and may like the input unit 25 if he/she likes the exhibit. Operate the button. The control unit 21 determines whether or not the operation of the like button by the user or the input of an impression in the input field has been accepted on the reproduction screen as shown in FIG. 14B (S63). When it is determined that either the operation of the like button or the input of the impression is accepted (S63: YES), the control unit 21 associates the content ID of the commentary content being reproduced with the operation of the like button. Is stored in the storage unit 22 (S64). The control unit 21 may be configured to store information indicating that the like button has been operated or the received impression in the heart rate variability DB 22b in association with the content ID.

When the control unit 21 determines that neither the operation of the like button nor the input of the impression is received (S63: NO), the process of step S64 is skipped, and the process proceeds to step S65. The control unit 21 determines whether or not an instruction to end the reproduction of the commentary content being reproduced is received by the user operating the end button on the reproduction screen as shown in FIG. 14B (S65). If the control unit 21 determines that the reproduction end instruction has not been received (S65: NO), the control unit 21 proceeds to the process of step S17, and if the control unit 21 determines that the reproduction end instruction has been received (S65: YES), the process proceeds to step S18. Move to processing. As a result, heartbeat data indicating the user's heartbeat at each predetermined time is generated and stored in the storage unit 22 based on the user's face image acquired at every predetermined time during the reproduction of the commentary content of each exhibit. Further, the control unit 21 performs the processes of steps S18 to S20 in FIG. 8 to generate the heartbeat data of the user (temporal change of the average pixel value in the hemoglobin image) based on the generated heartbeat data and the heartbeat fluctuation of the user while viewing the commentary content. Heart rate variability data indicating is generated and stored in the heart rate variability DB 22b.

After the processing of step S20, the control unit 21 determines whether or not an instruction to end the reproduction processing of the commentary content has been received, in response to an operation via the input unit 25 (S66). For example, in the content list shown in FIG. 14A or the reproduction screen shown in FIG. 14B, an end button for instructing the end of the operation of the terminal device 20 (end of the comment processing) is provided, and when the end button is operated, It may be configured to receive an instruction to end the reproduction process. Note that such an end button may be operated by the user when terminating the use of the terminal device 20, or may be operated by the museum staff or the like after the terminal device 20 is returned to the museum. When the control unit 21 determines that the end instruction of the reproduction process is not accepted (S66: NO), the control unit 21 returns to the process of step S61, returns to the content list display as illustrated in FIG. 14A, and determines that the end instruction is accepted. If so (S66: YES), the process ends. Through the above-described processing, the user can receive a comment on an exhibit such as an art museum, and can collect heartbeat variability data indicating heartbeat variability while the user is viewing the content.

FIG. 15 is a flowchart showing a procedure of emotion analysis processing by the server 10 of the third embodiment. The process shown in FIG. 15 is obtained by adding steps S71 to S73 instead of step S35 in the process shown in FIG. Description of steps similar to those in FIG. 9 is omitted. In the present embodiment, the control unit 11 of the server 10 performs the same processing as steps S31 to S34 in FIG. As a result, the server 10 receives the heart rate variability data transmitted by the terminal device 20, analyzes the user's feelings regarding each commentary content (exhibition corresponding to the commentary content) based on the received heart rate variability data, and the user The emotion data indicating the temporal change of each emotion is generated. In the present embodiment, the timing at which the terminal device 20 transmits the heartbeat fluctuation data to the server 10 may be the timing at which the user finishes using the terminal device 20. Further, in the present embodiment, the terminal device 20 transmits the heartbeat variability data to the server 10 in association with the identifier (for example, the terminal ID) assigned to the terminal device 20 or the identifier (for example, the user ID) assigned to the user. .. Therefore, the server 10 stores the heartbeat variability data in, for example, the heartbeat variability DB 12b in association with the terminal ID or the user ID, and associates the emotion data of each emotion generated in step S34 with the terminal ID or the user ID in the emotion DB 12c. Remember.

After the processing of step S34, the control unit 11 (location extraction unit) identifies the location where the user has a predetermined emotion based on the emotion data for each commentary content stored in the emotion DB 12c in association with the terminal ID or the user ID. Extract (S71). For example, the control unit 11 identifies commentary content whose score of the emotion parameter of joy is a predetermined value or more based on the emotional data for each commentary content, and extracts the content ID corresponding to the identified commentary content. Then, the control unit 11 (album generation unit) generates album data by using the commentary content corresponding to the extracted content ID or the thumbnail image of the exhibit (S72). As a result, it is possible to generate an album in which the user can collectively browse contents or exhibits that have a predetermined emotion. The control unit 11 stores the generated album data in the storage unit 12 in association with the terminal ID or the user ID (S73), and ends the process.

By the above-described processing, album data can be generated according to the emotion of the user who sees the commentary content and the exhibit, so that the album can be provided in which the user can browse the commentary content or the exhibit that unconsciously has a predetermined emotion. .. It should be noted that the emotions that generate the album data may be preset with any emotional parameters such as joy, anger, sadness, surprise, disgust, and relaxation, and are configured so that they can be changed by the user's selection. It may have been done. The server 10 is also configured to be able to provide album data stored in the storage unit 12 in association with the terminal ID or the user ID to, for example, a user terminal (not shown) owned by the user. For example, when the server 10 receives a request for album data by designating a terminal ID or a user ID from the user terminal, the server 10 may transmit the requested album data to the requesting user terminal. As a result, the user can browse the album on his/her own user terminal, so that not only can he/she appreciate the exhibits at the museum, but he/she can also look back at the exhibits or the commentary contents by the album after the observation at the museum.

When the server 10 transmits the album data to the user terminal, for example, even if the server 10 issues a QR code (registered trademark) corresponding to the terminal ID or the user ID and transmits the issued QR code to the user terminal. Good. In this case, the user terminal can request the album data based on the destination information (for example, URL (Uniform Resource Locator)) obtained by reading the QR code, and can acquire the album data from the server 10. Further, for example, in a configuration in which the server 10 stores album data in association with a user ID, the terminal device 20 may be used to download album data from the server 10 via the network N based on the user ID. .. In this case, on the terminal device 20 rented out in the same museum or another museum, the exhibit can be viewed while viewing the album created when the previous viewing was performed.

In the present embodiment, the server 10 may switch whether or not to generate album data based on the heartbeat variability data received from the terminal device 20, in response to a user request. For example, the server 10 may perform album data generation processing after receiving a request for album data from the terminal device 20 or the user terminal of the user, and transmit the album data to the terminal device 20 or the user terminal. With such a configuration, an increase in processing load can be suppressed without carelessly performing album data generation processing. Note that, also in the present embodiment, the control unit 11 of the server 10 performs the process of step S35 in FIG. 9, and based on the heartbeat variability data of many viewers collected from the plurality of terminal devices 20, for each content. An emotional report may be generated. In this case, it is possible to analyze the emotions of the visitors who appreciate the exhibits, and to understand whether the visitors are interested in each exhibit.

In the present embodiment, the content reproduced by the terminal device 20 may be content for explaining or guiding tourist spots, sightseeing spots, and the like in addition to commentary content for exhibits such as museums. It may be electronic data (content) that is itself an object of appreciation, such as a drama. When using contents for tourist guidance, an album that extracts places (sightseeing spots or tourist spots) where the user has a certain emotion based on the user's heart rate variability data, and summarizes photos or explanations of the extracted places Data can be generated. Further, when using video contents such as movies, animations, and dramas, a portion (scene) in which the user has a predetermined emotion is extracted based on the user's heart rate variability data, and, for example, an album that summarizes the scenes that the user likes. Data can be generated.

(Embodiment 4)
An analysis system for determining the health condition of the user based on the face image of the user when the user uses his/her own user terminal (terminal device 20) will be described. The analysis system of the present embodiment can be realized by the same devices as the devices of the first embodiment, and therefore the description of the configuration will be omitted. In the present embodiment, the terminal device 20 is a user terminal owned by the user.

FIG. 16 is a flowchart showing the procedure of the health condition determination processing by the terminal device 20 of the fourth embodiment. The following processing is executed by the control unit 21 according to the control program 22P stored in the storage unit 22 of the terminal device 20. In the present embodiment, for example, when the user is making a videophone call using the terminal device 20 or is performing an operation related to e-mail, LINE (registered trademark), and SNS (Social Networking Service), the control unit 21 of the terminal device 20 is Shooting by the camera 26 is started, and the face image of the user who is operating is acquired (S81). Here, the control unit 21 performs the same processing as step S12 in FIG.

The control unit 21 performs the same process as step S13 in FIG. 8 and determines the user's attributes (age and sex) based on the acquired face image (face image data) (S82). Next, the control unit 21 performs the same process as step S14 in FIG. 8 and performs a separation process for extracting a pigment component corresponding to hemoglobin on the face image acquired in step S81 to generate a hemoglobin image ( S83). Then, the control unit 21 calculates the hemoglobin pigment concentration (hemoglobin concentration) of the user based on the generated hemoglobin image (S84). For example, the control unit 21 calculates the average value (average pixel value) of the pixel values (the component amount of the hemoglobin component) of each pixel in the face area of the user in the hemoglobin image, and specifies the hemoglobin pigment concentration corresponding to the calculated value. To do. By storing the average pixel value in the hemoglobin image and the hemoglobin pigment concentration corresponding to the average pixel value in the storage unit 22 in association with each other, the control unit 21 controls the average pixel value of the hemoglobin image from the user's hemoglobin. The dye concentration can be specified.

The control unit 21 determines whether the calculated hemoglobin pigment concentration is within the predetermined range (S85), and when it is determined that the calculated hemoglobin dye concentration is not within the predetermined range (S85: NO), the user's health may not be good. It is notified that there is (S86). For example, the control unit 21 displays a message indicating that there is a possibility that the health condition is not good, and notifies the display unit 24 of the message. In addition, when the terminal device 20 has a lamp, the notification may be performed by lighting or blinking the lamp, and when the terminal device 20 has a speaker or a buzzer, the notification may be performed by audio output from the speaker or ringing of the buzzer. .. When determining that the hemoglobin dye concentration is within the predetermined range (S85: YES), the control unit 21 skips the process of step S86 and ends the process.

By the above-described processing, the health condition of the user can be determined based on the face image of the user, and when there is a possibility that the health condition is not good, it can be notified. Therefore, the user can judge the health condition without being aware of the fact that the user is using (operating) the terminal device 20. For example, since the hemoglobin concentration in blood increases when the amount of water in the body decreases, it is determined whether or not there is a sign of developing heat stroke depending on the hemoglobin concentration. In the present embodiment, since the user's hemoglobin concentration is detected from the user's face image, it is possible to determine whether there is a sign of developing heat stroke without performing a blood test. Therefore, when there is a possibility of heat stroke, an early treatment becomes possible by proposing hydration and guiding to the shade. Further, for example, a tablet terminal used by a worker who is working at a construction site or a construction site may be used as the terminal device 20. In this case, the health condition of the worker can be determined from the face image by capturing the face image of the worker who operates the tablet terminal. In addition, for example, when a tourist guide application is provided to tourists who are outdoors for a long time in a tourist area or the like, the tourist guide application may be additionally provided with a function for determining a health condition as in the present embodiment. .. In this case, the tourist guide application can provide the user with a tourist guide, determine the user's health condition, and notify the user if the health condition may not be good.

In the present embodiment, the health condition of the user is determined based on the hemoglobin image.For example, heartbeat data indicating the heartbeat of the user is generated based on the hemoglobin image, and the health condition of the user is determined in consideration of the heartbeat data. May be. In this case, not only changes in hemoglobin concentration but also health conditions such as arrhythmia can be checked based on fluctuations in heartbeat. The configuration of the present embodiment can also be applied to the emotion analysis system of the above-described first to third embodiments. When applied to the emotion analysis system of the embodiment 1-2, the terminal device 20 can determine the health condition based on the face image of the user while providing various contents to the user. When applied to the emotion analysis system of the third embodiment, the terminal device 20 rented out at an art museum or the like provides the user (entry person) with the commentary content of the exhibit and determines the health condition based on the face image of the user. it can.

In each of the above-described embodiments, the terminal device 20 may be realized by installing a predetermined application program on a smartphone, for example. Note that smartphones with cameras are configured to automatically perform image quality correction processing (image quality correction processing) such as white balance processing on the acquired captured image when the image is captured by the camera. It may have been. When realizing the terminal device 20 using a smartphone having such an image quality correction function, the control unit 21 performs the image quality correction process on the face image data when the camera 26 acquires the face image data. Function as. In such a configuration, the control unit (prohibition unit) 21 performs image quality correction processing on the face image of the viewer captured by the camera 26 when performing the emotion analysis of the viewer based on the face image of the viewer. Configured to prohibit execution. By not performing the image quality correction process that is automatically executed, the face image of the viewer can be faithfully acquired, and the emotion of the viewer can be accurately analyzed based on such a face image.

It should be considered that the embodiments disclosed this time are exemplifications in all points and not restrictive. The scope of the present invention is shown not by the above meaning but by the scope of the claims, and is intended to include meanings equivalent to the scope of the claims and all modifications within the scope.

10 server (information processing device)
11 control unit 12 storage unit 20 terminal device 21 control unit 22 storage unit 24 display unit 26 camera 101 heartbeat variability data acquisition unit 103 emotion analysis unit 104 report generation unit 202 imaging unit 204 hemoglobin image generation unit 205 heartbeat data generation unit 206 heartbeat variability Data generation unit 207 Output unit

Claims (15)

Display the content on the display,
Acquire face image data of the face of the user who visually recognizes the displayed content,
To generate hemoglobin image data by extracting a pigment component corresponding to hemoglobin from the face image data,
Generate heartbeat data indicating the heartbeat of the user based on the hemoglobin image data,
Frequency conversion is performed on the heartbeat data to generate heartbeat variability data indicating the heartbeat variability of the user,
An information processing method in which a computer executes a process of analyzing the emotion of the user based on the heart rate variability data and generating emotion data indicating the emotion of the user. The information processing method according to claim 1, wherein the computer executes a process of generating an emotion report for the content based on emotion data of a plurality of the users. The information processing method according to claim 2, wherein the computer executes a process of generating an emotion report indicating analysis values for a plurality of emotion parameters at each time point from the start of reproduction of the content that is a moving image. The computer executes a process of outputting display data for displaying analysis values for the plurality of emotional parameters based on the emotional report and the content in synchronization with an elapsed time from the start of reproduction of the content. The information processing method according to item 3. The computer according to claim 2, wherein the computer executes a process of generating an emotion report indicating the time from the start of reproduction at the time when the analysis values for a plurality of emotion parameters become maximum values for the content that is a moving image. The information processing method described in any one. The content includes a questionnaire having a plurality of questions,
6. The computer according to claim 1, wherein the computer executes a process of generating emotional data of the user for each question based on face image data of a user who answers each of the plurality of questions. Information processing method. The content includes a question regarding stress check,
Accept the answers to the above questions,
Generating emotional data of the user for the question based on face image data of a user who answers the question,
The information processing method according to any one of claims 1 to 6, wherein the computer executes a process of outputting the received answer and the generated emotion data in association with each other. From the content, based on the emotion data, extract a portion where the user has a predetermined emotion,
The information processing method according to any one of claims 1 to 7, wherein the computer executes a process of generating album data using data corresponding to the extracted portion. Storing the generated album data in a storage unit in association with the identifier assigned to the user,
The information processing method according to claim 8, wherein the computer executes a process of outputting the album data when the album data is requested based on the identifier. A display section for displaying content,
An acquisition unit that acquires face image data of the face of the user who visually recognizes the content displayed on the display unit;
An extraction unit that extracts hemoglobin image data by extracting a pigment component corresponding to hemoglobin from the face image data,
A heartbeat data generation unit that generates heartbeat data indicating the user's heartbeat based on the hemoglobin image data;
A heart rate variability data generation unit that performs frequency conversion on the heart rate data to generate heart rate variability data indicating the heart rate variability of the user,
An output unit that outputs the heartbeat variability data together with content information related to the content. A face image data acquired by the acquisition unit, and a storage unit that stores at least one of the hemoglobin image data generated by the extraction unit,
The erasing unit that erases the face image data or the hemoglobin image data stored in the storage unit from the storage unit after the output unit outputs the heartbeat variability data and the content information. Terminal device. A correction unit that performs image quality correction processing on the face image data when the acquisition unit acquires face image data;
The terminal device according to claim 10 or 11, further comprising: a prohibition unit that prohibits execution of the image quality correction process by the correction unit when the heartbeat variation data is generated based on the face image data. A data acquisition unit that acquires heartbeat variability data indicating heartbeat variability of the user generated based on face image data of the face of the user who visually recognizes the displayed content;
An emotion analysis unit that analyzes emotions of the user based on the heart rate variability data and generates emotion data indicating the emotions of the user;
An information processing device, comprising: a report generation unit that generates an emotion report for the content based on a plurality of emotion data of the users generated based on a plurality of heart rate variability data. The information processing device according to claim 13, wherein the data acquisition unit acquires the heartbeat variability data and the content information output by the terminal device according to any one of claims 10 to 12. A location extraction unit that extracts a location where the user has a predetermined emotion from the content based on the emotion data of the user;
The information processing apparatus according to claim 13 or 14, further comprising: an album generation unit that generates album data using data corresponding to the extracted location.