JP2021177362A - Information processing apparatus, information processing method, information processing program, and terminal apparatus - Google Patents

Abstract

To provide information useful for a user in response to change in feelings of the user which occurs upon browsing contents.SOLUTION: An information processing apparatus according to an embodiment of the present invention has an acquiring unit and a specifying unit. The acquiring unit acquires information relating to feelings of a user estimated based on an expression of the user indicated by captured image information obtained by capturing an image of a user browsing a content by imaging means of a terminal indicating a content. The specifying unit specifies a feeling point as a point of the content when the feelings of a user are changed, by collecting estimated result acquired by the acquiring unit.SELECTED DRAWING: Figure 6

Description

Embodiments of the present invention relate to information processing devices, information processing methods, information processing programs, and terminal devices.

Conventionally, there is known a technique of determining a user's psychological state and emotion based on biometric information acquired from the user and providing a service according to the determined user's psychological state and emotion. For example, an information terminal device that detects a user's emotion and processes a message being created by the user according to the detected emotion is disclosed.

For example, an information terminal device measures a user's biometric information by using a biosensor when the user composes a message, and uses the measured biometric information to determine the user's psychological state and emotional strength. The information indicating is calculated. Then, the information terminal device processes the mail created by the user based on the calculated information, and transmits the processed message to convey the user's emotions.

Japanese Unexamined Patent Publication No. 2013-029928

However, the above-mentioned prior art merely conveys the user's emotions to the destination of the message, and can provide meaningful information to the user in response to the change in emotions caused to the user by browsing the content. Not always possible.

The present application has been made in view of the above, and is an information processing device, an information processing method, which can provide information meaningful to the user in response to changes in emotions caused to the user by viewing the content. An object of the present invention is to provide an information processing program and a terminal device.

The information processing device according to the present application estimates the user who is viewing the content based on the facial expression of the user indicated by the image pickup information captured by the image pickup means of the terminal device displaying the content. An acquisition unit that acquires information about emotions, and a specific unit that identifies emotion points, which are points where the user's emotions have changed in the content, by aggregating the estimation results acquired by the acquisition unit. It is characterized by having.

According to one aspect of the embodiment, it is possible to provide information that is meaningful to the user in response to changes in emotions that occur in the user by browsing the content.

FIG. 1 is a diagram showing an example of information processing by the information processing apparatus according to the embodiment. FIG. 2 is a diagram showing an example of the presentation process according to the embodiment. FIG. 3 is a diagram showing an example of a display screen according to the embodiment. FIG. 4 is a diagram showing a configuration example of the information processing system according to the embodiment. FIG. 5 is a diagram showing a configuration example of the terminal device according to the embodiment. FIG. 6 is a diagram showing a configuration example of the information processing device according to the embodiment. FIG. 7 is a diagram showing an example of the imaging information storage unit according to the embodiment. FIG. 8 is a diagram showing an example of an estimated information storage unit according to the embodiment. FIG. 9 is a diagram showing an example of an overall aggregation result storage unit according to the embodiment. FIG. 10 is a diagram showing an example of the emotion point storage unit according to the embodiment. FIG. 11 is a diagram showing an example of the performer information storage unit according to the embodiment. FIG. 12 is a flowchart showing information processing executed by the information processing apparatus according to the embodiment. FIG. 13 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device.

Hereinafter, modes for implementing the information processing apparatus, information processing method, information processing program, and terminal apparatus according to the present application (hereinafter, referred to as “embodiments”) will be described with reference to the drawings. Note that this embodiment does not limit the information processing device, information processing method, information processing program, and terminal device according to the present application. Further, in the following embodiments, the same parts are designated by the same reference numerals, and duplicate description is omitted.

[1. An example of information processing]
First, the information processing realized by the information processing apparatus 100 according to the embodiment will be described. FIG. 1 is a diagram showing an example of information processing by the information processing apparatus 100 according to the embodiment. In the following description, as the information processing according to the embodiment, the information processing performed by the terminal device 10 and the information processing device 100 in cooperation with each other will be described. In the present embodiment, the information processing device 100 executes the information processing program according to the embodiment and performs information processing according to the embodiment by cooperating with the terminal device 10. Further, it is assumed that an application (hereinafter, may be referred to as "application AP"), which is an information processing program according to the embodiment, is also installed in the terminal device 10.

Prior to the description of FIG. 1, the information processing system 1 according to the embodiment will be described with reference to FIG. FIG. 4 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 4, the information processing system 1 according to the embodiment includes a terminal device 10, a content distribution device 30, and an information processing device 100. The terminal device 10, the content distribution device 30, and the information processing device 100 are connected to each other via a network N so as to be communicable by wire or wirelessly. The information processing system 1 shown in FIG. 4 may include a plurality of terminal devices 10, a plurality of content distribution devices 30, and a plurality of information processing devices 100.

The terminal device 10 is an information processing device used by the user. The terminal device 10 is, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), a head-mounted display, or the like. In the present embodiment, the terminal device 10 is a smartphone.

Further, the terminal device 10 has two built-in camera functions. One is the main camera, which is used to capture the scenery and people seen by the user. Therefore, the lens for the main camera is provided on the back side of the terminal device 10, for example, in the direction opposite to the user.

The other is an in-camera, which is used, for example, for video calls and face recognition. Therefore, the lens for the in-camera is provided for the user around the display (touch panel) of the terminal device 10. Therefore, the user uses the in-camera to check the image captured by the in-camera lens (for example, the image of his / her face) and the appearance of the image displayed on the display. You can shoot. In the present embodiment, this in-camera is referred to as an "example of imaging means".

Further, as described above, the application AP is installed in the terminal device 10, and the terminal device 10 can dynamically image the user according to the control of the application AP regardless of the operation by the user. .. For example, the terminal device 10 dynamically captures the user's facial expression only while the user is viewing the video content on a predetermined video site (referred to as “video site ST”) under the control of the application AP. More specifically, the terminal device 10 dynamically captures the facial expression of the user only while viewing all or arbitrary moving image contents distributed in the application AP.

In order to dynamically image the user, for example, when the user visits the video site ST or browses arbitrary video content on the video site ST, the terminal device 10 follows the control of the application AP. , To obtain the consent of the user to take an image. For example, if the user is allowed to image the user himself (if consent is obtained), the terminal device 10 performs the image of the user. On the other hand, if the user does not allow the user to be imaged (if consent is not obtained), the terminal device 10 does not image the user.

The content distribution device 30 is a server device or a cloud system that distributes content. For example, the content distribution device 30 distributes moving image content. For example, the content distribution device 30 distributes video content to the terminal device 10 via the video site ST. For example, it is assumed that the user visits the video site ST and specifies the video name or category of the video content to be viewed as a query. In this case, the content distribution device 30 receives a query from the terminal device 10 and displays a list of video contents corresponding to the received query in the video site ST.

Further, the content distribution device 30 distributes video content in a distribution form such as VOD (Video On Demand). For example, the content distribution device 30 distributes video content of various genres such as comedy programs, dramas, movies, and animations. In addition, the content distribution device 30 performs Internet live distribution.

Here, the premise for performing information processing according to the embodiment will be described. For example, when watching a laughing program on the video site ST, there is a case where you search for an interesting point and try to watch only that point, but in such a case, the user needs to move the seek bar etc. and find the interesting point by himself. It is troublesome. For this reason, there is a need to pick up and watch only interesting points.

In order to meet such needs, it is conceivable that a predetermined person in charge (for example, a connoisseur) watches a laughing program and searches for interesting points, but this work is also very troublesome. From this, if the laughing behavior of laughing can be estimated from the emotions (facial expressions) of the user who is watching the laughing program, the points where more users laughed can be extracted as interesting points. After that, the extracted points can be presented to the user who wants to watch this laughing program.

Also, in a comedy program, for example, each performer (for example, a comedy talent or a group) competes for the fun of the story based on the user's reaction to the performer's performance (that is, the story) and appears. There is something like ranking people. In such a laughing program, there is a case where a vote is actually received from each user who was watching the laughing program and ranked by the voting result. As an example, the dedicated server for totaling displays a "voting button" at the bottom of the area where the laughing program is played and displayed while the laughing program is being played and displayed. The user presses the "voting button" when he / she finds the story of this performer interesting. As a result, the dedicated server that aggregates aggregates the voting results for each performer, and assigns a higher ranking to the performers with the larger number of votes.

However, the number of votes calculated by such a voting system may not always accurately reflect the fun. For example, if one user can vote for a set of performers multiple times, an uninterested performer may be intentionally ranked higher. Also, since the action of voting is not a reflexive action but an action based on a firm intention, some users try to press the voting button in a fraudulent manner even if they do not actually find it interesting. There may be. Then, the number of votes does not necessarily accurately reflect the fun.

On the other hand, the user laughs reflexively at interesting material and does not react to uninteresting material (does not intentionally laugh), so identify that the user laughed surely and the user who laughed If the number of people can be totaled for each performer, it can be said that this totaled result reflects the user's intention (feeling of being interesting) more accurately than the above-mentioned number of votes. Then, the performers can be ranked more accurately.

Based on the above assumptions and problems, the information processing device 100 according to the embodiment uses an imaging means included in the terminal device displaying the content to a user who is viewing the content (for example, moving image content). Information on the user's emotions estimated based on the user's facial expression indicated by the captured image information is acquired. Then, the information processing device 100 identifies the emotional points, which are the points at which the user's emotions change in the content, by aggregating the acquired estimation results.

Hereinafter, an example of information processing according to the embodiment will be described. In the following example of information processing, the information processing apparatus 100 executes the information processing program to perform the above procedure. The information processing device 100 is, for example, a server device or a cloud system that performs information processing according to the embodiment. Further, the business owner who manages the content distribution device 30 and the information processing device 100 is referred to as "business owner X". From this, it can be said that the video site ST is the content managed and operated by the business owner X.

First, in the example of FIG. 1, two users, users U1 and U2, are illustrated, but this is an example, and the number of users is not limited to two. Further, the terminal device 10 used by the user U1 is referred to as a terminal device 10-1. Further, the terminal device 10 used by the user U2 is referred to as a terminal device 10-2. On the other hand, when it is not necessary to distinguish the terminal device for each user, it is simply referred to as the terminal device 10. Further, in the example of FIG. 1, it is assumed that both the users U1 and U2 are allowed to image themselves with the in-camera while viewing the video content on the video site ST. That is, the terminal device 10-1 recognizes that the user U1 may be imaged by the in-camera. Further, the terminal device 10-2 recognizes that the user U2 may be imaged by the in-camera.

In such a state, in the example of FIG. 1, it is assumed that the user U1 is viewing the video content VC1 on the video site ST by using the terminal device 10-1. Similarly, it is assumed that the user U2 is viewing the video content VC2 on the video site ST using the terminal device 10-2. Further, it is assumed that both the video contents VC1 and VC2 are funny programs.

The terminal device 10-1 controls the in-camera and captures the face (facial expression) of the user U1 while the user U1 is viewing the moving image content VC1 (step S1). For example, the terminal device 10-1 captures the facial expression of the user U1 as a moving image (face moving image) while the user U1 is viewing the moving image content VC1. Then, the terminal device 10-1 transmits the image pickup information FDA1 obtained by imaging the user U1 to the information processing device 100 (step S2). More specifically, the terminal device 10-1 analyzes the face moving image data obtained by imaging the user U1 to obtain the imaging information FDA1 including the estimation result based on the analysis.

For example, the terminal device 10-1 estimates information about the emotion of the user U1 based on face moving image data (an example of imaging information). Specifically, the terminal device 10-1 estimates the emotion expression behavior of the user U1 as information regarding the emotion of the user U1 based on the facial expression of the user U1 indicated by the face moving image data. For example, the terminal device 10-1 performs an estimation process for estimating the emotional expression behavior of the user U1 by performing facial expression analysis on the face moving image data. In addition, the terminal device 10-1 performs an estimation process for estimating the emotional expression behavior of the user U1 by analyzing the pupil of the user indicated by the face moving image data.

Here, the emotional expression behavior is a behavior related to so-called emotions, and includes “laughing”, “crying”, “surprise”, and the like. The following embodiments will be described with a particular focus on "laughing" behavior. Hereinafter, the emotional expression behavior is referred to as "laughing behavior". In the example of FIG. 1, it is assumed that the terminal device 10-1 presumes that the user U1 has performed a laughing action. Then, the terminal device 10-1 estimates (calculates) a feature amount indicating the degree of this laughing behavior (the degree of laughing or the degree of laughing). For example, the terminal device 10-1 can estimate a feature amount indicating the degree of laughing behavior with a numerical value of 1 to 10. For example, if the laughter of the user U1 is at the smile level, the terminal device 10-1 estimates the laughter degree "2" as a feature quantity indicating the degree of the laughing behavior. On the other hand, the terminal device 10 estimates the laughter degree "9" if the laughter of the user U1 is at the level of laughter.

The terminal device 10-1 continuously performs the above estimation process while imaging the user U1 at the exact timing when the user U1 is browsing the moving image content VC1, that is, in real time. Then, the terminal device 10-1 transmits the imaging information FDA1 including the estimation result to the information processing device 100, for example, every second. As an example, the terminal device 10-1 takes an image including a time position (time code) corresponding to the playback time of the video content VC1, information indicating the emotion expression behavior, and a feature amount of the emotion expression behavior. Information FDA1 is transmitted to the information processing apparatus 100 every second. The real-time processing can be realized by communicating via a wireless communication network such as 5G (Generation).

As described above, using the example of laughter, since the terminal device 10-1 continuously performs the above estimation process while imaging the user U1, for example, the time position "1 minute 53 seconds", the emotion table. The imaging information FDA1 including information such as the appearance behavior "laughing behavior" and the laughing degree "0" is transmitted to the information processing device 100. Further, the terminal device 10 transmits the imaging information FDA1 including information such as the time position "1 minute 54 seconds", the emotion expression action "laughing action", and the laughing degree "2" to the information processing device 100. Further, the terminal device 10 transmits the imaging information FDA1 including information such as the time position "1 minute 55 seconds", the emotion expression action "laughing action", and the laughing degree "9" to the information processing device 100. Note that the terminal device 10-1 may transmit the imaging information at arbitrary time intervals (for example, 3 seconds) instead of transmitting the imaging information every second.

Although the terminal device 10-1 has been described, the same applies to the terminal device 10-2. Specifically, the terminal device 10-2 controls the in-camera while the user U2 is viewing the moving image content VC2, and captures the face (facial expression) of the user U2 (step S1). For example, the terminal device 10-2 captures the facial expression of the user U2 as a facial motion while the user U2 is viewing the moving image content VC2. Then, the terminal device 10-2 transmits the image pickup information FDA2 obtained by imaging the user U2 to the information processing device 100 (step S2). Specifically, the terminal device 10-2 is estimated at the exact timing when the user U2 is browsing the video content VC2, that is, the estimation described using the terminal device 10-1 as an example while capturing the user U2 in real time. The process is performed continuously. Then, the terminal device 10-2 transmits the imaging information FDA2 including the estimation result to the information processing device 100, for example, every second. As an example, the terminal device 10-2 outputs the imaging information FDA1 including the time position corresponding to the playback time of the video content VC2, the information indicating the emotional expression behavior, and the feature amount of the emotional expression behavior per second. Each time, it is transmitted to the information processing apparatus 100.

Since the terminal device 10-2 continuously performs the above estimation process while imaging the user U2, for example, the time position "1 minute 53 seconds", the emotion expression behavior "laughing behavior", and the laughing degree "0". The imaging information FDA2 including such information is transmitted to the information processing apparatus 100. Further, the terminal device 10 transmits the image pickup information FDA2 including information such as the time position “1 minute 54 seconds”, the emotion expression action “laughing action”, and the laughing degree “3” to the information processing device 100. Further, the terminal device 10 transmits the imaging information FDA2 including information such as the time position “1 minute 55 seconds”, the emotion expression action “laughing action”, and the laughing degree “10” to the information processing device 100.

Hereinafter, the imaging information FDA1 and the imaging information FDA2 may not be distinguished and may be simply referred to as the imaging information FDA. The information processing device 100 receives the image pickup information FDA transmitted from the terminal device 10 (step S3). In other words, the information processing device 100 acquires the imaging information FDA from the terminal device 10. Further, the information processing apparatus 100 stores the received image pickup information FDA in the image pickup information storage unit 121 (step S4). At this time, the information processing device 100 may acquire attribute information related to the user's attributes from the terminal device 10. Here, the attribute information is information about the user's gender, age, interests and hobbies, information about the area including the user's place of residence and the user's location information.

The imaging information storage unit 121 stores the imaging information FDA obtained by imaging the user who is viewing the content with the in-camera (imaging means) of the terminal device 10 displaying the content. In the example of FIG. 1, the imaging information storage unit 121 has items such as "user ID", "moving image ID", and "imaging information".

The "user ID" indicates identification information that identifies the user or the user's terminal device 10. The "video ID" indicates the video content that the user watches and identifies the video content that is captured by the in-camera and that the user is watching. The "imaging information" is imaging information obtained by imaging a user who is viewing moving image content with an in-camera, and indicates imaging information including an estimation result by an estimation process of the terminal device 10. The imaging information may also include data of a facial motion image captured by the user.

That is, in the example of the imaging information storage unit 121 shown in FIG. 1, the user (user U1) identified by the user ID "U1" is viewing the moving image content (video content VC1) identified by the moving image ID "VC1". In the example, the image pickup information FDA1 including the facial expression of the user U1 is obtained by being imaged by the in-camera of the terminal device 10.

Next, the information processing device 100 specifies the time position where the laughing action is performed in the moving image content (step S5). As described above, the information processing apparatus 100 obtains estimation results such as time position (for example, 1 minute 55 seconds), emotional expression behavior (for example, laughing behavior), and laughter degree (feature amount) (for example, “9”). The including imaging information is received from the terminal device 10 (terminal devices 10-1 and 10-2 in the example of FIG. 1) every second. Therefore, the information processing device 100 specifies the time position where the laughing action is performed in the moving image content based on the estimation result (imaging information) by the terminal device 10. For example, in the information processing device 100, a time position in which the laughter degree, which is a feature amount, indicates a predetermined threshold value (for example, the laughter degree “5”) or more is set as a time position in which the user U1 performs a laughing action in the video content VC1. Identify. In such an example, it is assumed that the information processing apparatus 100 specifies the time position "t2, t21, t51 ..." Of the moving image content VC1 as the time position where the user U1 performs a laughing action.

Further, it is assumed that the information processing device 100 specifies the time position "t13, t31, t52 ..." Of the moving image content VC2 as the time position where the user U2 performs the laughing action.

Next, the information processing device 100 associates the emotion expression behavior estimated by the terminal device 10 with the time position specified in step S5 and stores it in the estimated information storage unit 122 (step S6). The estimation information storage unit 122 stores information regarding the estimation result of estimating the emotion expression behavior. In the example of FIG. 1, the estimation information storage unit 122 has items such as "moving image ID", "user ID", and "behavior information (laughing)". The information processing device 100 may estimate not only a laughing behavior but also a crying behavior, a surprised behavior, and the like as emotional expression behaviors. Therefore, the "behavior information" may include items such as "crying" and "surprise".

Further, the information processing apparatus 100 may estimate the attribute information of the user from the facial motion of the user by using a conventional technique such as image analysis. Then, the information processing device 100 may store the attribute information of the user in the estimation information storage unit 122 in association with the "behavior information". When the information processing device 100 has acquired the user's attribute information from the terminal device 10 in advance, the information processing device 100 stores the user's attribute information in the estimation information storage unit 122 in association with the user's "behavior information". You may.

The "video ID" is the video content that the user watches, and indicates identification information that identifies the video content that the user is watching, which is captured by the in-camera. The "user ID" indicates identification information that identifies the user who views the corresponding moving image content or the terminal device of the user. "Behavioral information (laughter)" indicates the time position at which the laughing behavior was performed among the emotional expression behaviors estimated by the estimation processing.

As described in the above example, the information processing apparatus 100 specifies that the laughing behavior of the user U1 is performed in the video content VC1 "t2, t21, t51 ...". Therefore, the information processing device 100 has a moving image ID “VC1”, a user ID “U1”, and behavior information (laughter) “t2, t21, t51 ...” As in the example of the estimation information storage unit 122 shown in FIG. Are associated and stored.

Further, as in the above example, the information processing apparatus 100 specifies that the laughing action of the user U2 is performed in the video content VC2 "t13, t31, t52 ...". Therefore, the information processing device 100 has a moving image ID “VC2”, a user ID “U2”, and behavior information (laughter) “t13, t31, t52 ...” As in the example of the estimation information storage unit 122 shown in FIG. Are associated and stored.

In addition, since the estimation information storage unit 122 stores the time position in which the emotion expression action is performed in the video content for the emotion expression action estimated to have been performed by each user in each video content, each user It can be said that it is a storage unit that stores the aggregated results of. On the other hand, the overall aggregation result storage unit 123, which will be described later, collects the aggregation results for each user and stores the aggregation results re-aggregated to see what happens when all users see it.

Next, the information processing device 100 totals the number of people who have performed emotional expression actions in the moving image content based on the time position specified in step S5 (step S7). For example, the information processing device 100 totals the number of people who have performed a laughing action in each video content for each time position of each video content. For example, the information processing apparatus 100 uses the information stored in the estimation information storage unit 122 to perform such aggregation.

In the example of FIG. 1, in the information processing device 100, at the time position "t1" of the video content VC1, "135" out of the total number of users who watched the video content VC1 during a predetermined period acted to laugh. It is assumed that the aggregated result with (135 people in action) is obtained. Further, in the information processing device 100, at the time position "t2" of the video content VC1, "693" out of the total number of users who watched the video content VC1 during the predetermined period acted to laugh (the number of actions 693). It is assumed that the aggregated result with (person) is obtained. Further, in the information processing device 100, at the time position "t3" of the video content VC1, "86 people" out of the total number of users who watched the video content VC1 during the predetermined period acted to laugh (the number of actions 86). It is assumed that the aggregated result with (person) is obtained.

Further, in the example of FIG. 1, at the time position "t1" of the video content VC2, the information processing device 100 makes "321 people" laugh out of the total number of users who watched the video content VC2 during a predetermined period. It is assumed that the total result of the visit (321 people in action) is obtained. Further, in the information processing device 100, at the time position "t2" of the video content VC2, "592 people" out of the total number of users who watched the video content VC2 during the predetermined period performed an action of laughing (number of actions 592). It is assumed that the aggregated result with (person) is obtained. Further, in the information processing device 100, at the time position "t3" of the video content VC2, "293 people" out of the total number of users who watched the video content VC2 during the predetermined period acted to laugh (the number of actions 293). It is assumed that the aggregated result with (person) is obtained.

Next, the information processing apparatus 100 stores the number of active persons in the total aggregation result storage unit 123 as the aggregation result in step S7 (step S8). The total total result storage unit 123 stores the number of users who have performed a laughing action out of the total number of users who have viewed each video content during a predetermined period, for each time position of each video content. .. In the example of FIG. 1, the total total result storage unit 123 has items such as "moving image ID" and "behavior information (laughing)". The information processing device 100 may estimate not only a laughing behavior but also a crying behavior, a surprised behavior, and the like as emotional expression behaviors. Therefore, the "behavior information" may include items such as "crying" and "surprise".

The "video ID" is the video content that the user watches, and indicates identification information that identifies the video content that the user is watching, which is captured by the in-camera. Items ("t1", "t2", "t3" ...) Associated with "behavior information (laughing)" indicate the time position of each video content, and the user who browsed the video content for a predetermined period of time. Of the total number of users, the number of users who performed a laughing action at that time position is input.

As in the above example, in the information processing device 100, the number of actions is "135" at the time position "t1" of the video content VC1, the number of actions is "693" at the time position "t2", and the number of actions is "693" at the time position "t3". The total result is "86 people". Therefore, the information processing apparatus 100 inputs "135 people" in the input fields corresponding to the moving image ID "VC1" and the time position "t1" as in the example of the total aggregation result storage unit 123 shown in FIG. Further, the information processing apparatus 100 inputs "693 people" in the input fields corresponding to the moving image ID "VC1" and the time position "t2" as in the example of the total aggregation result storage unit 123 shown in FIG. Further, the information processing apparatus 100 inputs "86 people" in the input fields corresponding to the moving image ID "VC1" and the time position "t3" as in the example of the total aggregation result storage unit 123 shown in FIG.

Further, as in the above example, the information processing device 100 has the number of actions "321" at the time position "t1" of the video content VC2, the number of actions "592" at the time position "t2", and the action at the time position "t3". The total number of people is "293". Therefore, the information processing apparatus 100 inputs "321 people" in the input fields corresponding to the moving image ID "VC2" and the time position "t1" as in the example of the total aggregation result storage unit 123 shown in FIG. Further, the information processing apparatus 100 inputs "592 people" in the input fields corresponding to the moving image ID "VC2" and the time position "t2" as in the example of the total aggregation result storage unit 123 shown in FIG. Further, the information processing apparatus 100 inputs "293 people" in the input fields corresponding to the moving image ID "VC2" and the time position "t3" as in the example of the total aggregation result storage unit 123 shown in FIG.

Next, the information processing device 100 presents information about the moving image content to the user based on the aggregation result in step S8, that is, the number of people acting (step S9). For example, the information processing device 100 is a graph showing the transition of the number of actions, which is the number of users who have performed emotional expression actions in the video content, and changes according to the time position of the video content. It is presented in association with the time position indicated by the seek bar displayed together with the moving image content.

As described above, the total total result storage unit 123 sets the number of actions, which is the number of users who performed a laughing action, out of the total number of users who watched each video content during a predetermined period, at the time position of each video content. Remember every time. For this reason, the total result stored in the total total result storage unit 123 can be rephrased as a transition of the number of active persons that changes according to the time position of the moving image content. Therefore, in step S9, the information processing device 100 changes the number of actions, which is the number of users who have performed emotional expression actions in the video content, and changes according to the time position of the video content. The display is controlled so that the graph shown is displayed in association with the time position indicated by the seek bar displayed together with the moving image content.

Here, FIG. 2 shows an example of the presentation process according to the embodiment. In FIG. 2, a case where the user U1 browses the moving image content VC2 will be described as an example. First, the terminal device 10 transmits a distribution request for streaming distribution of the video content VC2 on the video site ST to the content distribution device 30 in response to the operation of the user U1 (step S10). For example, if the user U1 specifies a query indicating the video content VC2 on the video site ST, the terminal device 10 transmits a distribution request including such a query to the content distribution device 30.

Subsequently, when the content distribution device 30 receives the distribution request, the content distribution device 30 notifies the information processing device 100 that the distribution request for the video content VC2 has been received from the terminal device 10 of the user U1 (step S11). For example, the content distribution device 30 notifies the information processing device 100 of information including the user ID “U1” and the moving image ID “VC2”.

When the information processing device 100 receives the notification from the content distribution device 30, the information processing device 100 is the number of users who have performed a laughing action in the video content VC2, and changes according to the time position of the video content VC2. A graph G showing the transition of the number of people to perform is generated (step S12). Specifically, the information processing device 100 accesses the total aggregation result storage unit 123 and acquires the number of actions associated with the moving image ID “VC2”. More specifically, the information processing device 100 changes the time position of the video content VC2 (for example, a change in the time position such as time positions t1, t2, and t3) as the number of actions associated with the video ID "VC2". Acquire the number of actions that change accordingly. In the example of FIG. 1, the information processing apparatus 100 has actions that change according to changes in the time position, such as "321 people" at the time position t1, "592 people" at the time position t2, and "293 people" at the time position t3. Acquire the transition (transition information) of the number of people.

Then, the information processing apparatus 100 generates the graph G based on the acquired transition information. For example, the information processing device 100 plots the graph G by plotting the number of actions corresponding to each time position, with the horizontal axis (X coordinate) as the time position of the moving image content VC2 and the vertical axis (Y coordinate) as the number of actions. Generate.

Next, the information processing device 100 tells the content distribution device 30 that the graph G generated in step S13 is displayed on the seek bar BR where the user can control the playback location (time position) of the video content VC2. Display control (step S13). Specifically, the information processing device 100 refers to the content distribution device 30 so that the time position indicated by the horizontal axis of the graph G, that is, the time position of the moving image content VC2 is displayed in association with the time position of the seek bar BR. Display control. For example, the information processing device 100 instructs the content distribution device 30 to distribute the graph G to the terminal device 10 so that the terminal device 10 displays the graph G on the seek bar BR. Further, the information processing device 100 transmits the graph G to the content distribution device 30.

The time position of the seek bar is associated with the time position of the moving image content VC2. For example, when the seek bar is set to the time position "32 minutes", the user U1 can watch the moving image content VC2 from the playback time "32 minutes". In such a state, the time position of the graph G is also associated with the time position of the seek bar. Therefore, the time position "32 minutes" of the seek bar coincides with the time position "32 minutes" of the graph G.

Return the description. The content distribution device 30 streams and distributes the video content VC2 in response to the display control from the information processing device 100 (step S14). For example, the content distribution device 30 distributes the graph G to the terminal device 10 so as to display the graph G on the seek bar BR when streaming the video content VC2. As a result, the graph G is displayed on the seek bar BR as shown in the display screen D of the terminal device 10 shown in FIG.

According to the example of the display screen D shown in FIG. 2, the area PL1 in which the video content VC2 is actually reproduced and displayed exists in the area AR1 included in the video site ST, and the video content VC2 is reproduced in the area PL1. The play button BT3 for starting is displayed. The area PL1 can be rephrased as the player PL1 that controls the reproduction of the moving image content VC2. The player PL1 may be, for example, a browser version player (web player) that controls playback of video content on a browser, or a player (application version player) as an application (application AP). In addition, the moving image content may be played in advance in a state of being selected from the playback positions with the largest number of active players among the time positions of the seek bar BR. In addition, the user may be presented with whether or not to play the video content from the playback position where the number of active players is the largest.

Further, a graph G is displayed on the seek bar BR. As described above, the time position of the seek bar BR and the time position of the graph G coincide with each other. Further, since the vertical axis of the graph G indicates the number of people acting, the user U1 can grasp at a glance the graph G at which time position other users often laughed. Therefore, for example, when the user U1 picks up and views only interesting points in the video content VC2, for example, by moving the cursor of the seek bar BR to the time position corresponding to the peak of the graph G, the interesting points can be easily obtained. You can move to the place of. This also eliminates the hassle of having a connoisseur find an interesting point.

Further, with reference to FIG. 3, an example of a display screen in the case of delivering the moving image content VC2 having a mark that abstracts emotions at a predetermined time position will be described. FIG. 3 is a diagram showing an example of a display screen according to the embodiment. Here, the content distribution device 30 will be described as streaming distribution of the video content VC2 having a mark that abstracts emotions at a predetermined time position of the video content VC2 in response to the display control from the information processing device 100. ..

According to the example of the display screen T shown in FIG. 3, the player PL2 in which the moving image content VC2 is actually reproduced and displayed is displayed in the area AR2 included in the moving image site ST. Further, according to the example of the display screen T shown in FIG. 3, the graph G is displayed on the seek bar BR.

Here, among the video content VC2, among the total number of users who watched the video content VC2 at the time position "t2" during the predetermined period, the largest number "592" performed the laughing behavior. It shall be. In this case, the information processing device 100 controls the display of the moving image content VC2 so as to attach the laughing emoticon mark MR at the time position “t2” to the content distribution device 30. For example, the information processing apparatus 100 controls the display at the time position “t2” of the moving image content VC2 in such a display manner that the laughing emoticon mark MR quickly pops out from the lower direction of the player PL2 to the vicinity of the center of the player PL2. In this case, the laughing emoticon mark MR is displayed so as to be superimposed on the moving image content VC2. As a result, the information processing device 100 can provide a service that creates a sense of unity among users even in real-time distribution without an audience.

Further, the moving image content VC2 may be played in a state of being selected in advance from the playing position having the largest number of active players among the time positions of the seek bar BR. Further, the user may be presented with whether or not to play the moving image content V2 from the playback position having the largest number of active players. For example, in the example of FIG. 3, a laughing emoticon mark MR is attached to the time position “t2” in the graph G. As a result, the user may be presented to play the moving image content V2 from the time position "t2", which is the playback position where the number of people who have performed the most laughing action is large.

It is not limited to the example of attaching the laughing emoticon mark MR, and instead of the mark that abstracts emotions, a laughing voice, a sound effect, or a character may be attached. As described above, the information processing apparatus 100 may display and control the display so as to add any information as long as the effect is such that the excitement of the moving image content can be produced. Further, in the above example, laughter has been described as an example of the user's emotions, but the above processing can also be applied to emotions such as crying and surprise.

As described above with reference to FIGS. 1 and 2, the information processing device 100 according to the embodiment allows the user who is viewing the content (for example, moving image content) to the terminal device 10 displaying the content. Acquires information on the user's emotions estimated based on the user's facial expression indicated by the imaged information captured by the in-camera. Then, the information processing device 100 identifies the emotional points, which are the points at which the user's emotions change in the content, by aggregating the acquired estimation results. In addition, the information processing device 100 presents information about the content based on the estimation result. As a result, the information processing device 100 according to the embodiment can provide information that is meaningful to the user in response to changes in emotions that occur in the user by viewing the content.

[2. Terminal device configuration]
Next, the terminal device 10 according to the embodiment will be described with reference to FIG. FIG. 5 is a diagram showing a configuration example of the terminal device 10 according to the embodiment. As shown in FIG. 5, the terminal device 10 includes a communication unit 11, a display unit 12, an imaging unit 13, and a control unit 14. The terminal device 10 is an information processing device used by the user.

(About communication unit 11)
The communication unit 11 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 11 is connected to the network N by wire or wirelessly, and transmits / receives information to / from, for example, the content distribution device 30 and the information processing device 100.

(About display unit 12)
The display unit 12 is a display device that displays various types of information, and corresponds to the display screen D shown in FIG. For example, a touch panel is adopted for the display unit 12. Further, the display unit 12 displays, for example, an image captured from the lens by the imaging unit 13.

(About the imaging unit 13)
The image pickup unit 13 is a device that has a built-in image pickup element and captures an image or a moving image. The image pickup device may be any of CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor) and the like. For example, the imaging unit 13 can take a picture of an image captured from the lens and currently displayed on the display unit 12 as a still image, or take a moving image. Further, the imaging unit 13 is assumed to correspond to the in-camera described with reference to FIG.

(About control unit 14)
In the control unit 14, various programs stored in the storage device inside the terminal device 10 are executed by the CPU (Central Processing Unit), MPU (Micro Processing Unit), etc. using the RAM (Random Access Memory) as the work area. Is realized by. Further, the control unit 14 is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Further, the control unit 14 is a processing unit executed by the information processing program (application AP) according to the embodiment.

As shown in FIG. 5, the control unit 14 includes a request unit 14a, a consent information reception unit 14b, a display control unit 14c, a camera control unit 14d, an acquisition unit 14e, an estimation unit 14f, and a transmission unit 14g. It has and realizes or executes the functions and actions of information processing described below. The internal configuration of the control unit 14 is not limited to the configuration shown in FIG. 5, and may be another configuration as long as it is a configuration for performing information processing described later. Further, the connection relationship of each processing unit included in the control unit 14 is not limited to the connection relationship shown in FIG. 5, and may be another connection relationship.

(Request unit 14a)
The requesting unit 14a requests the distribution of content (for example, moving image content). For example, the requesting unit 14a requests the content distribution device 30 to distribute the content. For example, the request unit 14a transmits a distribution request requesting the distribution of the content to the content distribution device 30. In the example of FIG. 1, the terminal device 10 transmits a distribution request for streaming distribution of the video content on the video site ST to the content distribution device 30 in response to the operation of the user U1. For example, if the user U1 specifies a query indicating the video content VC2 on the video site ST, the terminal device 10 transmits a distribution request including such a query to the content distribution device 30. In addition, the request unit 14a receives the content distributed from the content distribution device 30.

(Consent Information Reception Department 14b)
Consent information reception unit 14b receives consent information from the user indicating whether or not he / she agrees to be imaged by the in-camera (imaging unit 13) (whether or not he / she is permitted to be imaged). In the example of FIG. 1, whether or not the consent information receiving unit 14b agrees to be imaged by the in-camera (imaging unit 13) only while viewing arbitrary video content on the video site ST (imaged). Accepts consent information from the user indicating (whether or not to allow it). For example, the consent information receiving unit 14b receives consent information to the effect that when the "agreement button" displayed on the video site ST is pressed, the consent information is agreed to be imaged by the in-camera (imaging unit 13).

(About display control unit 14c)
The display control unit 14c performs display control for displaying various information on the display screen D (display unit 12) of the terminal device 10. For example, the display control unit 14c causes the display screen D to display the information received by the request unit 14a. For example, the display control unit 14c displays the moving image site ST on the display screen D. Further, the display control unit 14c displays the moving image content on the display screen D. For example, in the example of FIG. 2, the requesting unit 14a receives the moving image content VC2. In such a case, the display control unit 14c displays the player PL1, the graph G, and the seek bar BR in the area AR.

(About camera control unit 14d)
The camera control unit 14d captures the user by controlling the in-camera (imaging unit 13). For example, the camera control unit 14d controls the in-camera according to the consent information received by the consent information reception unit 14b. For example, when the camera control unit 14d receives the consent information to the effect that the consent information reception unit 14b agrees to be imaged, the camera control unit 14d only while the user is browsing arbitrary video content on the video site ST. Control the in-camera. That is, the camera control unit 14d controls the in-camera so that the user is imaged only while the user is browsing an arbitrary video content on the video site ST.

(About acquisition unit 14e)
The acquisition unit 14e acquires imaging information (face moving image data) obtained by imaging a user who is viewing the content with an in-camera included in the terminal device 10 displaying the content. For example, the acquisition unit 14e acquires imaging information from the camera control unit 14d.

Further, for example, the acquisition unit 14e acquires the imaging information obtained by imaging the user who is viewing the moving image content or the image content as the content. Video content is video content of various genres such as comedy programs, dramas, movies, and animations. On the other hand, the image content is, for example, various electronic books. In addition, the acquisition unit 14e acquires the imaging information obtained by imaging with the in-camera when the user's permission is obtained as the imaging information. For example, the acquisition unit 14e acquires the imaging information obtained by imaging with the in-camera while the content is displayed when the user's permission is obtained as the imaging information.

(About estimation unit 14f)
The estimation unit 14f performs the estimation process described in step S2 of FIG. Specifically, the estimation unit 14f estimates information about the user's emotions based on the facial expression of the user indicated by the imaging information acquired by the acquisition unit 14e. For example, the estimation unit 14f estimates the user's emotion expression behavior as information on the user's emotions based on the user's facial expression indicated by the imaging information. Emotional expression behaviors are behaviors that express emotions, such as laughing behaviors that occur when emotions such as funny occur, and crying behaviors that occur when emotions such as sadness occur. Also, for example, the estimation unit 14f estimates information about the user's emotions during playback of the content (that is, at the very moment when the user is viewing the content). Further, the estimation unit 14f estimates a feature amount indicating the degree of the user's emotion expression behavior as information on the user's emotion based on the user's facial expression indicated by the imaging information.

In the example of FIG. 1, the acquisition unit 14e acquires face moving image data (an example of imaging information) obtained by imaging by the camera control unit 14d and transmits it to the estimation unit 14f. Then, the estimation unit 14f estimates information about the user's emotions based on the face moving image data (an example of imaging information). Specifically, the estimation unit 14f estimates the user's emotional expression behavior as information on the user's emotions based on the user's facial expression indicated by the facial expression data. For example, the estimation unit 14f estimates the emotional expression behavior of the user by performing facial expression analysis on the face moving image data.

In addition, the estimation unit 14f estimates a feature amount indicating the estimated degree of emotional release behavior. For example, when the estimation unit 14f estimates "laughing behavior" as an emotional release behavior, it estimates (calculates) a feature amount indicating the degree of this laughing behavior (the degree of laughing or the degree of laughing). .. For example, if the user's laughter indicated by the face video data is at the smile level, the estimation unit 14f estimates the laughter degree "2" as a feature quantity indicating the degree of laughing behavior. On the other hand, the estimation unit 14f estimates the laughter degree "9" if the user's laughter indicated by the face video data is at the level of laughter.

The estimation unit 14f is not limited to the above example. Specifically, the estimation unit 14f may estimate the user's attribute information based on the facial expression of the user indicated by the imaging information acquired by the acquisition unit 14e. For example, the estimation unit 14f uses conventional techniques such as image analysis to provide feature information that characterizes user attributes such as eye, nose, mouth size, eyebrow shape, facial wrinkles, or hair length. Extract. Then, the estimation unit 14f may estimate the age and gender of the user as the attribute information of the user based on the extracted feature information.

Further, the estimation unit 14f continuously performs estimation processing, for example, every second, in response to the user being imaged at the exact timing when the user is browsing the moving image content, that is, in real time. Therefore, the transmission unit 14g, which will be described later, transmits information including the estimation result of the estimation process by the estimation unit 14f to the information processing apparatus 100, for example, every second. As an example, the transmission unit 14g contains information including a time position (time code) corresponding to the playback time of the video content, information indicating the emotion expression behavior, and a characteristic amount of the emotion expression behavior (FIG. 1). In the example of, the imaging information FDA1 and FDA2) are transmitted to the information processing apparatus 100 every second. That is, while the user is browsing the video content, the transmission unit 14g includes information including a time position (time code), information indicating the emotion expression behavior, and a feature amount of the emotion expression behavior, that is, The estimation result is transmitted to the information processing apparatus 100 one after another.

(About the transmitter 14g)
The transmission unit 14g transmits the estimation result by the estimation unit 14f. Specifically, the transmission unit 14g transmits information including the estimation result by the estimation unit 14f to the information processing device 100. In the example of FIG. 1, the transmission unit 14g transmits the imaging information FDA1 to the information processing device 100. Further, the transmission unit 14g transmits the image pickup information FDA2 to the information processing device 100.

The estimation process by the estimation unit 14f may be performed on the information processing apparatus 100 side. In this case, the information processing device 100 has a processing unit corresponding to the estimation unit 14f. Further, in this case, the transmission unit 14g continuously transmits the face moving image data to the information processing apparatus 100.

[3. Information processing device configuration]
Next, the information processing apparatus 100 according to the embodiment will be described with reference to FIG. FIG. 6 is a diagram showing a configuration example of the information processing apparatus 100 according to the embodiment. As shown in FIG. 6, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 is, for example, a server device that performs information processing according to the embodiment.

(About communication unit 110)
The communication unit 110 is realized by, for example, a NIC or the like. Then, the communication unit 110 is connected to the network N by wire or wirelessly, and transmits / receives information to / from, for example, the terminal device 10 or the content distribution device 30.

(About storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory or a storage device such as a hard disk or an optical disk. The storage unit 120 includes an imaging information storage unit 121, an estimation information storage unit 122, an overall aggregation result storage unit 123, an emotion point storage unit 124, and a performer information storage unit 125.

(About the image pickup information storage unit 121)
The image pickup information storage unit 121 stores the image pickup information obtained by capturing the image of the user who is viewing the content with the in-camera included in the terminal device 10 displaying the content. Here, FIG. 7 shows an example of the imaging information storage unit 121 according to the embodiment. In the example of FIG. 7, the imaging information storage unit 121 has items such as "user ID", "moving image ID", and "imaging information". Since the image pickup information storage unit 121 has already been described with reference to FIG. 1, the description thereof will be omitted.

(About the estimated information storage unit 122)
The estimation information storage unit 122 stores information regarding the estimation result of estimating the emotion expression behavior. Further, the estimation information storage unit 122 stores the time position in which the emotion expression action is performed in the video content for each user, because the estimation information storage unit 122 stores the time position in which the emotion expression action is performed in the video content. It can be said that it is a storage unit that stores the aggregated results of. Here, FIG. 8 shows an example of the estimation information storage unit 122 according to the embodiment. In the example of FIG. 1, the estimation information storage unit 122 has items such as "moving image ID", "user ID", and "behavior information". In addition, "behavioral information" includes items such as "laughing", "crying", and "surprise".

The "video ID" is the video content that the user watches, and indicates identification information that identifies the video content that the user is watching, which is captured by the in-camera. The "user ID" indicates identification information that identifies the user who views the corresponding moving image content or the terminal device of the user.

"Laughter" included in "behavior information" is the time position where the laughing action was performed among the emotional expression actions estimated by the estimation process, and the laughing action in the video content indicated by the corresponding video ID. Indicates the time position where Further, the "laughter" included in the "behavior information" indicates a time position specified by the aggregation unit 132, which will be described later, as having performed a laughing action in the moving image content based on the estimation result by the estimation unit 14f. "Crying" included in "behavior information" is the time position where the crying action was performed among the emotional expression actions estimated by the estimation process, and the crying action in the video content indicated by the corresponding video ID. Indicates the time position where Further, "crying" included in the "behavior information" indicates a time position specified by the aggregation unit 132, which will be described later, as having performed a crying action in the moving image content based on the estimation result by the estimation unit 14f. The "surprise" included in the "behavior information" is the time position where the surprising action was performed among the emotional expression actions estimated by the estimation process, and the surprising action in the video content indicated by the corresponding video ID. Indicates the time position where Further, "surprise" included in the "behavior information" indicates a time position specified by the aggregation unit 132, which will be described later, as having performed a surprising action in the moving image content based on the estimation result by the estimation unit 14f.

That is, in the example of FIG. 8, it is presumed that the user U1 performed a laughing action while browsing the video content VC1, and at the time positions t2, t21, and t51 in the playback time of the video content VC1. Here is an example of how this laughing behavior was identified.

In the present embodiment, the time position may be a time position of a certain point or a time range. For example, the time position "t2" may be a one-point time position such as "2 minutes 35 seconds" or a time range such as "2 minutes 35 seconds to 2 minutes 30 seconds". When the time position indicates a time position of one point, the time position is, for example, the time position where the emotion expression action is started, the time position where the emotion expression action is finished, and the emotion expression action is started. It may be any of the intermediate times in the time range from the time position to the time position where the emotional expression action is completed.

(About the total total result storage unit 123)
The total total result storage unit 123 determines the number of users who have performed laughing behavior (ratio of users who have performed laughing behavior) out of the total number of users who have viewed each video content during a predetermined period. Store each video content at each time position. Here, FIG. 9 shows an example of the total total result storage unit 123 according to the embodiment. In the example of FIG. 9, the total total result storage unit 123 has items such as “moving image ID” and “behavior information”. In addition, "behavior information" includes items such as "number of people who laughed (ratio)", "number of people who cried (ratio)", and "number of people who were surprised (ratio)". In addition, an advertisement indicating the time position of the video content is associated with each of the "number of people who laughed (ratio)", "number of people who cried (ratio)", and "number of people who were surprised (ratio)".

The "video ID" is the video content that the user watches, and indicates identification information that identifies the video content that the user is watching, which is captured by the in-camera.

The time position ("t1", "t2", "t3" ...), Which is an item associated with the "number of people laughing (ratio)", indicates the time position of each video content, and the video is displayed for a predetermined period of time. Of the total number of users who browsed the content, the number of users who performed the laughing action at that time position (the ratio of the users who performed the laughing action) is input. The time position ("t1", "t2", "t3" ...), Which is an item associated with the "number of people crying (ratio)", indicates the time position of each video content, and the video is displayed for a predetermined period of time. Of the total number of users who browsed the content, the number of users who performed the crying action at that time position (the ratio of the users who performed the crying action) is input. The time position ("t1", "t2", "t3" ...), Which is an item associated with the "surprised number of people (ratio)", indicates the time position of each video content, and the video is displayed for a predetermined period of time. Of the total number of users who browsed the content, the number of actions (percentage of users who performed a surprising action), which is the number of users who performed a surprising action at that time position, is input.

That is, in the example of FIG. 9, among the total number of users who browsed the video content VC1 during a predetermined period, the number of users who performed a laughing action at the time position t1 is "135". .. Further, the example of FIG. 9 shows an example in which the ratio of the number of users who performed a laughing action at the time position t1 to the total number of users who browsed the moving image content VC1 during a predetermined period is “20%”.

Further, in the example of FIG. 9, among the total number of users who browsed the video content VC2 during a predetermined period, the number of users who performed a laughing action at the time position t1 is "321". .. Further, the example of FIG. 9 shows an example in which the ratio of the number of users who performed a laughing action at the time position t1 to the total number of users who browsed the moving image content VC2 during a predetermined period is “5%”.

(About emotion point storage unit 124)
The emotion point storage unit 124 stores information about emotion points, which are points in which the user's emotions have changed in the content. When a change in emotion occurs, the user reflexively expresses the emotion in an action, for example, when an emotion such as funny emerges, the user performs a laughing action. For example, when feelings of sadness develop, they act to cry. For example, when a surprised emotion emerges, a surprised action is taken. For this reason, emotional points can be divided into interesting points, crying points, surprise points, and the like. Here, FIG. 10 shows an example of the emotion point storage unit 124 according to the embodiment. As shown in FIG. 10, the emotion point storage unit 124 is divided into emotion point storage units 124-1, 124-2, and 124-3.

First, the emotion point storage unit 124-1 will be described. The emotion point storage unit 124-1 stores information about the user's emotion points. In the example of FIG. 10, the emotion point storage unit 124-1 has items such as “moving image ID” and “emotion point”. In addition, the "emotion point" includes items such as "interesting point", "crying point", and "surprise point".

The "video ID" indicates identification information that identifies the video content viewed by the user. The "interesting point" is that the numerical value based on the number of users who performed the laughing action out of the total number of users who watched the corresponding video content during the predetermined period is a predetermined number or more (condition information). Indicates the time position. Such a numerical value is the total number of users who watched the corresponding video content during the predetermined period, which is the number of users who performed the laughing action, or the corresponding video content during the predetermined period. It is the ratio of the number of users who performed laughing behavior to the total number of users who watched.

For this reason, the "interesting point" is that the number of users who performed the laughing action out of the total number of users who watched the corresponding video content during the predetermined period was equal to or greater than the predetermined number. Indicates the time position. Alternatively, the "interesting point" indicates the time position in which the ratio of the number of users who performed the laughing action to the total number of users who viewed the corresponding video content during the predetermined period was equal to or higher than the predetermined ratio. That is, the "interesting point" is the time position that satisfies the above-mentioned condition information from the time positions stored in the total aggregation result storage unit 123 shown in FIG. In the example of FIG. 10, the "interesting point" is that the number of users who performed the laughing action out of the total number of users who watched the corresponding video content during the predetermined period was equal to or greater than the predetermined number. It shall indicate the time position.

For the "crying point", the numerical value based on the number of users who performed the crying action out of the total number of users who watched the corresponding video content during the predetermined period was a predetermined number or more (condition information). Indicates the time position. Such a numerical value is the total number of users who watched the corresponding video content during the predetermined period, which is the number of users who performed the crying action, or the corresponding video content during the predetermined period. It is the ratio of the number of users who performed crying behavior to the total number of users who watched.

For this reason, the "crying point" is that the number of users who performed the crying action out of the total number of users who watched the corresponding video content during the predetermined period was equal to or greater than the predetermined number. Indicates the time position. Alternatively, the "interesting point" indicates the time position in which the ratio of the number of users who performed the crying action to the total number of users who viewed the corresponding video content during the predetermined period was equal to or higher than the predetermined ratio. That is, the "crying point" is a time position that satisfies the above-mentioned condition information from the time positions stored in the total total result storage unit 123 shown in FIG. In the example of FIG. 10, the “crying point” is the number of users who performed the crying action among the total number of users who watched the corresponding video content during the predetermined period. It shall indicate the time position.

The "surprise point" was a numerical value based on the number of users who performed a surprising action out of the total number of users who watched the corresponding video content during a predetermined period, which was a predetermined number or more (condition information). Indicates the time position. Since the same applies to the "surprise point", further description will be omitted.

In addition, among the time positions indicating "emotion points", a check mark is given to the time position where the number of active persons is the largest (or the ratio is the highest). The "emotion point" is specified by the specific unit 133, which will be described later, and is input to the emotion point storage unit 124.

That is, in the example of FIG. 10, the time position "t1, t31, t62 ..." Is specified as an interesting point for the video content VC1, and the laughing behavior of the time positions "t1, t31, t62 ..." An example is shown in which the time position where the number of people who performed the above is the largest is the time position t31.

Further, in the example of FIG. 10, the time position "t13, t55, t61 ..." Is specified as an interesting point for the video content VC2, and the laughing behavior of the time positions "t13, t55, t61 ..." An example is shown in which the time position where the number of people who performed the above is the largest is the time position t61.

The estimated information storage unit 122 shown in FIG. 8 stores the time position in which each user performs an emotional expression action for each moving image content. Therefore, the time position stored in the estimated information storage unit 122 can be said to be an emotional point of each user.

Next, the emotion point storage unit 124-2 will be described. The emotion point storage unit 124-2 stores information about the emotion points of the user having the age for each age of the user. In the example of FIG. 10, the emotion point storage unit 124-2 has items such as "attribute (age)", "attribute (gender)", "video ID", and "emotion point". In addition, the "emotion point" includes items such as "interesting point", "crying point", and "surprise point". In the example of FIG. 10, an example in which the attributes are "age" and "gender" is shown, but the emotion point storage unit 124, for example, the attributes are the user's interests and hobbies, the user's place of residence, and the user. Emotion points in the case of information about the area including location information can also be memorized. That is, in the example of FIG. 10, what to do with the attribute is not limited.

The "attribute (age)" indicates the age of the user who viewed the video content. The "attribute (age)" may indicate the age of the user who has viewed the video content. "Attribute (gender)" indicates the gender of the user who viewed the video content. The "video ID" is video content viewed by a user with the corresponding attribute, and indicates identification information for identifying the video content being viewed by the user captured by the in-camera.

The "interesting point" is a numerical value based on the number of users who have performed a laughing action out of the total number of users of the corresponding age and gender viewing the video content during a predetermined period. Indicates the time position that was (condition information). Such a numerical value is the number of users of the age and gender who performed the laughing action among the total number of the users when the users of the corresponding age and gender watched the video content during the predetermined period. Or, the ratio of the number of users of such age and gender who performed a laughing action to the total number of users of the corresponding age and gender when viewing the video content during a predetermined period.

For this reason, the "interesting point" is the age and gender of the user who performed the laughing behavior out of the total number of users of the corresponding age and gender when viewing the video content during the predetermined period. Indicates the time position where the number of actions, which is the number of users, is equal to or greater than the predetermined number. Alternatively, the "interesting point" is the ratio of the number of users of such age and gender who performed laughing behavior to the total number of users of the corresponding age and gender when viewing the video content during a predetermined period. Indicates a time position where was greater than or equal to a predetermined proportion. In the example of FIG. 10, the "interesting point" is the age and gender of the user who performed the laughing behavior among the total number of users of the corresponding age and gender when viewing the video content during the predetermined period. It shall indicate the time position where the number of actions, which is the number of users, was equal to or greater than the predetermined number.

The "crying point" is the number of users of the age and gender who performed the crying behavior out of the total number of users of the corresponding age and gender when viewing the video content during the predetermined period. Indicates the time position where the numerical value based on the number of people is equal to or greater than the predetermined number (condition information). The "surprise point" is the number of users in the age group who performed a surprising action out of the total number of users of the corresponding age group and gender when viewing the video content during a predetermined period. Indicates the time position where the base numerical value is equal to or greater than a predetermined number (condition information). Since the concept of "crying point" and "surprise point" is the same as that of "interesting point", further explanation is omitted.

In addition, among the time positions indicating "emotion points", a check mark is given to the time position where the number of active persons is the largest (or the ratio is the highest). The "emotion point" is specified by the specific unit 133, which will be described later, and is input to the emotion point storage unit 124.

That is, in the example of FIG. 10, the time position "t14, t21, t39 ..." Is specified as an interesting point for the video content VC1, and the laughing behavior of the time positions "t14, t21, t39 ..." An example is shown in which the time position where the number of people who performed the above is the largest is the time position t31.

Further, in the example of FIG. 10, an example is shown in which the time position "t13, t55, t61 ..." Is specified as an interesting point for the moving image content VC2 for a male user who is a teenage user. Further, among the time positions "t13, t55, t61 ...", the time position in which the number of people who performed the laughing action is the largest is the time position t21 for the male teenage user.

The estimated information storage unit 122 shown in FIG. 8 stores the time position in which each user performs an emotional expression action for each moving image content. Therefore, the time position stored in the estimated information storage unit 122 can be said to be an emotional point of each user.

(About performer information storage 125)
The performer information storage unit 125 stores information regarding emotional expression behavior performed on performers (for example, talents, entertainers, etc.) who appear in the video content. The performer information storage unit 125 is obtained, for example, by aggregating the information stored in the estimation information storage unit 122. Here, FIG. 11 shows an example of the performer information storage unit 125 according to the embodiment. As shown in FIG. 11, the performer information storage unit 125 is divided into a performer information storage unit 125-1, a performer information storage unit 125-2, and the like.

First, the performer information storage unit 125-1 will be described. The performer information storage unit 125-1 is a user who was watching the video content VC1 at this time when the performer was performing in the video content VC1 for each performer appearing in the video content VC1. Among them, information on the number of users who have performed emotional expression behavior is stored. That is, the performer information storage unit 125-1 stores information such as how much the user who is the viewer laughed at each performer appearing in the video content VC1. In the example of FIG. 11, the performer information storage unit 125-1 has items such as “video ID”, “behavior information”, and “performer”. Further, the "performer" includes symbols (TR11, TR12, TR13, etc.) that conceptually indicate information (for example, name, group name, etc.) indicating each performer.

The "video ID" indicates identification information that identifies the video content viewed by the user. The "behavior information" indicates an emotion-releasing action performed by the user in the corresponding video content.

Further, in the video content VC1, the numerical value "30%" associated with the performer "TR11" and the action information "laughing" performs the action of laughing when the performer "TR11" is performing in the video content VC1. Shows the percentage of users. Further, in the video content VC1, the numerical value "50%" associated with the performer "TR12" and the action information "laughing" performs the action of laughing when the performer "TR12" is performing in the video content VC1. Shows the percentage of users. Further, in the video content VC1, the numerical value "15%" associated with the performer "TR13" and the action information "laughing" performs the action of laughing when the performer "TR13" is performing in the video content VC1. Shows the percentage of users.

Next, the performer information storage unit 125-2 will be described. Since the target video content of the performer information storage unit 125-2 is different from that of the performer information storage unit 125-1, there is a difference that the performer is performing, but in reality, the performer information storage unit It is the same as the part 125-1.

For example, in the video content VC2, the numerical value "3%" associated with the performer "TR21" and the action information "laughing" performs the action of laughing when the performer "TR21" is performing in the video content VC2. Shows the percentage of users. Further, in the video content VC2, the numerical value "3%" associated with the performer "TR22" and the action information "laughing" performs the action of laughing when the performer "TR22" is performing in the video content VC2. Shows the percentage of users. Further, in the video content VC2, the numerical value "3%" associated with the performer "TR23" and the action information "laughing" performs the action of laughing when the performer "TR23" is performing in the video content VC2. Shows the percentage of users.

Returning to FIG. 6, the control unit 130 is realized by executing various programs stored in the storage device inside the information processing device 100 by using the RAM as a work area by the CPU, MPU, or the like. Further, the control unit 130 is realized by, for example, an integrated circuit such as an ASIC or FPGA.

As shown in FIG. 6, the control unit 130 includes a reception unit 131, an aggregation unit 132, a specific unit 133, a presentation unit 134, and an editorial unit 135, and functions and operations of information processing described below. To realize or execute. The internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 6, and may be another configuration as long as it is a configuration for performing information processing described later. Further, the connection relationship of each processing unit included in the control unit 130 is not limited to the connection relationship shown in FIG. 6, and may be another connection relationship.

(About receiver 131)
The receiving unit 131 receives various information. Specifically, the receiving unit 131 receives the information transmitted from the terminal device 10. For example, the receiving unit 131 receives information including the estimation result of the estimation process by the terminal device 10. As described above, the estimation unit 14f performs estimation processing in real time when the user is browsing the moving image content, and the transmission unit 14g sequentially transmits information including the estimation result to the information processing device 100 in real time. Therefore, the receiving unit 131 receives the information in real time when the user is browsing the moving image content. In the example of FIG. 1, the terminal device 10 receives the imaging information FDA1 and FDA2. For example, the receiving unit 131 receives the attribute information of the user. At this time, the receiving unit 131 may receive the user's attribute information from the terminal device 10, or receives the user's attribute information from an external server having a storage unit in which the attribute information is stored in advance for each user. May be good.

(About tabulation section 132)
The totaling unit 132 performs a totaling process for totaling the estimation results estimated by the estimation unit 14f. For example, the aggregation unit 132 specifies the time position where the emotional expression action is performed in the moving image content based on the estimation result estimated by the estimation unit 14f. Then, the specific unit 133 totals the number of times the emotion expression action is performed in the moving image content based on the specified time position. In the example of FIG. 1, the aggregation unit 132 performs an action in which the user U1 laughs at a time position at which the laughter degree, which is a feature amount, indicates a predetermined threshold value (for example, the laughter degree “5”) or more in the video content VC1. Specify as a time position. Further, the aggregation unit 132 stores the information in the estimation information storage unit 122.

Further, for example, the aggregation unit 132 aggregates the number of people who have performed a laughing action in each video content for each time position of each video content. For example, the aggregation unit 132 performs such aggregation using the information stored in the estimation information storage unit 122. In addition, the aggregation unit 132 aggregates the ratio of the number of users who have performed a laughing action in the video content to the total number of users who have viewed the video content for each time position of the video content. do. For example, the aggregation unit 132 performs such aggregation using the information stored in the estimation information storage unit 122. Further, the totaling unit 132 stores the totaled total result in the total totaled result storage unit 123.

In addition, the aggregation unit 132 is a user who has performed an emotional expression action among the users who were watching the video content VC1 when the performer was performing in the content for each performer appearing in the content. Aggregate information about the number of people. For example, the aggregation unit 132 aggregates the percentage of users who laugh when the performer is performing in the content. For example, the aggregation unit 132 aggregates the information stored in the estimation information storage unit 122 to calculate the percentage of users who perform a laughing action when the performer is performing in the content. In addition, the aggregation unit 132 stores the aggregation result at this time in the performer information storage unit 125.

(About specific part 133)
The identification unit 133 identifies an emotion point, which is a point at which the user's emotion changes in the content, by aggregating the estimation results estimated by the estimation unit 14f. As described with reference to FIG. 10, emotion points are divided into fun points, crying points, surprise points, and the like.

For example, the identification unit 133 identifies emotion points for each user by aggregating the estimation results estimated by the estimation unit 14f for each user. The time position for each user stored in the estimation information storage unit 122 can be said to be an emotion point for each user. In addition, the identification unit 133 identifies the emotion points by totaling the number of appearances of the time position, which is the emotion points for each user. In other words, the specific unit 133 identifies emotion points by totaling the number of actions, which is the number of users who have performed emotion expression actions in the content.

For example, the specific unit 133 specifies a point at which a numerical value based on the number of active persons satisfies a predetermined condition information as an emotional point. Specifically, when the content is a moving image content, the specifying unit 133 specifies as an emotion point a time position in which the numerical value based on the number of active persons satisfies the predetermined condition information in the playing time of the moving image content. For example, the specific unit 133 points out an interesting point (emotion) at a time position in which the number of users who have performed a laughing action out of the total number of users who have viewed the corresponding video content during a predetermined period is equal to or greater than the predetermined number of people. Specify as an example of points). Alternatively, the specific unit 133 is at a time position in which the ratio of the number of active users, which is the number of users who have performed a laughing action, to the total number of users who have viewed the corresponding video content during a predetermined period is equal to or greater than the predetermined ratio. Is specified as an interesting point (an example of an emotional point). Further, the specific unit 133 stores the specified emotion point in the emotion point storage unit 124-1.

In addition, the identification unit 133 identifies emotion points for each user attribute by aggregating the estimation results estimated by the estimation unit 14f for each user attribute of the user. Further, the specific unit 133 stores the specified emotion point in the emotion point storage unit 124-2.

In addition, the identification unit 133 may specify emotion points for each user based on the aggregation result obtained by totaling the estimation results for each user and the attribute information for each user. For example, the identification unit 133 may specify the emotional points of such users by referring to the emotional points of other users whose ages of the users are the same or similar. In addition, the identification unit 133 may specify the emotional points of the user by referring to the emotional points of other users whose genders are the same or similar to each other. In addition, the specific unit 133 may specify the emotional points of the user by referring to the emotional points of other users who have the same or similar interests and hobbies and tastes of the user.

(About presentation unit 134)
The presentation unit 134 presents information about the content based on the estimation result estimated by the estimation unit 14f. For example, when the user browses the content, the presentation unit 134 presents information about the content based on the estimation result estimated for the content. As described with reference to FIG. 2, when the content is video content, the presentation unit 134 is the number of users who have performed emotional expression behavior in the content, and is the time position of the video content. The display is controlled so that the graph showing the transition of the number of action persons changing according to the content is displayed (presented) in association with the time position indicated by the seek bar displayed together with the content. For example, in the example of FIG. 2, the presentation unit 134 performs the processing in steps S13 and S14.

In addition, the presentation unit 134 ranks the content based on the number of users who have performed emotional expression behavior in the content. Then, the presentation unit 134 presents the content in the ranking format based on the given ranking information. For example, the presentation unit 134 presents a predetermined number of high-ranked contents to the user as a popular content ranking. This point will be described with reference to the example of FIG.

In the example of FIG. 9, the total total result storage unit 123 sets the number of actions, which is the number of users who performed a laughing action, out of the total number of users who watched each video content during a predetermined period, for each video content. Store for each time position. Therefore, the presentation unit 134 ranks the contents based on the number of actions stored in the total total result storage unit 123. For example, the presentation unit 134 calculates the ratio of the number of users who have performed a laughing action to the total number of users who have viewed the video content for each time position of the video content. In addition, this calculation may be performed by the totaling unit 132.

Next, the presentation unit 134 extracts the highest ratio from each moving image content. In the example of FIG. 9, the video content VC1 has a time position t2 of “46%”, and the video content VC2 has a time position t1 of “5%”. Then, for example, the presentation unit 134 determines the top five video contents having a higher ratio as the video contents to be presented, and assigns a higher rank to the video contents having a higher ratio. Although not shown in FIG. 9, for convenience of explanation, the presentation unit 134 has video content VC5 “1st place”, video content VC1 “2nd place”, video content VC4 “3rd place”, video content VC2 “4th place”, and the like. It is assumed that the video content VC3 is ranked as "5th place".

Then, the presentation unit 134 presents the five video contents that have been ranked to the user in the form of, for example, "popular video ranking of this week". For example, when the user U1 accesses the video site ST, the presentation unit 134 displays the "popular video ranking of this week" in a predetermined page of the video site ST. If the video content of interest is included in the "popular video ranking of this week", the user U1 can jump to the video viewing page by selecting it.

Further, as another example, the presentation unit 134 can recommend the content corresponding to each user to the user based on the estimation result estimated by the estimation unit 14f. Specifically, the presentation unit 134 is an emotion point specified for the user by aggregating the estimation results estimated by the estimation unit 14f for each user, and the emotion of the user changes in the content. Based on emotional points, which are points, we recommend content according to the user. As an example, the presentation unit 134 recommends the content personalized to the user for each user as a "video list for you" or the like. This point will be described with reference to the example of FIG.

In the example of FIG. 8, the estimation information storage unit 122 stores the time position in which each user performs an emotional expression action for each moving image content. Therefore, the time position stored in the estimated information storage unit 122 can be said to be an emotional point of each user. Therefore, the presentation unit 134 analyzes the content of the video content at this emotion point. Taking the user U1 as an example, the presentation unit 134 analyzes who was the performer at the time position t2 of the video content VC1 and what kind of content the performer was performing. Further, the presentation unit 134 also analyzes the time positions t21 and t51 of the moving image content VC1 in the same manner. Further, based on the analysis result, the presentation unit 134, for example, what kind of genre of video content the user U1 likes, what kind of performer the user U1 likes, and how the user U1 likes it. Learn the tendency of user U1 with respect to video content, such as whether or not he likes acting (for example, comedy material).

Here, as a simple example, it is assumed that the presentation unit 134 has obtained a learning result that "a plurality of groups tend to prefer a program in which a plurality of groups show off their comics" for the user U1. In such a state, it is assumed that the user U1 has accessed the video site ST. In such a case, the presentation unit 134 displays a "video list for you" in a predetermined page of the video site ST. Here, the content distribution device 30 stores various contents to be distributed to the user in the storage unit. Therefore, the presentation unit 134 accesses the storage unit of the content distribution device 30 and selects "a program in which a plurality of groups show their comics" (video content). Then, the presentation unit 134 instructs the content distribution device 30 to distribute the selected moving image content to the user U1. For example, the presentation unit 134 distributes the selected video content so that the selected video content is displayed as a “video list for you”.

As a result, the information processing device 100 can recommend the content that the user U1 is likely to like. As a result, for example, the user U1 can easily watch his / her favorite video content without actively searching for it in a situation where the video content he / she wants to watch is not decided but he / she wants to watch it if there is something interesting. become able to. That is, the information processing device 100 can recommend the content suitable for the user without giving the user a troublesome operation.

(About editorial department 135)
The editorial unit 135 edits the moving image content. Specifically, the editorial unit 135 edits the content based on the emotion points specified by the specific unit 133. Note that editing in this embodiment may include the concept of generating new content. For example, the editorial unit 135 extracts partial content that corresponds to emotional points from the content, and generates new content by combining the extracted partial content. For example, the editorial unit 135 extracts partial content that is content corresponding to each emotional point of the content, and generates new content by combining the extracted partial content. This point will be described with reference to the example of FIG.

In the example of the emotion point storage unit 124-1 shown in FIG. 10, the interesting points of the moving image content VC1 are the time positions t2, t31, and t62. The interesting points of the moving image content VC2 are the time positions t13, t55, and t61.

In this case, the editorial unit 135 extracts the partial content around the time position t2, the partial content around the time position t31, and the partial content around the time position t62 from the moving image content VC1. For example, the editorial unit 135 extracts the moving image content VC1 corresponding to the time positions t0 to t4 as the partial content VC11 around the time position t2. Further, the editorial unit 135 extracts the moving image content VC1 corresponding to the time positions t29 to t33 by VC12 with the partial content around the time position t31. Further, the editorial unit 135 extracts the moving image content VC1 corresponding to the time positions t00 to t64 as the partial content VC13 around the time position t62.

Further, the editorial unit 135 extracts the partial content around the time position t13, the partial content around the time position t55, and the partial content around the time position t61 from the moving image content VC2, respectively. For example, the editorial unit 135 extracts the moving image content VC2 corresponding to the time positions t11 to t15 as the partial content VC21 around the time position t13. Further, the editorial unit 135 extracts the moving image content VC2 corresponding to the time positions t53 to t57 by VC22 with the partial content around the time position t55. Further, the editorial unit 135 extracts the moving image content VC2 corresponding to the time positions t59 to t63 as the partial content VC23 around the time position t61.

Then, the editorial unit 135 generates a new moving image content VC11-21 by combining (connecting) the partial contents VC11, VC12, VC13, VC21, VC22, and VC23 extracted as described above. From this, it can be said that the video content VC11-21 is a video content composed of only interesting points. In addition, the presentation unit 134 may present the moving image content VC11-21 according to the access from the user.

As a result, the information processing device 100 can dynamically generate moving image content that the user can enjoy more. In addition, the information processing device 100 can sell the moving image content VC11-21 to a predetermined business owner. In the above example, the information processing device 100 edits at an interesting point, but the information processing device 100 may perform the same editing at a crying point or a surprise point. Further, the information processing apparatus 100 may edit by interweaving interesting points, crying points, and surprise points.

In addition, the editorial unit 135 may add a mark that abstracts emotions to a predetermined time position of the moving image content. For example, among the video content VC1, "693", which is the largest number of users who watched the video content VC1 during a predetermined period at the time position "t2", performed a laughing action (the number of actions was 693). It is assumed that the aggregated result of is obtained. In this case, the editorial unit 135 may add a mark such as a laughing emoticon at the time position "t2". In this case, the laughing emoticon mark is displayed so as to be superimposed on the moving image content VC1. It is not limited to the above example, and instead of the mark that abstracts the emotion, a laughter, a sound effect, or a character may be added. As described above, the editorial unit 135 may add any information as long as the effect is such that the excitement of the moving image content can be produced. Further, in the above example, laughter has been described as an example of the user's emotions, but the above editing process can also be applied to emotions such as crying and surprise.

[4. Processing procedure]
Next, the procedure of information processing according to the embodiment will be described with reference to FIG. FIG. 12 is a flowchart showing information processing executed by the information processing apparatus 100 according to the embodiment. In the example of FIG. 12, the procedure of information processing performed by the terminal device 10 and the information processing device 100 in cooperation with each other is shown. Further, the terminal device 10 and the information processing device 100 perform information processing by executing the information processing program according to the embodiment. In the example of FIG. 12, the moving image content viewed by the user is referred to as the moving image content VC1.

First, the consent information receiving unit 14b of the terminal device 10 determines whether or not the user has permitted imaging based on the consent information received from the user (step S101). If the user does not permit the imaging (step S101; No), the consent information receiving unit 14b ends the process without imaging the user. On the other hand, the camera control unit 14d determines whether or not the viewing of the video content VC1 has been started when it is determined by the consent information reception unit 14b that the user has permitted the imaging (step S101; Yes) (step S101; Yes). S102). If the viewing of the moving image content VC1 has not been started (step S102; No), the camera control unit 14d waits until the viewing is started. On the other hand, when the viewing of the moving image content VC1 is started (step S102; Yes), the camera control unit 14d takes an image of the user (step S103).

Since the camera control unit 14d continues imaging while the user is viewing the video content VC1, the estimation unit 14f is based on the imaged data (face video data) obtained by the camera control unit 14d. In addition to estimating the emotional expression behavior, an estimation process for estimating a feature amount indicating the degree of the estimated emotional expression behavior is performed (step S104). For example, the estimation unit 14f performs this estimation process every second until the user finishes viewing the video content VC1 in real time when the user is browsing the video content VC1. Then, the transmission unit 14g transmits information including the estimation result by the estimation unit 14f to the information processing apparatus 100 every second (step S105).

When the imaging information is received by the receiving unit 131, the aggregation unit 132 specifies the time position where the emotion expression action is performed in the video content VC1 based on the estimation result estimated by the estimation unit 14f (step S206). ). For example, the aggregation unit 132 specifies a time position in which the feature amount indicates a predetermined threshold value or more as a time position in which the user performs the corresponding emotion expression action in the moving image content VC1.

Next, the aggregation unit 132 performs various aggregations based on the estimation result by the estimation unit 14f and the time position specified in step S206 (step S207). For example, the aggregation unit 132 aggregates the number and proportion of users who have performed a laughing action, and stores the total aggregation result storage unit 123.

Next, the identification unit 133 specifies an emotional point (for example, an interesting point) (step S208). For example, the specific unit 133 refers to the overall aggregation result storage unit 123, and performs an action of laughing at a predetermined ratio or more in the video content or a time position in which a predetermined number of people or more laugh at each video content. Specify the time position as an interesting point.

In such a state, the receiving unit 131 determines whether or not the access from the user has been received (step S209). For example, the receiving unit 131 determines whether or not a distribution request for video content has been received from the user. If the receiving unit 131 has not received the access (step S209; No), the receiving unit 131 waits until it is received. On the other hand, when the access is received by the receiving unit 131 (step S209; Yes), the presenting unit 134 responds to the video content (here, the video content VC2) that the user intends to watch. The content (content to be presented) is generated (step S210). For example, as described with reference to FIG. 2, the presentation unit 134 accesses the overall aggregation result storage unit 123, acquires line information, and shows the transition of the number of actions in the video content VC2 based on the acquired transition information. Generate graph G.

Next, the presentation unit 134 presents the graph G to the user who sends the distribution request (step S211). For example, the information processing device 136 controls the content distribution device 30 so that the graph G is displayed on the seek bar BR where the user can control the playback location (time position) of the moving image content VC2. For example, the information processing device 100 instructs the content distribution device 30 to distribute the graph G to the terminal device 10 so that the terminal device 10 displays the graph G on the seek bar BR. Further, the information processing device 100 transmits the graph G to the content distribution device 30.

[5. Modification example]
The terminal device 10 and the information processing device 100 according to the above embodiment may be implemented in various different forms other than the above embodiment. Therefore, in the following, other embodiments of the terminal device 10 and the information processing device 100 will be described.

[5-1. content〕
In the above embodiment, an example has been shown in which the content to be processed by the information processing apparatus 100 is a moving image content, but the content to be processed may be an image content. That is, the information processing device 100 according to the embodiment refers to the user who is viewing the image content based on the facial expression of the user indicated by the image pickup information captured by the image pickup means included in the terminal device 10 displaying the image content. Get information about the estimated user's emotions. Then, the information processing device 100 identifies the emotional points, which are the points at which the user's emotions change in the image content, by aggregating the acquired estimation results.

On the other hand, when the content is an image content (for example, an electronic book), the information processing device 100 specifies a page of the image content whose numerical value based on the number of active persons satisfies a predetermined condition information as an emotion point. ..

In this case, the aggregation unit 132 aggregates the number of people who performed the laughing action in each image content for each page of each image content. For example, the aggregation unit 132 aggregates the number of active people using the information stored in the estimation information storage unit 122. In addition, the aggregation unit 132 aggregates the ratio of the number of users who have performed a laughing action in the image content to the total number of users who have viewed the image content for each page of the image content during a predetermined period. .. For example, the aggregation unit 132 aggregates the ratio of the number of users by using the information stored in the estimation information storage unit 122. Further, the totaling unit 132 stores the totaled total result in the total totaled result storage unit 123.

In addition, the specific unit 133 specifies a point at which the numerical value based on the number of active persons satisfies the predetermined condition information as an emotional point. Specifically, the specific unit 133 specifies, among the pages of the image content, the page whose numerical value based on the number of active persons satisfies the predetermined condition information as the emotion point. For example, the specific unit 133 specifies, as an interesting point, a page in which the number of users who have performed a laughing action, which is the number of users who have performed a laughing action, is equal to or greater than the predetermined number of users who have viewed the corresponding image content during a predetermined period. .. In addition, the specific unit 133 displays a page in which the ratio of the number of users who have performed a laughing action to the total number of users who have viewed the corresponding image content during a predetermined period is equal to or greater than the predetermined ratio. Identify as an interesting point.

As a result, the information processing device 100 can provide information that is meaningful to the user in response to changes in emotions that occur in the user by browsing the content. For example, the information processing device 100 provides new image content that is a collection of only interesting point pages, or presents pages that are points of laughter in a graph when a user browses and uses the image content. be able to.

[5-2. Terminal equipment]
In the above embodiment, the terminal device 10 has shown an example of estimating information about emotions, but the estimation process executed by the terminal device 10 may be performed on the information processing device 100 side. On the other hand, in addition to performing the estimation process, the terminal device 10 may perform a total process performed by the total unit 132 executed by the information processing device 100, a specific process performed by the specific unit 133, and the like.

[5-3. Estimate concentration ratio]
In the above embodiment, the estimation unit 14f has shown an example in which the estimation unit 14f performs an estimation process for estimating the emotion expression behavior and the feature amount of the emotion expression behavior as information on the user's emotion. The degree of concentration of the user on the content may be estimated as information on the user's emotions based on the facial expression of the user. For example, a user performs emotional expression behaviors such as laughing, crying, and surprise because he / she concentrates on viewing video content. Therefore, the degree of concentration of the user on the content can be said to be information on the user's emotions. The estimation unit 14f can estimate the degree of concentration (an index value indicating the degree of concentration) by using the same method as the estimation process described so far. Hereinafter, this example will be described as appropriate with reference to the example of FIG.

For example, the estimation unit 14f analyzes the facial expression of the facial expression data to analyze the facial expression of the user and the video content VC1 (or the screen of the terminal device 10 on which the video content VC1 is displayed) at which time position of the video content VC1. Judge and measure the viewing mode, such as whether you paid attention to. Then, the estimation unit 14f estimates the degree of concentration based on this result, for example, every second.

Further, the transmission unit 14g transmits information including the estimation result (concentration ratio) to the information processing apparatus 100, for example, every second. As an example, the transmission unit 14g transmits information including a time position (time code) corresponding to the reproduction time of the moving image content and a degree of concentration to the information processing apparatus 100 every second. That is, the transmission unit 14g sequentially transmits information including the time position (time code) and the degree of concentration to the information processing device 100 while the user is browsing the moving image content. For example, the transmission unit 14g transmits information such as the time position "1 minute 53 seconds" and the concentration degree "10" to the information processing device 100. Further, for example, the transmission unit 14g transmits information such as the time position "1 minute 54 seconds" and the concentration degree "8" to the information processing device 100. Further, for example, the transmission unit 14g transmits information such as the time position "1 minute 55 seconds" and the concentration degree "7" to the information processing device 100.

Here, assuming that the video content VC1 is an advertising video, the information processing device 100 can measure the advertising effect based on the concentration level received from the terminal device 10, so how is it based on the measured advertising effect? It is possible to analyze whether the advertisement delivery is good and to feed back the analysis result to the advertiser.

[5-4. Content distribution device]
In the above embodiment, the content distribution device 30 has shown an example of distributing various contents, but the information processing device 100 may perform the content distribution by having the function of the content distribution device 30. In this case, the information processing device 100 has a storage unit that stores various contents received from the business owner (for example, a content provider).

[5-5. Voice information]
In the above embodiment, the information processing device 100 shows an example of estimating information about the user's emotions based on the user's facial expression indicated by the imaging information. However, the information processing device 100 is a sound collecting means included in the terminal device 10. The voice information collected by the (for example, a microphone) may be acquired, and the information regarding the user's emotion may be estimated based on the acquired voice information.

For example, the information processing device 100 acquires the laughter of the user collected by the microphone included in the terminal device 10. Then, since the information processing device 100 has acquired the laughter of the user, the information regarding the emotion of the user may be presumed to be "laughter". At this time, the information processing device 100 analyzes the user's voice information by using a conventional technique such as voice analysis.

The modified example is not limited to the above example. For example, the information processing device 100 may acquire voice information collected by the sound collecting means of the terminal device 10 and combine the acquired voice information with the imaging information to estimate information on the user's emotions. ..

Further, the information processing device 100 is not limited to voice information, for example, sensing information acquired from a gyro sensor and an acceleration sensor that detect the movement of the terminal device 10, a user's heart rate, a user's body temperature, and the like. Information on the user's emotions may be estimated based on the sensing information on the biometric information.

[6. Hardware configuration]
Further, the terminal device 10, the content distribution device 30, and the information processing device 100 according to the above-described embodiment are realized by, for example, a computer 1000 having a configuration as shown in FIG. Hereinafter, the information processing apparatus 100 will be described as an example. FIG. 13 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores a program executed by the CPU 1100, data used by such a program, and the like. The communication interface 1500 receives data from another device via the communication network 50 and sends it to the CPU 1100, and transmits the data generated by the CPU 1100 to the other device via the communication network 50.

The CPU 1100 controls an output device such as a display or a printer, and an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Further, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides the program or data to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the function of the control unit 130 by executing the program loaded on the RAM 1200. Further, the data in the storage unit 120 is stored in the HDD 1400. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be acquired from another device via the communication network 50.

Further, for example, when the computer 1000 functions as the terminal device 10, the CPU 1100 of the computer 1000 realizes the function of the control unit 14 by executing the program loaded on the RAM 1200.

[7. others〕
Of the processes described in each of the above embodiments, all or part of the processes described as being automatically performed can be performed manually, or all the processes described as being performed manually. Alternatively, a part thereof can be automatically performed by a known method. In addition, the processing procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically dispersed / physically distributed in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

In addition, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the estimation unit can be read as an estimation means or an estimation circuit.

[8. effect〕
As described above, the information processing apparatus 100 according to the embodiment includes a receiving unit 131 (an example of an acquisition unit) and a specific unit 133. The receiving unit 131 acquires information on the user's emotions estimated based on the user's facial expression indicated by the imaging information captured by the imaging means included in the terminal device 10 displaying the content for the user who is viewing the content. do. The identification unit 133 identifies an emotion point, which is a point in which the user's emotion changes in the content, by aggregating the estimation results acquired by the reception unit 131.

As a result, the information processing device 100 according to the embodiment identifies the emotional points, which are the points at which the user's emotions change in the content, by aggregating the estimation results. It is possible to provide meaningful information to the user in response to the emotional change that occurs in.

Further, in the information processing apparatus 100 according to the embodiment, the receiving unit 131 acquires information on the user's emotions estimated in real time based on the facial expression of the user, and the specific unit 133 acquires the information on the user's emotions in real time by the receiving unit 131. Emotion points are specified in real time for each user by aggregating the estimated results for each user.

As a result, the information processing device 100 according to the embodiment aggregates the estimation results acquired in real time for each user to identify emotion points for each user in real time. It is possible to provide meaningful information to the user in real time in response to changes in emotions.

Further, in the information processing apparatus 100 according to the embodiment, the receiving unit 131 acquires the attribute information related to the user's attributes, and the specific unit 133 collects the estimation results acquired by the receiving unit 131 for each user. , The emotion point is specified for each user based on the attribute information for each user.

As a result, the information processing apparatus 100 according to the embodiment obtains content in order to identify emotion points for each user based on the aggregated result obtained by totaling the acquired estimation results for each user and the attribute information for each user. It is possible to provide meaningful information to the user according to the change in emotions caused to the user by viewing.

Further, in the information processing device 100 according to the embodiment, the identification unit 133 identifies emotion points by totaling the number of actions, which is the number of users who have performed emotion expression actions in the content.

As a result, the information processing device 100 according to the embodiment can view the content in order to identify the emotion point by totaling the number of users who have performed the emotion expression action in the content. It is possible to provide information that is meaningful to the user in response to changes in emotions that occur in the user.

Further, in the information processing device 100 according to the embodiment, the specific unit 133 specifies a point at which the numerical value based on the number of active persons satisfies the predetermined condition information as an emotional point.

As a result, the information processing device 100 according to the embodiment identifies the point at which the numerical value based on the number of active persons satisfies the predetermined condition information as the emotion point, and therefore responds to the change in emotion caused by the user by viewing the content. Therefore, it is possible to provide meaningful information to the user.

Further, in the information processing apparatus 100 according to the embodiment, when the content is a moving image content, the specific unit 133 sets a time position in the playback time of the moving image content in which a numerical value based on the number of active persons satisfies a predetermined condition information. Identify as an emotional point.

As a result, when the content is video content, the information processing device 100 according to the embodiment identifies as an emotion point a time position in which the numerical value based on the number of active persons satisfies predetermined condition information in the playback time of the video content. Therefore, it is possible to provide information that is meaningful to the user in response to changes in emotions that occur in the user by viewing the content.

Further, in the information processing device 100 according to the embodiment, when the content is an image content, the specific unit 133 points out the page of the image content in which the numerical value based on the number of active persons satisfies the predetermined condition information. Identify as.

As a result, when the content is image content, the information processing device 100 according to the embodiment identifies the page of the image content whose numerical value based on the number of active persons satisfies the predetermined condition information as an emotion point. , It is possible to provide meaningful information to the user according to the change in emotions caused to the user by viewing the content.

Further, the information processing apparatus 100 according to the embodiment further includes an editorial unit 135 that edits the content based on the emotional points.

As a result, the information processing device 100 according to the embodiment edits the content based on the emotional points, and therefore provides meaningful information to the user according to the change in the emotion generated by the user by viewing the content. can do.

Further, in the information processing apparatus 100 according to the embodiment, the editorial unit 135 extracts partial content which is the content corresponding to the emotional point from the content, and generates new content by combining the extracted partial content.

As a result, the information processing apparatus 100 according to the embodiment extracts the partial content which is the content corresponding to the emotional point from the content, and generates new content by combining the extracted partial content, so that the content is viewed. As a result, it is possible to provide meaningful information to the user in response to changes in emotions that occur in the user.

Further, in the information processing apparatus 100 according to the embodiment, the editorial unit 135 extracts partial content which is the content corresponding to each emotional point of the content, and generates new content by combining the extracted partial content.

As a result, the information processing device 100 according to the embodiment extracts the partial content which is the content corresponding to the emotional point of each content, and generates new content by combining the extracted partial content, so that the content is viewed. It is possible to provide meaningful information to the user according to the change in emotions caused by the user.

Further, the information processing apparatus 100 according to the embodiment further includes a presenting unit 134 that presents information about the content based on the estimation result acquired by the receiving unit 131.

As a result, since the information processing device 100 according to the embodiment presents information about the content based on the acquired estimation result, it is meaningful to the user according to the change in emotions caused to the user by viewing the content. Information can be provided.

Further, in the information processing apparatus 100 according to the embodiment, when the user browses the content, the presentation unit 134 presents information about the content based on the estimation result estimated for the content.

As a result, when the user browses the content, the information processing device 100 according to the embodiment presents information about the content based on the estimation result estimated for the content, and therefore, the emotion generated by the user by viewing the content. It is possible to provide meaningful information to the user according to the change of.

Further, in the information processing apparatus 100 according to the embodiment, when the content is video content, the presentation unit 134 is the number of users who have performed emotional expression behavior in the content, and is the number of people who acted. The display is controlled so that the graph showing the transition of the number of actions changing according to the time position of the content is displayed in association with the time position indicated by the seek bar displayed together with the content.

As a result, when the content is video content, the information processing device 100 according to the embodiment is the number of users who have performed emotional expression actions in the content, and is the number of people who act, and the time position of the video content. Since the display control is performed so that the graph showing the transition of the number of actions that changes according to the content is displayed in association with the time position indicated by the seek bar displayed together with the content, the emotional change caused to the user by viewing the content. Depending on the situation, it is possible to provide information that is meaningful to the user.

Further, in the information processing device 100 according to the embodiment, the presentation unit 134 ranks based on the ranking information ranked in the content based on the number of actions, which is the number of users who have performed emotional expression actions in the content. Present the content in a format.

As a result, the information processing device 100 according to the embodiment displays the content in a ranking format based on the ranking information ranked in the content based on the number of actions, which is the number of users who have performed emotional expression actions in the content. Since it is presented, it is possible to provide information that is meaningful to the user in response to changes in emotions that occur in the user by viewing the content.

Further, in the information processing apparatus 100 according to the embodiment, the presenting unit 134 recommends the content according to the user based on the estimation result acquired by the receiving unit 131.

As a result, the information processing device 100 according to the embodiment recommends the content according to the user based on the acquired estimation result, so that the user responds to the change in emotions caused by the user by viewing the content. It is possible to provide meaningful information for the user.

Further, in the information processing device 100 according to the embodiment, the presenting unit 134 is an emotion point specified for the user by totaling the estimation results acquired by the receiving unit 131 for each user, and is a user in the content. Based on the emotional points, which are the points at which the emotions of the user change, the content according to the user is recommended.

As a result, the information processing device 100 according to the embodiment is an emotion point specified for the user by totaling the acquired estimation results for each user, and is a point at which the user's emotion changes in the content. Since the content according to the user is recommended based on the emotional point, it is possible to provide meaningful information to the user according to the change in emotion caused to the user by viewing the content.

Further, the terminal device 10 according to the embodiment includes an acquisition unit 14e, an estimation unit 14f, and a transmission unit 14g. The acquisition unit 14e acquires the imaging information obtained by imaging the user who is viewing the content by the imaging means. The estimation unit 14f estimates information about the user's emotions based on the facial expression of the user indicated by the imaging information acquired by the acquisition unit 14e. The transmission unit 14g transmits the estimation result estimated by the estimation unit 14f to the information processing device 100.

As a result, the terminal device 10 according to the embodiment transmits the estimation result regarding the user's emotions estimated based on the user's facial expression indicated by the acquired imaging information to the information processing device 100, so that the content can be viewed. It is possible to provide information that is meaningful to the user in response to changes in emotions that occur in the user.

The embodiments of the present application have been described in detail with reference to some drawings, but these are examples, and various modifications are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to practice the present invention in other improved forms.

1 Information processing system 10 Terminal device 12 Display unit 13 Imaging unit 14 Control unit 14a Request unit 14b Consent information reception unit 14c Display control unit 14d Camera control unit 14e Acquisition unit 14f Estimate unit 14g Transmission unit 30 Content distribution device 100 Information processing device 120 Storage unit 121 Imaging information storage unit 122 Estimated information storage unit 123 Overall total result storage unit 124 Emotion point storage unit 125 Performer information storage unit 130 Control unit 131 Reception unit 132 Total unit 133 Specific unit 134 Presentation unit 135 Editorial unit

Claims (19)

An acquisition unit that acquires information about the user's emotions estimated based on the user's facial expression indicated by the image pickup information captured by the image pickup means of the terminal device displaying the content for the user who is viewing the content. When,
An information processing device including a specific unit that identifies an emotional point, which is a point at which the user's emotion changes in the content, by aggregating the estimation results acquired by the acquisition unit. .. The acquisition unit
Based on the facial expression of the user, information on the emotion of the user estimated in real time is acquired, and the information is obtained.
The specific part is
The information processing apparatus according to claim 1, wherein the emotion points are specified in real time for each user by aggregating the estimation results acquired in real time by the acquisition unit for each user. The acquisition unit
Acquire attribute information related to the user's attribute,
The specific part is
The second aspect of claim 2 is characterized in that the emotion point is specified for each user based on the aggregation result obtained by the acquisition unit for each user and the attribute information for each user. The information processing device described. The specific part is
The information according to any one of claims 1 to 3, wherein the emotional points are specified by aggregating the number of users who have performed emotional expression behavior in the content. Processing equipment. The specific part is
The information processing device according to claim 4, wherein a point at which a numerical value based on the number of active persons satisfies a predetermined condition information is specified as the emotion point. The specific part is
The fifth aspect of the fifth aspect is that when the content is a video content, the time position where the numerical value based on the number of active persons satisfies the predetermined condition information is specified as the emotion point in the playback time of the video content. The information processing device described. The specific part is
When the content is image content, claim 5 or 6 is characterized in that, among the pages of the image content, the page whose numerical value based on the number of active persons satisfies the predetermined condition information is specified as the emotion point. The information processing device described. The information processing apparatus according to any one of claims 1 to 7, further comprising an editorial unit that edits the content based on the emotional point. The editorial department
The information processing apparatus according to claim 8, wherein the partial content, which is the content corresponding to the emotional point, is extracted from the content, and new content is generated by combining the extracted partial content. The editorial department
The information processing apparatus according to claim 9, wherein partial content, which is content corresponding to the emotional point of each of the contents, is extracted, and new content is generated by combining the extracted partial contents. The information processing apparatus according to any one of claims 1 to 10, further comprising a presenting unit that presents information about the content based on the estimation result acquired by the acquiring unit. The presentation unit
The information processing device according to claim 11, wherein when the user browses the content, information about the content is presented based on an estimation result estimated for the content. The presentation unit
When the content is video content, it is the number of actions that is the number of users who have performed emotional expression actions in the content, and indicates the transition of the number of actions that changes according to the time position of the video content. The information processing apparatus according to claim 12, wherein the graph is displayed and controlled so as to be displayed in association with the time position indicated by the seek bar displayed together with the content. The presentation unit
Claims 11 to 11 characterized in that the content is presented in a ranking format based on the ranking information ranked in the content based on the number of users who have performed emotional expression behavior in the content. The information processing apparatus according to any one of 13. The presentation unit
The information processing apparatus according to any one of claims 11 to 14, wherein the content according to the user is recommended based on the estimation result acquired by the acquisition unit. The presentation unit
Emotion points that are identified for the user by aggregating the estimation results acquired by the acquisition unit for each user, and are points at which the user's emotions change in the content. The information processing apparatus according to claim 15, wherein the content according to the user is recommended based on the information processing apparatus. It is an information processing method executed by a computer.
An acquisition step of acquiring information on the user's emotions estimated based on the facial expression of the user indicated by the imaging information captured by the imaging means of the terminal device displaying the content for the user who is viewing the content. When,
An information processing method including a specific step of identifying emotion points, which are points in which the user's emotions change in the content, by aggregating the estimation results acquired by the acquisition process. .. Acquisition procedure for acquiring information on the user's emotions estimated based on the facial expression of the user indicated by the imaging information captured by the imaging means of the terminal device displaying the content. When,
By aggregating the estimation results acquired by the acquisition procedure, the computer is made to execute a specific procedure for identifying emotion points, which are points where the user's emotions have changed in the content. Information processing program. An acquisition unit that acquires imaging information obtained by imaging a user who is viewing content by an imaging means, and an acquisition unit.
An estimation unit that estimates information about the user's emotions based on the facial expression of the user indicated by the imaging information acquired by the acquisition unit, and an estimation unit.
A terminal device including a transmission unit that transmits an estimation result estimated by the estimation unit to an information processing device.

Images