JP2007034664A - Emotion estimation device and method, recording medium and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To further accurately estimate a user's emotion without increase in the user's burden. <P>SOLUTION: A learning part 103 learns, when stimulation information is presented during learning processing, an emotion determination model showing the relation between a characteristic vector showing the user's state supplied from a sensor part and the user's interest based on user instruction information intentionally inputted by the user, the characteristic vector, and an emotion map mutually relating the instruction content and interest of the user. After the learning processing, an estimation part 104 estimates the user's emotion to the stimulation information based on the characteristic vector and the emotion determination model. This invention is applicable to a recommendation system for recommending contents to users. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an emotion estimation apparatus and method, a recording medium, and a program, and more particularly, to an information processing system, an emotion estimation apparatus and method, a recording medium, and a program that can estimate a user's emotion more accurately.

  In recent years, a system (hereinafter referred to as a recommendation system) that estimates the degree of interest and interest of a user and recommends content, products, etc. that the user is likely to be interested based on the estimation result has become widespread.

  As a method of estimating the degree of interest and interest of a user that is generally used in the recommendation system, a method that directly inquires the user about the interest or interest in the target and uses the user's answer to that, that is, explicit There are a technique that uses feedback (Explicit Feedback) and a technique that uses a device operation performed by a user when an object is presented, that is, a technique that uses implicit feedback.

  One system that uses explicit feedback is a dialog system that interacts with a user based on a scenario prepared in advance and estimates the user's interest and degree of interest based on what the user utters. .

  Also, in one of the systems using implicit feedback, the user's biological information is detected, the emotional state of the user is analyzed based on the biological information, and the content suitable for the user is recommended based on the emotional state of the user The system which performs is proposed (for example, refer patent document 1).

JP 2003-92701 A

  However, in the method using explicit feedback, the user must show an explicit response to the system query, which places a burden on the user. For example, in a dialogue system, a user must answer a system query in a clear and recognizable voice. If the user is not interested and continues to respond, the system may not be able to recognize the user's reaction and may continue to interact.

  In the method using implicit feedback, for example, the user's interest and interest cannot be estimated unless the user operates any device. In addition, if the target to be presented to the user is video content, it is not known whether the user is actually viewing the video content, and thus there is a case where the user learns incorrectly.

  Further, in the invention described in Patent Document 1, an emotional state is analyzed based on a rule that has been created in advance using a common model for all users, and corresponds to individual differences such as biological information. This is not taken into consideration. Further, in the invention described in Patent Document 1, it is not proposed to estimate the user's interest or degree of interest.

  The present invention has been made in view of such a situation, and makes it possible to estimate a user's emotion more accurately without increasing the burden on the user.

  The emotion estimation apparatus according to the first aspect of the present invention includes a feature amount acquisition unit that acquires feature amount data indicating a feature amount of a user's state that changes due to the expression of emotion, and when a stimulus is given to the user. Input information acquisition means for acquiring input information intentionally input by the user, information indicating the user's emotion with respect to the stimulus predicted by the input information input by the user, and the input information are associated in advance. Based on the emotion data set and the feature data acquired when the user inputs the input information when the user is given the first stimulus as the stimulus, Learning means for learning a model indicating a relationship between the feature quantity data and the user's emotion; the feature quantity data acquired when a second stimulus is given; and Based on Dell, including an estimation unit for estimating the user's feelings when given the second stimulation.

  The input information may be information indicating a processing command to a device that gives the first stimulus to the user.

  In the emotion data, information indicating the degree of interest of the user with respect to the stimulus predicted by the input information input by the user and the input information may be set in advance in association with each other. .

  Detection means for detecting a feature value of the user's state and outputting the feature value data may be further provided.

  The stimulus given to the user may be information presented to the user.

  Information presentation control means for controlling presentation of information recommended to the user can be further provided based on the estimated emotion of the user.

  The emotion estimation method or program according to the first aspect of the present invention acquires feature amount data indicating a feature amount of a user's state that changes due to the expression of an emotion, and when the user is given a stimulus, Information indicating intention of the user is acquired, and information indicating the user's emotion with respect to the stimulus predicted by the input information input by the user and emotion data in which the input information is set in advance in association with each other And when the user receives the first stimulus as the stimulus, based on the feature amount data acquired when the user inputs the input information, the feature amount data and the user's Based on the feature amount data acquired when the model indicating the relationship with emotion is learned and the second stimulus is given, and the model, the second stimulus is Comprising the step of estimating the emotion of the user when was e.

  An emotion estimation apparatus according to a second aspect of the present invention includes a feature amount acquisition unit that acquires feature amount data indicating a feature amount of a user's state that changes due to expression of emotion, and the user's emotion with respect to a first stimulus. Based on emotion data that is set in advance in association with information indicating and information indicating the first stimulus, and the feature amount data acquired when the first stimulus is given to the user, Based on the learning means for learning a model indicating the relationship between the feature quantity data and the user's emotion, the feature quantity data acquired when a second stimulus is given, and the model, the second Estimating means for estimating the user's emotion when the stimulus is applied.

  In the emotion data, information indicating the degree of interest of the user with respect to the first stimulus and information indicating the first stimulus may be set in advance in association with each other.

  Detection means for detecting a feature value of the user's state and outputting the feature value data may be further provided.

  The first stimulus or the second stimulus given to the user may be information presented to the user.

  Information presentation control means for controlling presentation of information recommended to the user can be further provided based on the estimated emotion of the user.

  The emotion estimation method or program according to the second aspect of the present invention acquires feature amount data indicating a feature amount of a user's state that changes due to the expression of emotion, information indicating the user's emotion with respect to a first stimulus, and The feature amount based on emotion data in which information indicating the first stimulus is set in advance in association with the feature amount data acquired when the first stimulus is given to the user When learning the model indicating the relationship between the data and the emotion of the user, and applying the second stimulus based on the feature amount data acquired when the second stimulus is applied and the model Estimating the user's emotions.

  In the first aspect of the present invention, feature amount data indicating a feature amount of a user's state that changes due to the expression of emotion is acquired, and when a stimulus is given to the user, the feature amount data is intentionally input by the user. Information indicating the user's emotion with respect to the stimulus predicted by the input information input by the user and emotion data in which the input information is set in advance in association with the user. When the first stimulus, which is the stimulus, is given, the relationship between the feature amount data and the user's emotion is obtained based on the feature amount data acquired when the user inputs the input information. When the model shown is learned and the second stimulus is given, the feature amount data obtained when the second stimulus is given, and the second stimulus is given based on the model Feelings over The is estimated.

  In the second aspect of the present invention, feature amount data indicating a feature amount of a user's state that changes due to expression of emotion is acquired, information indicating the user's emotion with respect to the first stimulus, and the first stimulus On the basis of emotion data that is set in advance in association with information indicating the above and the feature amount data acquired when the first stimulus is given to the user. The feature amount data acquired when a model showing a relationship with emotion is learned and a second stimulus is given, and the user's emotion when the second stimulus is given based on the model Is estimated

  As described above, according to the first aspect or the second aspect of the present invention, the user's emotion can be estimated. Moreover, according to the 1st side surface or the 2nd side surface of this invention, a user's emotion can be estimated more correctly, without increasing a user's burden.

  Embodiments of the present invention will be described below. Correspondences between constituent elements described in the claims and specific examples in the embodiments of the present invention are exemplified as follows. This description is to confirm that specific examples supporting the invention described in the claims are described in the embodiments of the invention. Therefore, even if there are specific examples that are described in the embodiment of the invention but are not described here as corresponding to the configuration requirements, the specific examples are not included in the configuration. It does not mean that it does not correspond to a requirement. On the contrary, even if a specific example is described here as corresponding to a configuration requirement, this means that the specific example does not correspond to a configuration requirement other than the configuration requirement. not.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, in the future, a divisional application will be made. Nor does it deny the existence of an invention added by amendment.

  The emotion estimation apparatus according to the first aspect of the present invention (for example, the content recommendation control unit 13 in FIG. 1) uses feature amount data (for example, a feature vector) indicating the feature amount of the user's state that changes due to the expression of emotion. A feature amount acquisition means to acquire (for example, the feature amount acquisition unit 102 in FIG. 2) and input information (for example, user command information) intentionally input by the user when a stimulus is given to the user Input information acquisition means (for example, the input information acquisition unit 101 in FIG. 2), information indicating the user's feelings with respect to the stimulus predicted by the input information input by the user, and the input information are associated in advance. Emotion data (for example, the emotion map in FIG. 3 or FIG. 4) and the user is given the first stimulus as the stimulus, the user Learning means for learning a model (for example, an emotion discrimination model in FIG. 5) indicating a relationship between the feature amount data and the user's emotion based on the feature amount data acquired when force information is input. For example, based on the learning unit 103), the feature amount data acquired when the second stimulus is given, and the model, the emotion of the user when the second stimulus is given is estimated. And estimation means (for example, the estimation unit 104).

  In the emotion estimation apparatus according to the first aspect of the present invention, the emotion data includes information indicating a degree of interest of the user with respect to the stimulus predicted by the input information input by the user (for example, degree of interest). ) And the input information may be set in advance in association with each other.

  The emotion estimation apparatus according to the first aspect of the present invention further includes detection means (for example, the sensor unit 41 in FIG. 1) that detects a feature amount of the user's state and outputs the feature amount data. Can do.

  The emotion estimation apparatus according to the first aspect of the present invention is an information presentation control means for controlling presentation of information recommended to the user based on the estimated emotion of the user (for example, the recommendation processing control unit 43 in FIG. 1). ).

  The emotion estimation method or program according to the first aspect of the present invention acquires feature amount data (for example, a feature vector) indicating a feature amount of a user's state that changes due to the expression of emotion (for example, step S41 in FIG. 8). ) When a stimulus is given to the user, input information (for example, user command information) intentionally input by the user is acquired (for example, step S42 in FIG. 8), and the input by the user is performed. Information indicating the user's emotion with respect to the stimulus predicted by the input information and emotion data (for example, the emotion map of FIG. 3 or FIG. 4) set in association with the input information, and the stimulus to the user When the first stimulus is given, the feature amount data acquired when the user inputs the input information (for example, emotion discrimination in FIG. 5) Based on (Dell), a model indicating the relationship between the feature quantity data and the user's emotion is learned (for example, steps S43 and S44 in FIG. 8 and step S61 in FIG. 9). Including estimating the user's emotion when the second stimulus is given based on the feature amount data acquired when given and the model (for example, steps S102 to S105 in FIG. 10) .

  The emotion estimation apparatus according to the second aspect of the present invention (for example, the content recommendation control unit 13 in FIG. 1) uses feature amount data (for example, a feature vector) indicating the feature amount of the user's state that changes due to the expression of emotion. Feature amount acquisition means (for example, the feature amount acquisition unit 102 in FIG. 2) to be acquired, information indicating the user's emotion with respect to the first stimulus, and information indicating the first stimulus are set in advance in association with each other. Based on emotion data (for example, the emotion map of FIG. 11) and the feature data acquired when the user is given the first stimulus, the feature data and the user's emotion Learning means (for example, the learning unit 103) for learning a model indicating the relationship (for example, the emotion discrimination model in FIG. 5), the feature amount data acquired when the second stimulus is given, and the model Based on the estimation means estimates the emotion of the user of when given the second stimulus, including, for example, the estimation unit 104) and.

  In the emotion estimation apparatus according to the second aspect of the present invention, the emotion data includes information (for example, degree of interest) indicating the degree of interest of the user with respect to the stimulus with respect to the first stimulus and the first stimulus. Can be set in advance in association with each other.

  The emotion estimation apparatus according to the second aspect of the present invention further includes detection means (for example, the sensor unit 41 in FIG. 1) that detects a feature amount of the user's state and outputs the feature amount data. Can do.

  The emotion estimation apparatus according to the second aspect of the present invention is an information presentation control means for controlling presentation of information recommended to the user based on the estimated emotion of the user (for example, the recommendation processing control unit 43 in FIG. 1). ).

  The emotion estimation method or program according to the second aspect of the present invention acquires feature amount data (for example, feature vector) indicating the feature amount of the user's state that changes due to the expression of emotion (for example, step S42 in FIG. 8). ), Emotion data (for example, the emotion map of FIG. 11) in which information indicating the user's emotion with respect to the first stimulus and information indicating the first stimulus are set in advance in association with each other, Based on the feature amount data acquired when one stimulus is given, a model (for example, an emotion discrimination model in FIG. 5) showing the relationship between the feature amount data and the user's emotion is learned (for example, For example, based on steps S223 and S224 in FIG. 13 and step S61 in FIG. 9, the feature amount data acquired when the second stimulus is given, and the model , Including the said estimation of the user feelings when given a second stimulus (e.g., step S102 to S105 in FIG. 10) steps.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing an embodiment of a content recommendation system 1 to which the present invention is applied.

  The content recommendation system 1 is configured to include a content presentation control unit 11, a remote controller 12, an output unit 14, and a content recommendation control unit 13.

  The content presentation control unit 11 controls the presentation of content to the user based on a processing command input by the user by operating the remote controller 12 or a command from the content recommendation control unit 13.

  In this specification, the content is, for example, a television broadcast program, a movie, a photo, a song, etc. (moving image, still image, audio, or a combination thereof) generally called content. Etc.).

  The content presentation control unit 11 includes a content reception unit 21, a content storage unit 23, a meta information reception unit 22, a meta information storage unit 24, a meta information extraction unit 25, a content reproduction unit 26, a content information presentation unit 27, and a user input The acquisition unit 28 is configured to be included.

  The content receiving unit 21 is, for example, from a server connected to a network content presentation control unit 11 such as the Internet (not shown) or an external information processing device (not shown) connected to the content presentation control unit 11. Receive content such as video content and audio content. The content receiving unit 21 causes the content storage unit 23 to store the received content. In addition, when the received content includes meta information that is information related to the content, the content receiving unit 21 supplies the meta information to the meta information receiving unit 22.

  The meta information receiving unit 22 displays meta information corresponding to the content received by the content receiving unit 21, for example, a server connected to the content presentation control unit 11 via a network such as the Internet (not shown), or content presentation Received from an external information processing apparatus (not shown) connected to the control unit 11. The meta information receiving unit 22 causes the meta information accumulating unit 24 to accumulate the received meta information or the meta information acquired from the content receiving unit 21.

  The content storage unit 23 and the meta information storage unit 24 are configured by a recording medium such as a hard disk, for example. The content storage unit 23 and the meta information storage unit 24 may be configured by one recording medium, or may be configured by different recording media.

  The meta information extraction unit 25 reads the meta information requested from the content reproduction unit 26 or the content information presentation unit 27 from the meta information storage unit 24, and supplies the read meta information to the content reproduction unit 26 or the content information presentation unit 27. .

  The content playback unit 26 reads the content stored in the content storage unit 23 based on a processing command input by the user by operating the remote controller 12 or a command from the recommendation processing control unit 43. Play the read content. The content reproduction unit 26 supplies image data or audio data obtained by reproducing the content to the output unit 14. Further, the content information presentation unit 27 supplies the meta information supplied from the meta information extraction unit 25 to the emotion estimation unit 42 as necessary.

  The content information presenting unit 27 performs a meta of content stored in the meta information accumulating unit 24 via the meta information extracting unit 25 based on a processing command input by the user by operating the remote controller 12. Information is read, and based on the read meta information, image data or audio data for presenting recommended information for recommending content to the user is generated. The content information presentation unit 27 supplies the generated image data or audio data to the output unit 14. In addition, the content information presentation unit 27 supplies information notifying that the recommendation information has been presented to the content reproduction unit 26. Furthermore, the content information presentation unit 27 supplies meta information that is the basis of the recommendation information to the emotion estimation unit 42 as necessary.

  The user input acquisition unit 28 intentionally inputs a user operation by operating the remote controller 12, and indicates information (hereinafter referred to as user command information) indicating a processing command to the content presentation control unit 11. Get from. The user input acquisition unit 28 supplies user command information to the emotion estimation unit 42, the content reproduction unit 26, or the content information presentation unit 27 as necessary.

  As will be described later with reference to FIGS. 6 to 10, the content recommendation control unit 13 expresses the user's feelings regarding the stimulus given to the user (for example, the content presented on the output unit 14 or recommendation information). presume. Based on the estimation result, the content recommendation control unit 13 controls the content reproduction unit 26 to control the presentation of content, which is information recommended to the user, or the reproduction or stop of the content. The content recommendation control unit 13 is configured to include a sensor unit 41, an emotion estimation unit 42, and a recommendation processing control unit 43.

  Hereinafter, the content or recommendation information presented on the output unit 14 serving as a stimulus for estimating the user's emotion is also referred to as stimulus information as appropriate. Further, in the following description, an example will be described in which the degree of interest indicating the degree of interest of the user with respect to stimulus information is mainly estimated.

  The sensor unit 41 detects the feature amount of the user that changes due to the expression of emotion. The sensor unit 41 supplies expression information including a feature vector constituted by the detected feature amount to the emotion estimation unit 42. The sensor unit 41 is configured to include an image feature extraction unit 51 and an audio feature extraction unit 52.

  The image feature extraction unit 51 includes, for example, a CCD (Charge Coupled Device) camera, a CMOS (Complementary Metal Oxide Semiconductor) camera, and the like, captures a user, and generates image data based on the captured image. The image feature extraction unit 51 extracts (detects), from the image data, the feature amount of the user's state that changes due to the expression of emotion. The feature amount extracted by the image feature extraction unit 51 includes, for example, facial expression, face orientation, position of each part, movement or shape, line of sight, face color, posture, gesture, position and movement of each part of the body, etc. Is included.

  The voice feature extraction unit 52 includes, for example, a microphone, and generates voice data based on the voice emitted from the user. The voice feature extraction unit 52 extracts (detects), from the voice data, the feature amount of the user's state that changes due to the expression of emotion. The feature amount extracted by the speech feature extraction unit 52 includes, for example, the phoneme of the voice uttered by the user, the sound quality, and the prosody.

  The feature vector included in the expression vector is composed of feature amounts extracted (detected) by the image feature extraction unit 51 and the audio feature extraction unit 52. The sensor unit 41 is provided with sensors for detecting feature quantities of the user's state extracted from data other than image data or audio data such as sweat volume, body temperature, blood pressure, heart rate, etc. You may make it add the feature-value detected by the sensor to the component (variable) of a feature vector.

  As will be described later with reference to FIGS. 6 to 13, the emotion estimation unit 42 estimates a user's emotion for the stimulus information presented in the output unit 14. The emotion estimation unit 42 supplies data indicating the estimation result to the recommendation processing control unit 43.

  As will be described later with reference to FIG. 10, the recommendation processing control unit 43 controls the content reproduction unit 26 based on the user's emotion, and presents content that is recommended to the user or reproduces the content. Or stop is controlled.

  The output unit 14 includes, for example, a display such as a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display), a speaker, and the like, and an image based on image data supplied from the content reproduction unit 26 or the content information presentation unit 27. Is displayed, or audio based on audio data supplied from the content reproduction unit 26 or the content information presentation unit 27 is output.

  FIG. 2 is a block diagram showing details of the emotion estimation unit 42 of FIG. The emotion estimation unit 42 includes an input information acquisition unit 101, a feature amount acquisition unit 102, a learning unit 103, an estimation unit 104, an emotion map storage unit 105, an expression-emotion data storage unit 106, and an emotion discrimination model storage unit 107. Configured to include.

  The input information acquisition unit 101 acquires user command information from the user input acquisition unit 28. Further, the input information acquisition unit 101 acquires meta information of content from the content reproduction unit 26 or the content information presentation unit 27. The input information acquisition unit 101 supplies the acquired user command information or meta information to the emotion detection unit 111.

  The feature amount acquisition unit 102 acquires expression information from the sensor unit 41. The feature amount acquisition unit 102 supplies information indicating the feature vector included in the expression information to the data collection unit 112 or the matching unit 121.

  As will be described later with reference to FIG. 6 to FIG. 9 or FIG. 11 to FIG. 13, the learning unit 103 indicates the relationship between the feature vector indicating the user state that changes due to the expression of emotion and the user's emotion. Learn the emotion discrimination model used to discriminate emotions. The learning unit 103 is configured to include an emotion detection unit 111, a data collection unit 112, and an emotion discrimination model learning unit 113.

  As will be described later with reference to FIG. 8 or FIG. 13, the emotion detection unit 111 is presented to the user based on the emotion map stored in the emotion map storage unit 105 and the user command information or meta information. The user's emotion for the stimulus information is detected. The emotion detection unit 111 supplies information for notifying the detected emotion to the data collection unit 112.

  FIG. 3 is a diagram illustrating an example of an emotion map. In the emotion map of FIG. 3, information indicating the degree of interest of the user with respect to stimulus information predicted by the command content indicated by the user command information input by the user and the command content are set in advance in association with each other. That is, in the emotion map of FIG. 3, when the stimulus information is presented to the user, the user command that is information that the user intentionally inputs to the content presentation control unit 11 that presents (gives) the stimulus information to the user. The degree of interest of the user with respect to the stimulus information, which is predicted based on the command content of the information, is set.

  For example, in the emotion map of FIG. 3, when the user commands the reproduction of the content, it is predicted that the content is highly interested, so the degree of interest is set to +1. Also, when the user commands fast-forwarding of content playback, it is predicted that the user is not interested in the content being played back, so the degree of interest is set to -1. Furthermore, when the user commands the stop of playback during the playback of the content or commands the skip of the playback, it is predicted that the user is less interested in the content being played back, so the degree of interest is set to -2. .

  FIG. 4 is a diagram illustrating another example of the emotion map. In the emotion map shown in FIG. 3, the command content is limited to a single one, whereas in the emotion map shown in FIG. 4, the command content is a plurality of commands arranged in time series. .

  For example, in the emotion map of FIG. 4, if the user commands the reproduction of the content and commands the skipping of the reproduction of the content within 30 seconds, it is predicted that the interest in the reproduced portion is low. 1 is set. In addition, when the user instructs to reproduce the content and repeats the instruction to skip the reproduction of the content twice within 30 seconds, it is predicted that the interest in the reproduced portion is further less. Is set. Further, when the user instructs the reproduction of the content and then plays it to the end as it is, it is predicted that the content is highly interested, so the degree of interest is set to +1. In addition, after the user commands the skip of content playback, if the user commands the playback of the skipped content, and 30 seconds or more have passed, the interest in the skipped content is predicted to be high. It is set to +2.

  Note that the emotion map shown in FIG. 3 or 4 is generated by, for example, other learning processing performed in advance. Further, information indicating emotions other than the degree of interest (for example, emotions or the degree of individual emotions) may be set in the emotion map. In addition, information that is intentionally input by the user when stimulus information other than the command content is presented (for example, information indicating the content of the user's response to the question) may be set in the emotion map.

  The data collection unit 112 temporarily stores information indicating the feature vector supplied from the feature amount acquisition unit 102 in the expression-emotion data storage unit 106. When the data collection unit 112 acquires data indicating the user's emotion from the emotion detection unit 111, the data collection unit 112 acquires the data within a predetermined time (for example, 30 seconds) (hereinafter referred to as a determination time) immediately before the expression-emotion. Expression-emotional data is generated in which data in which the feature vectors stored in the data storage unit 106 are arranged in time series (hereinafter referred to as a feature vector group) and the notified emotion are associated with each other. In other words, the expression-emotion data is data in which a user's emotion with respect to stimulus information given to the user is associated with a feature vector group indicating the user's state at that time. When the emotion map of FIG. 3 or FIG. 4 is used, the expression-emotion data is data in which the feature vector group is associated with the degree of interest of the user. The data collection unit 112 accumulates the generated expression-emotion data in the expression-emotion data storage unit 106. In addition, when the accumulated amount of the expression-emotion data exceeds a predetermined threshold value, the data collection unit 112 supplies information that notifies the emotion discrimination model learning unit 113.

  As will be described later with reference to FIG. 9, the emotion discrimination model learning unit 113 learns an emotion discrimination model by analyzing the expression-emotion data stored in the expression-emotion data storage unit 106. The emotion discrimination model learning unit 113 causes the emotion discrimination model accumulation unit 107 to accumulate data indicating the learned emotion discrimination model. In addition, the emotion discrimination model learning unit 113 supplies the matching unit 121 with data indicating a conversion vector for converting the feature vector into the principal component vector.

  FIG. 5 is a diagram showing an example of an emotion discrimination model learned when the emotion map of FIG. 3 or FIG. 4 is used. In the emotion discrimination model of FIG. 5, a principal component vector group and an estimated interest level are set in association with each other. The principal component vector is a vector group composed of vectors (hereinafter referred to as principal component vectors) obtained by converting each variable of the feature vector included in the feature vector group of the expression-emotional data into principal components by principal component analysis. The estimated degree of interest is information set based on the degree of interest associated with the feature vector group before conversion in the expression-emotion data. That is, the emotion discrimination model in FIG. 5 is a model that indicates a feature amount indicating the user's state and the user's emotion (degree of interest).

  When information indicating emotions other than the interest level is set in the emotion map, information based on those emotions is set instead of the estimated interest level of the emotion discrimination model.

  As will be described later with reference to FIG. 10, the estimation unit 104 estimates the user's emotion for the stimulus information presented to the user. The estimation unit 104 supplies information for notifying the estimation result to the recommendation processing control unit 43. The estimation unit 104 is configured to include a matching unit 121 and an emotion output unit 122.

  The matching unit 121 converts the feature vector supplied from the feature amount acquisition unit 102 into a principal component vector, as will be described later with reference to FIG. The matching unit 121 searches the emotion discrimination model for a vector group that matches the principal component vector group in which the principal component vectors within the discrimination time are arranged in time series. When the matching unit 121 detects a vector group that matches the principal component vector, the matching unit 121 supplies the emotion output unit 122 with information for notifying the user's emotion corresponding to the vector group (in the case of FIG. 5, the estimated interest level). To do.

  The emotion output unit 122 supplies information for notifying the user's emotion estimated by the matching unit 121 to the recommendation processing control unit 43 as necessary.

  The emotion map storage unit 105, the expression-emotion data storage unit 106, and the emotion discrimination model storage unit 107 are configured by a recording medium such as a hard disk, for example. The emotion map storage unit 105, the expression-emotion data storage unit 106, and the emotion discrimination model storage unit 107 may be configured by one recording medium, or may be configured by different recording media. May be.

  Next, processing of the content recommendation system 1 will be described with reference to FIGS. As described above, in the following description, an example in which the user's interest level is estimated will be described.

  First, the learning process executed by the content recommendation system 1 will be described with reference to the flowchart of FIG.

  In step S <b> 1, the emotion estimation unit 42 executes the expression-emotion data accumulation process. Details of the expression-emotion data accumulation process will be described later with reference to FIG. 7, but the expression-emotion data is accumulated by this process.

  In step S <b> 2, the data collection unit 112 determines whether or not the amount of expression-emotion data accumulated exceeds a threshold value. If it is determined that the accumulated amount of the expression-emotional data does not exceed the threshold value, the process returns to step S1 until the determination is made that the accumulated amount of the expressed-emotional data exceeds the threshold value in step S2. The processes of S1 and S2 are repeatedly executed. That is, the expression-emotion data is accumulated until the threshold value is exceeded.

  If it is determined in step S2 that the accumulated amount of the expression-emotional data exceeds the threshold, the process proceeds to step S3.

  In step S3, the emotion discrimination model learning unit 113 executes an emotion discrimination model learning process. The details of the emotion discrimination model learning process will be described later with reference to FIG. 9. By this process, the emotion discrimination model is learned based on the expression-emotion data.

  In step S4, the emotion estimation unit 42 starts a content automatic reproduction control process described later with reference to FIG. 10, and the learning process ends.

  Next, with reference to FIG. 7, the details of the expression-emotion data accumulation process in step S1 of FIG. 6 will be described.

  In step S21, the content information presentation unit 27 presents recommendation information to the user. Specifically, the content information presentation unit 27 reads meta information of content recommended to the user from the meta information storage unit 24 via the meta information extraction unit 25. Based on the read meta information, the content information presentation unit 27 generates information related to content recommended to the user, that is, image data and audio data for presenting the recommendation information. The content information presentation unit 27 supplies the generated image data and audio data to the output unit 14. The output unit 14 displays an image based on the image data and outputs sound based on the sound data. That is, content recommendation information as stimulus information is presented to the user.

  The recommendation information presented to the user includes, for example, the title, genre, outline, performer, production date / time, and reproduction time of the recommended content. In addition, the recommended content may be selected at random from the content in which the meta information is stored in the meta information storage unit 24, or based on the user's preference information learned in advance. The content suitable for the user's preference may be selected.

  In step S <b> 22, the data collection unit 112 executes the expression-emotion data addition process. Details of the expression-emotion data addition process will be described later with reference to FIG. 4, but new expression-emotion data is accumulated by this process.

  Next, with reference to FIG. 8, the details of the expression-emotion data addition process in step S22 of FIG. 7 will be described.

  In step S41, the data collection unit 112 starts accumulation of feature vectors. Specifically, the feature amount acquisition unit 102 is a table including a feature vector composed of the user feature amount extracted by the image feature extraction unit 51 and the user feature amount extracted by the audio feature extraction unit 52. Out information is acquired from the sensor unit 41. The feature amount acquisition unit 102 supplies data indicating the feature amount vector included in the acquired expression information to the data collection unit 112. The data collection unit 112 starts accumulation of data indicating the acquired feature vector in the expression-emotion data accumulation unit 106.

  In step S42, the user input acquisition unit 28 determines whether or not a processing command is input by the user. The user input acquisition unit 28 determines that the process command is input by the user when the user command information indicating the process command input by operating the remote controller 12 by the user is acquired from the remote controller 12. Proceed to S43.

  In step S43, the emotion detection unit 111 detects the user's emotion. Specifically, the user input acquisition unit 28 supplies user command information to the content reproduction unit 26, the content information presentation unit 27, and the emotion detection unit 111. The content reproduction unit 26 or the content information presentation unit 27 starts predetermined processing based on the command content indicated in the user command information. For example, when the user instructs the reproduction of the content recommended by the recommendation information, the content reproduction unit 26 reads the content instructed to reproduce from the content storage unit 23. The content reproduction unit 26 supplies image data and audio data obtained by reproducing the read content to the output unit 14. The output unit 14 displays an image based on image data, that is, an image of content recommended to the user, and outputs audio based on the audio data, that is, audio of content recommended to the user.

  The emotion detection unit 111 detects an interest level corresponding to the command content indicated in the user command information from the emotion maps stored in the emotion map storage unit 105. The degree of interest detected at this time indicates the degree of interest of the user with respect to stimulus information presented to the user, that is, recommendation information or content. The emotion detection unit 111 supplies information for notifying the detected degree of interest to the data collection unit 112.

  In step S44, the data collection unit 112 accumulates the expression-emotional data. Specifically, the data collection unit 112 includes a feature vector group in which feature vectors accumulated in the immediately previous discrimination time among the feature vectors accumulated in the expression-emotion data accumulation unit 106 are arranged in time series. Then, the expression-emotion data in association with the notified interest level is generated. That is, the expression-emotion data generated at this time is data in which the feature vector group indicating the user's state with respect to the stimulus information is associated with the degree of interest of the user with respect to the stimulus information. The data collection unit 112 stores the generated expression-emotion data in the meta information storage unit 24.

  Thereafter, the process returns to step S42.

  If it is determined in step S42 that the user has not input a processing instruction, the process proceeds to step S45.

  In step S45, the content reproduction unit 26 determines whether or not the reproduction of the content has ended. When the predetermined time has not elapsed since the recommendation information was presented, when the content is being reproduced, or when the predetermined time has not elapsed since the reproduction of the content was stopped, It is determined that the content reproduction has not ended, and the process returns to step S42.

  Thereafter, in step S45, the processes in steps S42 to S45 are repeatedly executed until it is determined that the reproduction of the content is completed, and the expression-emotional data is accumulated every time the user inputs a process command.

  In step S45, the content reproduction unit 26 determines that the content has not been reproduced even after a predetermined time has elapsed since the recommendation information was presented, or the predetermined time has elapsed since the content reproduction was stopped. In this case, it is determined that the reproduction of the content has ended, and the process proceeds to step S46.

  In step S46, the data collection unit 112 stops the accumulation of feature vectors in the expression-emotion data storage unit 106, and the expression-emotion data addition process ends.

  Next, details of the emotion discrimination model learning process in step S3 in FIG. 6 will be described with reference to the flowchart in FIG.

  In step S61, the emotion discrimination model learning unit 113 learns an emotion discrimination model based on the expression-emotion data, and the emotion discrimination model learning process ends. Specifically, the data collection unit 112 supplies the emotion determination model learning unit 113 with information that notifies that the accumulated amount of the expression-emotion data exceeds the threshold value.

  The emotion discrimination model learning unit 113 reads the expression-emotion data stored in the expression-emotion data storage unit 106. The emotion discrimination model learning unit 113 obtains the principal component of the feature vector by performing principal component analysis on the feature vector included in the expression-emotional data. The emotion discrimination model learning unit 113 calculates, for each feature vector included in the expression-emotion data, a principal component vector obtained by converting each variable of each feature vector into a principal component. In addition, the emotion discrimination model learning unit 113 converts the interest level of the expression-emotion data into the estimated interest level. The estimated interest level is set to be interesting when the interest level is equal to or greater than a predetermined threshold (for example, 0), and is set to no interest when the degree of interest is less than the threshold.

  The emotion discrimination model learning unit 113 is an emotion discrimination model composed of data obtained by converting a combination of a feature vector group and an interest level of each expression-emotion data into a combination of a principal component vector group and an estimated interest level (FIG. 5). Is generated. At this time, a principal component vector group that is close to each other is integrated into one vector group, or a principal component vector group that is isolated from another principal component vector group (distance away from a predetermined threshold) It may be deleted.

  The emotion discrimination model learning unit 113 causes the emotion discrimination model accumulation unit 107 to accumulate data indicating the emotion discrimination model. In addition, the emotion discrimination model learning unit 113 supplies the matching unit 121 with data indicating a conversion vector for converting the feature vector into the principal component vector.

  Next, the content automatic reproduction control process performed subsequent to the learning process of FIG. 6 will be described with reference to the flowchart of FIG.

  In step S101, the recommendation information is presented to the user as in the process of step S21 of FIG.

  In step S102, the matching unit 121 acquires a feature amount vector. Specifically, the matching unit 121 acquires data indicating the feature amount vector included in the expression information supplied from the sensor unit 41 from the feature amount acquisition unit 102.

  In step S103, the matching unit 121 performs a matching process. Specifically, the matching unit 121 converts a feature vector within a determination time after recommendation information is presented to the user into a principal component vector, and a principal component vector group (hereinafter, referred to as a time series) of the converted principal component vectors. , Referred to as an input principal component vector group). The matching unit 121 reads data indicating the emotion discrimination model accumulated in the emotion discrimination model accumulation unit 107. The matching unit 121 obtains a correlation between each principal component vector group included in the emotion discrimination model and the input principal component vector group. For example, the matching unit 121 calculates the correlation coefficient between the principal component vectors corresponding to the principal component vector in order from the head principal component vector for the principal component vector group and the input principal component vector group included in the emotion discrimination model. Find the sum of correlation coefficients.

  In step S104, the matching unit 121 determines whether there is a matching principal component vector group. Specifically, the matching unit 121 determines that the maximum value among the sums of correlation coefficients of the principal component vector groups included in the emotion discrimination model for the input principal component vector group calculated in step S103 exceeds a predetermined threshold value. Then, it is determined that there is a principal component vector group that matches the input principal component vector group in the emotion discrimination model, and the process proceeds to step S105. Of the principal component vector groups included in the emotion discrimination model, the principal component vector group that maximizes the sum of the correlation coefficients with the input principal component vector group is the user state corresponding to the recommendation information presented in step S101. It is a principal component vector group indicating a close state.

  In step S105, the matching unit 121 determines whether the user is interested in the recommended content. Specifically, the matching unit 121 determines that the estimated interest level corresponding to the principal component vector group that maximizes the sum of the correlation coefficients with the input principal component vector group is set to be interesting when the estimated content of interest is set to the user. Is determined to be interested, and the process proceeds to step S106.

  In step S106, the recommendation processing control unit 43 automatically reproduces the content. Specifically, the matching unit 121 supplies information indicating that the user is interested in the recommended content to the recommendation processing control unit 43 via the emotion output unit 122. The recommendation processing control unit 43 supplies information for instructing reproduction of the recommended content to the content reproduction unit 26. The content playback unit 26 reads the content for which playback has been commanded from the content storage unit 23. The content reproduction unit 26 supplies image data and audio data obtained by reproducing the read content to the output unit 14. The output unit 14 displays an image based on image data, that is, an image of content recommended to the user, and outputs audio based on the audio data, that is, audio of content recommended to the user.

  In step S105, if the estimated interest level corresponding to the principal component vector group having the maximum correlation coefficient with the input principal component vector group is set to be uninteresting, the user is not interested in the recommended content. The process of step S106 is skipped, the content is not automatically reproduced, and the process proceeds to step S107.

  If it is determined in step S104 that there is no matching principal component vector group, that is, there is no principal component vector group that matches the principal component vector group indicating the current user state in the emotion discrimination model, and the user for the recommended content If the degree of interest cannot be estimated, the processes of steps S105 and S106 are skipped, and the process proceeds to step S107.

  In step S107, the expression-emotion data addition process described above with reference to FIG. 8 is executed, and the content automatic reproduction control process ends. Thus, even after the learning process is finished and the automatic content reproduction control process is started, the expression-emotional data is accumulated every time the content is recommended to the user. And the reliability of the user's interest degree which the emotion estimation part 42 estimates can be made higher by performing a learning process newly based on the accumulated expression-emotion data.

  As described above, by using the emotion discrimination model learned based on the user's state detected when the stimulus information is presented during the learning period, each user's interest level is estimated by estimating the user's degree of interest. The degree of interest of the user can be estimated more accurately according to the difference in individual responses. In addition, when learning the emotion discrimination model, the user does not need to take a special action, so the burden on the user does not increase. In addition, when it is determined that the user is interested, the recommended content is automatically reproduced, so that convenience for the user is improved.

  In the above description, an example is shown in which the relationship between the user state and the degree of interest is learned by detecting the user state when a processing command is input. It is also possible to learn the relationship between the user's state and the degree of interest by detecting the state. Hereinafter, learning of the relationship between the user state and the degree of interest by detecting the user state with respect to the content of the stimulus given to the user is referred to as stimulus correspondence learning.

  With reference to FIGS. 11 to 13, processing of the content recommendation system 1 in the case of performing stimulus correspondence learning will be described.

  FIG. 11 shows an example of an emotion map used in the case of stimulating correspondence learning. In the emotion map of FIG. 11, information indicating the degree of interest of the user with respect to the stimulus given to the user and information indicating the content of the stimulus are set in advance in association with each other. The stimulus content set in the emotion map includes data indicating the characteristics of the stimulus given to the user, such as the name of the talent, the name of the song, the program name, and the genre name (eg, sports, news, etc.). For example, in the emotion map of FIG. 11, since the user is very interested in the talent A, the degree of interest is set to +3. Further, since the user is highly interested in the music B, the degree of interest is set to +1. Further, since the user is very interested in the program C, the degree of interest is set to +2. Also, since the user's interest in the genre D is very low, the interest level is set to -2.

  Note that the emotion map shown in FIG. 11 is generated by, for example, another learning process performed in advance or a questionnaire for the user.

  Next, processing executed by the content recommendation system 1 in the case of performing stimulus correspondence learning will be described with reference to the flowcharts of FIGS. 12 and 13.

  The learning process in the case of performing stimulus correspondence learning is the same as the learning process described with reference to FIG.

  Next, with reference to FIG. 12, the expression-emotion data accumulation process in step S1 of FIG.

  In step S201, recommendation information is presented to the user in the same manner as in step S21 of FIG. At this time, the content information presentation unit 27 supplies the meta information that is the basis of the recommendation information to the emotion detection unit 111 via the input information acquisition unit 101.

  In step S <b> 202, the data collection unit 112 executes an expression-emotion data addition process. Details of the expression-emotion data addition process will be described later with reference to FIG. 13, but new expression-emotion data is accumulated by this process.

  Next, with reference to FIG. 13, the details of the expression-emotional data adding process in step S202 of FIG. 12 will be described.

  During the execution of the expression-emotion data addition process, the user operates the remote controller 12 to input a process command, and the content playback unit 26 or content from the remote controller 12 via the user input acquisition unit 28. When user command information is supplied to the information presentation unit 27, the content reproduction unit 26 or the content information presentation unit 27 starts a predetermined process based on the command content indicated in the user command information.

  For example, when the user instructs the reproduction of the content recommended by the recommendation information, the content reproduction unit 26 reads the content instructed to reproduce from the content storage unit 23. The content reproduction unit 26 supplies image data and audio data obtained by reproducing the read content to the output unit 14. The output unit 14 displays an image based on image data, that is, an image of content recommended to the user, and outputs audio based on the audio data, that is, audio of content recommended to the user. In addition, the content reproduction unit 26 reads the meta information of the content to be reproduced from the meta information storage unit 24 via the meta information extraction unit 25. The content reproduction unit 26 appropriately supplies meta information corresponding to the scene being reproduced to the emotion detection unit 111 via the input information acquisition unit 101 during the reproduction of the content.

  In step S221, as in the process of step S41 of FIG. 8, accumulation of feature vectors is started.

  In step S222, the emotion detection unit 111 determines whether a predetermined stimulus is given to the user. Specifically, when the recommendation information is presented to the user, the emotion detection unit 111 includes the stimulus content defined in the emotion map in the meta information supplied from the content information presentation unit 27. It is determined whether or not. If the stimulus information defined in the emotion map is included in the meta information, the emotion detection unit 111 determines that a predetermined stimulus is given to the user, and the process proceeds to step S223.

  In addition, when the content is being played back, the emotion detection unit 111 supplies the stimulus content defined in the emotion map in the meta information corresponding to the scene of the content currently being played, supplied from the content playback unit 26. Whether or not is included is determined. When the stimulus information defined in the emotion map is included in the meta information, it is determined that a predetermined stimulus is given to the user, and the process proceeds to step S223.

  In step S223, the emotion detection unit 111 detects the user's emotion. Specifically, the emotion detection unit 111 detects the degree of interest corresponding to the stimulus content determined to be defined in the emotion map in step S222 from the emotion map. The emotion detection unit 111 supplies information for notifying the detected degree of interest to the data collection unit 112. The degree of interest detected at this time indicates the degree of interest of the user with respect to stimulus information presented to the user, that is, recommendation information or content.

  In step S224, the expression-emotion data is accumulated as in the process of step S43 of FIG.

  Thereafter, the process returns to step S222.

  In step S222, when the recommendation information is presented to the user, when the stimulus information defined in the emotion map is not included in the meta information supplied from the content information presentation unit 27, the emotion detection unit 111 is displayed. Determines that the predetermined stimulus is not given to the user, and the process proceeds to step S225. In addition, when the content is being played back, when the stimulus information defined in the emotion map is not included in the meta information supplied from the content playback unit 26, the emotion detection unit 111 causes the user to receive a predetermined stimulus. It is determined that it has not been given, and the process proceeds to step S225.

  In step S225, it is determined whether or not the reproduction of the content has been completed, similar to the process in step S45 of FIG. 8 described above. If it is determined that the content reproduction has not ended, the process returns to step S222.

  Thereafter, until it is determined in step S225 that the reproduction of the content has been completed, the processing in steps S222 to S225 is repeatedly executed, and the scene of the content including the stimulus content defined in the emotion map is presented to the user. In addition, the expression-emotion data is accumulated.

  If it is determined in step S225 that the content reproduction has ended, the process proceeds to step S226.

  In step S226, similar to the process of step S46 of FIG. 13 described above, the accumulation of feature vectors is terminated, and the expression-emotion data addition process is terminated.

  The emotion discrimination model learning process in step S3 of FIG. 6 in the case of performing stimulus correspondence learning is the same as the emotion discrimination model learning process described with reference to FIG. That is, the emotion discrimination model shown in FIG. 5 is learned based on the expression-emotion data based on the emotion map shown in FIG.

  In this manner, the emotion discrimination model can be learned in the same manner as the processing described above with reference to FIGS. Further, by using the learned emotion discrimination model, it is possible to accurately estimate the user's degree of interest with respect to the stimulus content not defined in the emotion map.

  As described above, the feature amount data indicating the feature amount of the user state that changes due to the expression of emotion is acquired, and when the user is given a stimulus, the input information intentionally input by the user is acquired, When the first stimulus that is the stimulus is given to the user and the emotion data that is set in association with the information indicating the user's emotion with respect to the stimulus predicted by the input information input by the user and the input information, Features acquired when learning a model indicating the relationship between the feature data and the user's emotion based on the feature data acquired when the user inputs input information, and applying a second stimulus When estimating the user's emotion when the second stimulus is applied based on the quantity data and the model, the user's emotion can be estimated. Moreover, a user's emotion can be estimated more correctly, without increasing a user's burden.

  Further, feature amount data indicating the feature amount of the user's state that changes due to the expression of emotion is acquired, and information indicating the user's emotion with respect to the first stimulus and information indicating the first stimulus are set in advance in association with each other. Based on the sentiment data and the feature data acquired when the first stimulus is given to the user, a model indicating the relationship between the feature data and the user's emotion is learned, and the second stimulus In the case where the user's emotion when the second stimulus is applied is estimated based on the feature amount data acquired when the second stimulus is applied and the model, the user's emotion can be estimated. Moreover, a user's emotion can be estimated more correctly, without increasing a user's burden.

  In the above description, an example in which content is automatically reproduced using the estimated degree of interest of the user has been described. However, other processes can be automated. For example, when it is estimated that the degree of interest of the user is low during the reproduction of the content, the reproduction of the content may be automatically stopped. Also, for example, when it is estimated that the user's degree of interest is low while watching a television broadcast program, the program is automatically switched to the back program, or a recording medium such as an HDD (hard disk) or DVD (Digital Versatile Disc) It is also possible to automatically switch to playback of the content recorded in Furthermore, when the content is text information, more detailed information may be presented to the user when the degree of interest of the user is estimated to be high. As a result, user convenience is improved.

  Further, for example, the present invention is applied to a conversation system used for a pet robot or the like, and the conversation is continued, stopped, or the topic is changed according to the degree of interest of the user. Good. Thereby, it is possible to entertain the user without getting bored.

  Furthermore, you may make it set the interest degree set to expression-emotion data using information other than an emotion map. For example, if the recommended content is music, the rhythm or beat of the music being played is analyzed by music feature analysis, and the rhythm or beat of the music is detected by detecting the shaking of a part of the user's body. The degree of synchronization with shaking of a part of the user's body is obtained. When the rhythm or beat of the music is synchronized with the shaking of a part of the user's body, the user's interest at that time can be set higher (for example, to +5). .

  Further, the estimated interest degree of the emotion discrimination model in FIG. 5 is not the binary value of interest and not interested, but the interest degree is used as it is, or the number of discrete values that the estimated interest degree can take is increased or estimated. The degree of interest may be a continuous amount or the user's degree of interest may be estimated in more detail.

  Furthermore, in the above description, an example of estimating the degree of interest has been described. However, emotions other than the degree of interest (for example, emotions, degrees of individual emotions, etc.) may be estimated. In this case, for example, it can be realized by setting an emotion other than the degree of interest in the emotion map shown in FIG. 3, FIG. 4, or FIG. Similarly to the case of using the degree of interest, it is possible to use the user's emotion estimated for various types of recommendation processing and automatic processing.

  In the above description, an example in which a processing command is input by operating the remote controller 12 is described. However, a method of inputting a processing command is not limited to a specific method. For example, a processing command may be input by voice input or a gesture, or a processing command may be input by operating a button or switch (not shown) provided in the content presentation control unit 11. Is possible.

  Furthermore, the emotion discrimination model can be learned using information that the user intentionally inputs to the stimulus information other than the processing command. For example, an emotion discrimination model can be learned on the basis of a feature vector obtained when a user is asked about the current emotion and an answer is given to the user.

  Further, instead of the emotion discrimination model shown in FIG. 5, for example, the relationship between the detected feature quantity of the user and the degree of interest is represented by a linear or nonlinear function, and the degree of interest of the user is calculated by function calculation. May be.

  Furthermore, the interest level in units of content can be obtained by adding the interest levels detected during the reproduction of the content. The obtained interest level of the content unit can be used for learning the user preference of the content unit by associating it with the meta information of the content.

  In the above description, examples of presenting content recommendation information or content as a stimulus to be given to the user have been described. However, for example, various products, restaurant menus, various pamphlets, and the like may be given. It may be. For example, it is possible to acquire a user's state when a stimulus other than content is given and input information intentionally input by the user and learn an emotion discrimination model using an emotion map prepared in advance. it can.

  In addition to the system for recommending information such as the content described above to the user, the present invention is a device that controls processing using the user's emotion, for example, a pet robot that reacts to the user's emotion, the user's emotion It can be applied to game machines or software that switch stories and the like.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 14 is a block diagram showing an example of the configuration of a personal computer that executes the above-described series of processing by a program. A CPU (Central Processing Unit) 901 executes various processes according to a program stored in a ROM (Read Only Memory) 902 or a recording unit 908. A RAM (Random Access Memory) 903 appropriately stores programs executed by the CPU 901, data, and the like. The CPU 901, ROM 902, and RAM 903 are connected to each other by a bus 904.

  An input / output interface 905 is also connected to the CPU 901 via the bus 904. The input / output interface 905 is connected to an input unit 906 made up of a keyboard, mouse, microphone, etc., and an output unit 907 made up of a display, a speaker, etc. The CPU 901 executes various processes in response to a command input from the input unit 906. Then, the CPU 901 outputs the processing result to the output unit 907.

  The recording unit 908 connected to the input / output interface 905 includes, for example, a hard disk, and stores programs executed by the CPU 901 and various data. A communication unit 909 communicates with an external device via a network such as the Internet or a local area network.

  Further, the program may be acquired via the communication unit 909 and stored in the recording unit 908.

  A drive 910 connected to the input / output interface 905 drives a removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the program or data recorded therein. Get etc. The acquired program and data are transferred to and stored in the recording unit 908 as necessary.

  As shown in FIG. 14, a program recording medium that stores a program that is installed in a computer and can be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory, DVD (Digital Versatile Disc), a magneto-optical disk, a removable medium 911 which is a package medium composed of a semiconductor memory, a ROM 902 in which a program is temporarily or permanently stored, and a recording unit 908 It is comprised by the hard disk etc. which comprise. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 909 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the order described, but is not necessarily performed in time series. Or the process performed separately is also included.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  Furthermore, the embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

It is a block diagram which shows one Embodiment of the content recommendation system to which this invention is applied. It is a block diagram which shows one Embodiment of a structure of the emotion estimation part of FIG. It is a figure which shows the example of an emotion map. It is a figure which shows the other example of an emotion map. It is a figure which shows the example of an emotion discrimination | determination model. It is a flowchart for demonstrating the learning process performed by the content recommendation system of FIG. It is a flowchart for demonstrating the detail of the expression-emotional data accumulation | storage process of step S1 of FIG. It is a flowchart for demonstrating the detail of the expression-emotional data addition process of step S22 of FIG. It is a flowchart for demonstrating the detail of the emotion discrimination | determination model learning process of step S3 of FIG. It is a flowchart for demonstrating the content automatic reproduction | regeneration control processing performed by the content recommendation system of FIG. It is a figure which shows the further another example of an emotion map. It is a flowchart for demonstrating the detail of the other example of the expression-emotional data accumulation | storage process of step S1 of FIG. It is a flowchart for demonstrating the detail of the expression-emotional data addition process of step S202 of FIG. And FIG. 11 is a block diagram illustrating an example of a configuration of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Content recommendation system, 11 Remote controller, 12 Content presentation control part, 13 Content recommendation control part, 14 Output part, 25 Meta information extraction part, 26 Content reproduction part, 27 Content information presentation part, 28 User input acquisition part, 41 Sensor Unit, 42 emotion estimation unit, 43 recommendation processing control unit, 101 input information acquisition unit, 102 feature quantity acquisition unit, 103 learning unit, 104 estimation unit, 111 emotion detection unit, 112 data collection unit, 113 emotion discrimination model learning unit, 121 matching unit, 122 emotion output unit, 901 CPU, 902 ROM, 903 RAM, 908 recording unit, 910 drive, 911 removable media

Claims (16)

Feature quantity acquisition means for acquiring feature quantity data indicating a feature quantity of a user's state that changes due to the expression of emotion;
Input information acquisition means for acquiring input information intentionally input by the user when a stimulus is given to the user;
Information indicating the emotion of the user with respect to the stimulus predicted by the input information input by the user and emotion data in which the input information is set in advance in association with the user, and the first that is the stimulus to the user Learning means for learning a model indicating a relationship between the feature amount data and the user's emotion based on the feature amount data acquired when the user inputs the input information when a stimulus is given When,
An emotion estimation apparatus comprising: estimation means for estimating the emotion of the user when the second stimulus is applied based on the feature amount data acquired when the second stimulus is applied and the model; . The emotion estimation apparatus according to claim 1, wherein the input information is information indicating a processing instruction to a device that gives the user the first stimulus. The information indicating the degree of interest of the user with respect to the stimulus predicted by the input information input by the user and the input information are set in association with the emotion data in advance. Emotion estimation device. The emotion estimation apparatus according to claim 1, further comprising detection means for detecting a feature amount of the user's state and outputting the feature amount data. The emotion estimation apparatus according to claim 1, wherein the stimulus given to the user is information presented to the user. The emotion estimation apparatus according to claim 5, further comprising information presentation control means for controlling presentation of information recommended to the user based on the estimated emotion of the user. Acquire feature quantity data indicating the feature quantity of the user's state that changes due to the expression of emotions,
When stimulation is given to the user, input information that is intentionally input by the user is acquired,
Information indicating the user's emotions with respect to the stimulus predicted by the input information input by the user and emotion data in which the input information is set in advance in association with the user, and the first that is the stimulus to the user When a stimulus is applied, based on the feature amount data acquired when the user inputs the input information, a model indicating a relationship between the feature amount data and the user's emotion is learned.
An emotion estimation method including a step of estimating an emotion of the user when the second stimulus is applied based on the feature amount data acquired when the second stimulus is applied and the model. Acquire feature quantity data indicating the feature quantity of the user's state that changes due to the expression of emotions,
When stimulation is given to the user, input information that is intentionally input by the user is acquired,
Information indicating the emotion of the user with respect to the stimulus predicted by the input information input by the user and emotion data in which the input information is set in advance in association with the user, and the first that is the stimulus to the user When a stimulus is applied, based on the feature amount data acquired when the user inputs the input information, a model indicating a relationship between the feature amount data and the user's emotion is learned.
A program for causing a computer to execute a step of estimating an emotion of the user when the second stimulus is given based on the feature amount data obtained when the second stimulus is given and the model. Feature quantity acquisition means for acquiring feature quantity data indicating a feature quantity of a user's state that changes due to the expression of emotion;
Acquired when the first stimulus is given to the user, and emotion data in which information indicating the user's emotion with respect to the first stimulus and information indicating the first stimulus are set in advance in association with each other. Learning means for learning a model indicating a relationship between the feature amount data and the user's emotion based on the feature amount data;
An emotion estimation apparatus comprising: estimation means for estimating the emotion of the user when the second stimulus is applied based on the feature amount data acquired when the second stimulus is applied and the model; . The emotion estimation apparatus according to claim 9, wherein information indicating a degree of interest of the user with respect to the first stimulus and information indicating the first stimulus are set in advance in the emotion data. The emotion estimation apparatus according to claim 9, further comprising detection means for detecting a feature amount of the user's state and outputting the feature amount data. The emotion estimation apparatus according to claim 9, wherein the first stimulus or the second stimulus given to the user is information presented to the user. The emotion estimation apparatus according to claim 12, further comprising information presentation control means for controlling presentation of information recommended to the user based on the estimated emotion of the user. Acquire feature quantity data indicating the feature quantity of the user's state that changes due to the expression of emotions,
Acquired when the first stimulus is given to the user, and emotion data in which information indicating the user's emotion with respect to the first stimulus and information indicating the first stimulus are set in advance in association with each other. Based on the feature amount data, a model indicating a relationship between the feature amount data and the user's emotion is learned,
An emotion estimation method including a step of estimating an emotion of the user when the second stimulus is applied based on the feature amount data acquired when the second stimulus is applied and the model. Acquire feature quantity data indicating the feature quantity of the user's state that changes due to the expression of emotions,
Acquired when the first stimulus is given to the user, and emotion data in which information indicating the user's emotion with respect to the first stimulus and information indicating the first stimulus are set in advance in association with each other. Based on the feature amount data, a model indicating a relationship between the feature amount data and the user's emotion is learned,
A program for causing a computer to execute a step of estimating an emotion of the user when the second stimulus is given based on the feature amount data obtained when the second stimulus is given and the model.   A recording medium on which the program according to claim 8 or 15 is recorded.

Images