JP7278972B2 - Information processing device, information processing system, information processing method, and program for evaluating monitor reaction to merchandise using facial expression analysis technology - Google Patents

Description

The present invention relates to an information processing device, an information processing system, and a program.

Generally, in market research on products, consumer questionnaires are used to collect evaluations of products. By obtaining responses from a large number of consumers, it is expected that the accuracy of product evaluation will be improved.

Japanese Patent Laying-Open No. 2003-308415 discloses a technique for easily obtaining many questionnaire results by using a system including a user terminal and a server.

In general, it is preferable that the evaluation of products be objective.
However, consumer subjectivity is reflected in the results of the questionnaire in Japanese Patent Application Laid-Open No. 2003-308415. Therefore, the evaluation obtained from the questionnaire results does not have objectivity for the product.

An object of the present invention is to obtain an objective evaluation index for a product.

One aspect of the present invention is
A means for acquiring facial expression information related to a user's facial expression,
Equipped with means for specifying product information about the product,
comprising means for analyzing changes in the facial expression based on the facial expression information when the product information is specified;
means for calculating a reaction score, which is an indicator of the user's reaction to the product, based on the results of the analysis;
It is an information processing device.

According to the present invention, it is possible to obtain an index of a user's objective evaluation of a product.

1 is a schematic diagram of an information processing system according to a first embodiment; FIG. 1 is a block diagram showing the configuration of an information processing system according to a first embodiment; FIG. 1 is an explanatory diagram of an outline of a first embodiment; FIG. It is a figure which shows the data structure of the user information database of 1st Embodiment. It is a figure which shows the data structure of the merchandise information database of 1st Embodiment. 4 is a sequence diagram of information processing according to the first embodiment; FIG. 7 is a diagram showing an example of a screen displayed in the information processing of FIG. 6; FIG. 7 is a diagram showing an example of a screen displayed in the information processing of FIG. 6; FIG. FIG. 11 is an explanatory diagram of the overview of the second embodiment; FIG. 11 is a sequence diagram of information processing according to the second embodiment; 11 is a diagram showing an example of a screen displayed in the information processing of FIG. 10; FIG. 11 is a diagram showing an example of a screen displayed in the information processing of FIG. 10; FIG. It is a figure which shows the example of a screen displayed in the information processing of the modification 1 of 2nd Embodiment. FIG. 11 is an explanatory diagram of the outline of the third embodiment; FIG. 12 is a sequence diagram of information processing according to the third embodiment; 16 is a diagram showing an example of a screen displayed in the information processing of FIG. 15; FIG.

An embodiment of the present invention will be described in detail below with reference to the drawings. In the drawings for describing the embodiments, in principle, the same constituent elements are denoted by the same reference numerals, and repeated description thereof will be omitted.

(1) First Embodiment A first embodiment will be described.

(1-1) Configuration of Information Processing System The configuration of the information processing system of the first embodiment will be described. FIG. 1 is a schematic diagram of an information processing system according to the first embodiment. FIG. 2 is a block diagram showing the configuration of the information processing system of the first embodiment.

As shown in FIGS. 1 and 2, the information processing system 1 includes a client device 10 and a server 30. FIG.
The client device 10 and server 30 are connected via a network (for example, the Internet or an intranet) NW.

The client device 10 is an example of an information processing device that transmits requests to the server 30 . The client device 10 is, for example, a smart phone, a tablet terminal, or a personal computer.

The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10 . Server 30 is, for example, a web server.

(1-1-1) Configuration of Client Device The configuration of the client device 10 will be described.

As shown in FIG. 2, the client device 10 includes a storage device 11, a processor 12, an input/output interface 13, a communication interface 14, a display 15, a camera 16, and an audio input/output device 17.

Storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of ROM (Read Only Memory), RAM (Random Access Memory), and storage (eg, flash memory or hard disk).

Programs include, for example, the following programs.
・OS (Operating System) program ・Application (for example, web browser) program that executes information processing

The data includes, for example, the following data.
・Databases referenced in information processing ・Data obtained by executing information processing (that is, execution results of information processing)

The processor 12 is configured to implement the functions of the client device 10 by activating programs stored in the storage device 11 . Processor 12 is an example of a computer.

The input/output interface 13 is configured to acquire a user's instruction from an input device connected to the client device 10 and output information to an output device connected to the client device 10 .
Input devices are, for example, keyboards, pointing devices, touch panels, or combinations thereof.

Communication interface 14 is configured to control communications between client device 10 and server 30 .

Display 15 is configured to display an image. The display 15 is, for example, any of the following.
・Liquid crystal display ・Organic EL (Electro Luminescence) display

Camera 16 is configured to capture images and generate image data for the captured images. The camera 16 has, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The audio input/output device 17 is configured to input/output audio. The voice input/output device 17 is a combination of a microphone for inputting voice and a speaker for outputting voice.

(1-1-2) Server Configuration The configuration of the server 30 will be described.

As shown in FIG. 2, the server 30 includes a storage device 31, a processor 32, an input/output interface 33, and a communication interface .

Storage device 31 is configured to store programs and data. Storage device 31 is, for example, a combination of ROM, RAM, and storage (eg, flash memory or hard disk).

Programs include, for example, the following programs.
- OS program - Application program that executes information processing

The data includes, for example, the following data.
・Databases referenced in information processing ・Execution results of information processing

The processor 32 is configured to implement the functions of the server 30 by activating programs stored in the storage device 31 . Processor 32 is an example of a computer.

The input/output interface 33 is configured to acquire user instructions from input devices connected to the server 30 and to output information to output devices connected to the server 30 .
Input devices are, for example, keyboards, pointing devices, touch panels, or combinations thereof.
An output device is, for example, a display.

Communication interface 34 is configured to control communications between server 30 and client device 10 .

(1-2) Overview of First Embodiment An overview of the first embodiment will be described. FIG. 3 is an explanatory diagram of the outline of the first embodiment.

As shown in FIG. 3, the client device 10 acquires facial expression information (for example, moving images or facial electromyography) relating to facial expressions of the user.

The client device 10 identifies product information related to products to be evaluated.
When the client device 10 succeeds in identifying the product information, the client device 10 analyzes the facial expression information to calculate a facial expression change index, which is an index relating to changes in the user's facial expression.
The client device 10 calculates a reaction score, which is an index related to the user's reaction to the product, based on the facial expression change index obtained by the analysis.

In this way, when the client device 10 can specify product information, the client device 10 calculates the reaction score of the user to the product based on the change in the user's facial expression.
This makes it possible to obtain an objective evaluation index for the product.

(1-3) Database The database of the first embodiment will be described. The following databases are stored in the storage device 31.

(1-3-1) User Information Database The user information database of the first embodiment will be described. FIG. 4 is a diagram showing the data structure of the user information database of the first embodiment.

The user information database of FIG. 4 stores user information about users.
The user information database includes a "user ID" field, a "user name" field, and a "user attribute" field. Each field is associated with each other.

A user ID for identifying a user is stored in the "user ID" field.

The “user name” field stores information (for example, text) regarding the name of the user.

The "user attribute" field stores user attribute information about user attributes.
The "user attribute" field includes a "gender" field and an "age" field.

The "gender" field stores information about the gender of the user.

The "age" field stores information about the user's age.

(1-3-2) Product Information Database The product information database of the first embodiment will be described. FIG. 5 is a diagram showing the data structure of the product information database of the first embodiment.

The product information database in FIG. 5 stores product information on products.
The product information database includes a "product ID" field, a "product name" field, a "product image" field, a "feature pattern" field, a "product attribute" field, and a "reaction score" field. Each field is associated with each other.

A product ID for identifying a product is stored in the “product ID” field.

The “product name” field stores information (for example, text) regarding the name of the product.

The “product image” field stores image information of the product (for example, at least one of product body, product package, and product usage state).

The "feature pattern" field stores information about the pattern of features on the product image.

The "product attribute" field stores product attribute information relating to product attributes.
The "product attribute" field includes a "type" field and a "manufacturer" field.

The "type" field stores information about the type of product (for example, lipstick or lotion).

The "manufacturer" field stores information about the manufacturer of the product (for example, text indicating the manufacturer's name).

The "reaction score" field stores a numerical value (hereinafter referred to as "reaction score") indicating the level of user reaction to the product.
The "response score" field includes an "overall average" field and an "attribute average" field.

The "total average" field stores the average reaction score of all users.

The "attribute average" field stores the average reaction score for each user attribute.
The 'attribute average' field includes a 'sex' field and an 'age' field.

The "gender" field stores the average reaction score for each gender.

The "age" field stores the average reaction score for each age.

(1-4) Information Processing Information processing according to the first embodiment will be described. FIG. 6 is a sequence diagram of information processing according to the first embodiment. 7 and 8 are diagrams showing screen examples displayed in the information processing of FIG.

As shown in FIG. 6, the client device 10 executes a login request (S100).
Specifically, processor 12 displays screen P100 ( FIG. 7 ) on display 15 .

A screen P100 includes a button object B100 and a field object F100.
A field object F100 is an object that receives an input of a user ID.
A button object B100 is an object that receives a user instruction for login.

When the user inputs the user ID into the field object F100 and operates the button object B100, the processor 12 transmits login request data to the server 30. FIG.
The login request data includes the user ID entered in field object F100.

After step S100, the server 30 performs login authentication (S300).
Specifically, when the user ID included in the login request data is included in the user information database (FIG. 4), the processor 32 transmits login response data for authenticating login to the client device 10 .

After step S300, the client device 10 acquires facial expression information (S101).

As a first example of step S101, the processor 12 displays a screen P101 (FIG. 7) on the display 15 upon receiving the login response data.

The screen P101 includes a display object A101 and a button object B101.
An image captured by the camera 16 is displayed on the display object A101. The images captured by the camera 16 include an image IMG101a of the user's face and an image IMG101b of the product.
The button object B101 is an object that receives a user instruction for acquiring video data.

When the user operates the button object B<b>101 , the processor 12 generates video data (an example of “facial expression information”) and stores the video data in the storage device 11 . The video data includes moving images captured by the camera 16 and audio input by the audio input/output device 17 .

As a second example of step S101, the client device 10 is connected to a measuring device. The measuring device is configured, for example, to acquire facial muscle signals related to facial muscle movements from electrodes worn on the user's face. The processor 12 acquires facial muscle signals from the measuring device.

After step S101, the client device 10 executes product identification (S102).

As a first example of step S102, the processor 12 identifies the product ID of the product based on a user instruction (for example, product information for a field object included in the screen displayed on the display 15).

As a second example of step S102, the processor 12 extracts the feature amount of the image IMG101b by applying the feature amount analysis to the product image IMG101b included in the video data acquired in step S101.
Processor 12 identifies the product ID of the product represented by image IMG101b by comparing the extracted feature amount with the information in the "characteristic pattern" field of the product information database (FIG. 5).

As a third example of step S102, when the user utters the product information (for example, product name), the voice input/output device 17 inputs the voice uttered by the user.
Processor 12 identifies the product name by analyzing the voice input by voice input/output device 17 .
The processor 12 refers to the product information database (FIG. 5) to identify the product ID associated with the identified product name.

After step S102, the client device 10 performs facial expression analysis (S103) on the facial expression information obtained in step S101.

As a first example of step S103, the processor 12 applies facial action coding system (hereinafter referred to as "FACS") theory to the facial image IMG101a included in the video data acquired in step S101. By performing image analysis based on the facial features, action unit information is generated regarding the level of change in motion for each part of the face. This action unit information is an example of an expression change index indicating a change in the user's expression with respect to the product.

As a second example of step S103, the processor 12 generates a facial muscle electromyogram (an example of a "facial expression change index") from the facial muscle signal acquired in step S101.

After step S103, the client device 10 performs emotion estimation (S104) based on the facial expression change index obtained in step S103.

As a first example of step S104, the processor 12 applies the action unit information of each part to an emotion estimation formula for estimating an emotion, so that each image frame constituting the video data has an emotion related to the user's emotion. Infer emotional information.
Emotion information includes, for example, the following information.
・Information about the type of emotion (for example, joy, sadness, anger, surprise, fear, or irritation) ・Information about the intensity of the emotion

As a second example of step S104, the storage device 11 stores reference information indicating the relationship between the facial electromyogram and the user's emotion. The processor 12 compares the electromyogram of facial muscles with the reference information stored in the storage device 11 to estimate emotion information related to the user's emotion.

As a third example of step S104, the processor 12 applies a predetermined algorithm to the emotion information estimated from the action unit information and the emotion information estimated from the facial electromyogram to obtain the action unit information and facial expression Estimate emotional information based on both electromyography.

After step S104, the client device 10 calculates a reaction score (S105) based on the facial expression change index obtained in step S104.
Specifically, the processor 12 refers to the product information database (FIG. 5) to identify information regarding the product name associated with the product ID identified in step S102.
The processor 12 calculates the reaction score by applying the emotion information estimated in step S104 (that is, the emotion information obtained from the change in the user's facial expression) to the reaction score calculation formula.
The processor 12 displays on the display 15 a screen P102 (FIG. 8) based on the emotion information estimated in step S104 and the reaction score calculated in step S105.

The screen P102 includes display objects A101, A102a, and A102b, and a button object B102.
The display object A101 is the same as the screen P101 (FIG. 7).
A graph image IMG102 is displayed in the display object A102a. The graph image IMG 102 shows transitions of emotional information (EMOTION LEVEL) for each image frame (F) constituting video data (that is, user time-series changes in emotions).
The product information specified in step S102 and the reaction score calculated in step S105 are displayed on the display object A102b.
The button object B102 is an object that receives a user instruction for executing the update request (S106).

After step S105, the client device 10 executes the update request (S106).
Specifically, when the user operates the button object B102, the processor 12 transmits update request data to the server 30. FIG.
The update request data contains the following information.
・Product ID specified in step S102
- The reaction score calculated in step S105

After step S106, the server 30 updates the database (S301).
Specifically, processor 32 refers to the user information database (FIG. 4) to identify user attribute information associated with the user ID included in the login request data.
The processor 32 refers to the product information database (FIG. 5) to identify a record associated with the product ID included in the update request data (hereinafter referred to as "update target record").
The processor 32 updates the value of the "reaction score" field of the record to be updated as follows.
- In the "overall average" field, the average value is updated based on the response score included in the update request data.
- In the "attribute average" field, the average value is updated based on the response score included in the update request data and the specified user attribute information.

According to this embodiment, the reaction score is calculated based on changes in the user's facial expression. As a result, it is possible to obtain an index of the user's objective evaluation of the product.

(2) Second Embodiment A second embodiment will be described. The second embodiment is an example of acquiring a user's reaction score to a product based on changes in the user's facial expression during conversation.

(2-1) Outline of Second Embodiment An outline of the second embodiment will be described. FIG. 9 is an explanatory diagram of the outline of the second embodiment.

As shown in FIG. 9, the client device 10-1 transmits a moving image including the user A's face to the server 30. As shown in FIG. The client device 10 - 2 transmits the moving image including the user B's face to the server 30 . As a result, user A of the client device 10-1 and user B of the client device 10-2 make video calls via the server 30. FIG.

When user A, who owns a product, uses client device 10-1 to capture a moving image including his/her own face, server 30 identifies product information related to the product to be evaluated.
When the server 30 succeeds in identifying the product information, the server 30 analyzes the image of the user A's face included in the moving image.
The server 30 calculates a reaction score, which is an index of user A's reaction to the product, based on the analysis result.

In this way, the server 30 calculates the reaction score of the user A to the product based on the analysis result of the face image of the user A during the conversation between the user A who owns the product and the user B.

As a result, it is possible to obtain an objective evaluation index of the product of the user A who is having a conversation with the user B about the product.

(2-2) Information processing Information processing according to the second embodiment will be described. FIG. 10 is a sequence diagram of information processing according to the second embodiment. 11 and 12 are diagrams showing screen examples displayed in the information processing of FIG.

As shown in FIG. 10, the client devices 10-1 and 10-2 execute login requests (S100) as in the first embodiment (FIG. 6).
After step S100, the server 30 performs login authentication (S300) as in the first embodiment.

After step S300, the client devices 10-1 and 10-2 initiate a video call sequence (S310) via the server 30. FIG.
Specifically, processor 12 of client device 10 - 1 transmits a moving image including user A's face to server 30 . Processor 32 transmits this moving image to client device 10-2.
Processor 12 of client device 10 - 2 transmits the moving image including user B's face to server 30 . The processor 32 transmits this moving image to the client device 10-1.

Processor 12 of client device 10-1 displays screen P110-1 (FIG. 11A) on display 15. FIG.
The screen P110-1 includes display objects A110-1a to A110-1b.
An image captured by the camera 16 of the client device 10-1 is displayed in the display object A110-1a. The images captured by the camera 16 include an image IMG110-1a of the user A's face and an image IMG110-1b of the product.
The moving image transmitted from the client device 10-2 is displayed on the display object A110-1b. This moving image includes an image IMG110-1c of the face of user B of client device 10-2.

Processor 12 of client device 10-2 displays screen P110-2 (FIG. 11B) on display 15. FIG.
Screen P110-2 includes display objects A110-2a to A110-2b.
An image captured by the camera 16 of the client device 10-2 is displayed in the display object A110-2a. The image captured by this camera 16 includes an image IMG110-2a of user B's face.
A moving image transmitted from the client device 10-1 is displayed on the display object A110-2b. This moving image includes an image IMG110-2b of the face of user A of client device 10-1 and an image IMG110-2c of a product.

After step S310, the server 30 executes product identification (S311).

As a first example of step S<b>311 , the processor 12 transmits to the server 30 user instructions (for example, product information for field objects included in the screen displayed on the display 15 ).
Processor 32 identifies the product ID of the product based on the user instruction sent by processor 12 .

As a second example of step S311, processor 32 extracts the feature amount of image IMG110-1b by applying feature amount analysis to product image IMG110-1b included in the video data acquired in step S310. do.
Processor 32 identifies the product ID of the product represented by image IMG110-1b by comparing the extracted feature quantity with the information in the “feature pattern” field of the product information database (FIG. 5).

As a third example of step S311, when the user utters the product information (for example, product name), the voice input/output device 17 inputs the voice uttered by the user.
The processor 12 transmits the voice input by the voice input/output device 17 to the server 30 .
Processor 32 identifies the product name by analyzing the voice transmitted by processor 12 .
Processor 32 identifies the product ID associated with the identified product name by referring to the product information database (FIG. 5).

After step S311, the server 30 performs face image analysis (S312).
Specifically, the processor 32 performs image analysis based on the FACS theory on the face image IMG110-1a included in the video data acquired in step S310, so that for each part of the user A's face, Generate action unit information about the level of movement change.

After step S312, the server 30 performs emotion estimation (S313).
Specifically, the processor 32 applies the action unit information of each part of the user A to the emotion estimation formula, thereby estimating emotion information related to the emotion of the user A for each image frame forming the video data.

After step S313, the server 30 performs reaction score calculation (S314).
Specifically, the processor 32 refers to the product information database (FIG. 5) to identify information regarding the product name associated with the product ID identified in step S311.
The processor 32 applies the user A's emotion information estimated in step S313 (that is, the emotion information corresponding to all the image frames forming the video data) to the reaction score calculation formula to calculate the user A's reaction score. to calculate
Processor 32 transmits information regarding the calculated reaction scores to client devices 10-1 and 10-2.

After step S314, the client devices 10-1 and 10-2 present reaction scores (S110).
Specifically, the processor 12 displays on the display 15 a screen P112 (FIG. 12) based on the reaction score transmitted from the server 30 in step S314.

The screen P112 includes display objects A112a-112b and a button object B112 in addition to the same objects (display objects A110-1a and A110-1b) as the screen P110-1 (FIG. 11).
A graph image IMG112a is displayed on the display object A112a. The graph image IMG112a shows transitions (i.e., 4 shows a time-series change in user A's feelings toward the product).
User A's reaction score is displayed on the display object A 112b.
The button object B112 is an object that receives a user's instruction to end shooting a moving image.

After step S110, the client device 10-1 executes the update request (S112).
Specifically, when user A operates button object B 112 , processor 12 transmits update request data to server 30 .

After step S112, the server 30 updates the database (S315).
Specifically, processor 32 refers to the user information database (FIG. 4) to identify user attribute information associated with the user ID included in the login request data.
The processor 32 refers to the product information database (FIG. 5) to identify a record associated with the product ID identified in step S311 (hereinafter referred to as "update target record").
The processor 32 updates the value of the "reaction score" field of the record to be updated as follows.
- In the "overall average" field, the average value is updated based on the reaction score calculated in step S314.
- In the "attribute average" field, the average value is updated based on the reaction score calculated in step S314 and the specified user attribute information.

A person's emotions appear more strongly during a conversation with another person. According to the second embodiment, the reaction score is calculated based on the change in facial expression of user A who is having a conversation with user B about products. As a result, it is possible to improve the accuracy of the user A's objective evaluation index for the product.

(2-3) Modification of Second Embodiment A modification of the second embodiment will be described.

(2-3-1) Modification 1 of Second Embodiment
Modification 1 of the second embodiment will be described. In Modification 1 of the second embodiment, reaction scores of both users in conversation are calculated. FIG. 13 is a diagram showing an example of a screen displayed in information processing according to modification 1 of the second embodiment.

The processor 32 of Modification 1 of the second embodiment calculates the reaction score of User A and the reaction score of User B in step S312.

The processor 12 of Modification 1 of the second embodiment displays a screen P114 (FIG. 13) on the display 15 in step S110.

Screen P114 includes display objects A114a-A114b in addition to the same objects as screen P112 (FIG. 12).
A graph image IMG114a is displayed on the display object A114a. The graph image IMG 114a shows changes in emotional information (EMOTION LEVEL) for each image frame (F) that constitutes the video data for each type of user B's emotion (for example, happiness (HAPPY) and sadness (SAD)) (that is, 4 shows a time-series change in user A's feelings toward the product).
User B's reaction score is displayed on the display object A 114b.

(2-3-2) Modification 2 of Second Embodiment
Modification 2 of the second embodiment will be described. In Modified Example 2 of the second embodiment, the reaction score is calculated by prioritizing the facial expression of the user who owns the product among the users who are in conversation.

In step S312, the processor 32 of Modification 2 of the second embodiment identifies a user who corresponds to any of the following (that is, a user who has a relatively high interest in the product) as the first user, and the first user A user other than the user is specified as a second user.
- The user who specified the product information - The user of the client device 10 whose video data contains the image of the product - The user who first uttered the sound of the product information

The emotion estimation formula stored in the storage device 31 includes a first weighting factor for the first user and a second weighting factor for the second user. The first weighting factor is greater than the second weighting factor. That is, in this emotion estimation formula, the reaction score is calculated with more emphasis placed on the facial expression of the first user than that of the second user.

The processor 32 generates first action unit information for the first user and second action unit information for the second user in step S312.

In step S313, the processor 32 estimates emotion information for each image frame forming the video data by applying the first action unit information and the second action unit information to the emotion estimation formula. In this case, the first action unit information is applied to the first weighting factor. The second action unit information is applied to the second weighting factor.

According to Modified Example 2 of the second embodiment, when calculating reaction scores based on changes in the facial expressions of a plurality of users, emphasis is placed on the facial expressions of user A who owns the product. As a result, it is possible to improve the accuracy of the user's objective evaluation index for the product.

(3) Third Embodiment A third embodiment will be described. The third embodiment is an example of acquiring a user's reaction score to a product based on changes in the user's facial expression during conversation with a virtual person (hereinafter referred to as "avatar").

(3-1) Outline of Third Embodiment An outline of the third embodiment will be described. FIG. 14 is an explanatory diagram of the outline of the third embodiment.

As shown in FIG. 14, the user of the client device 10 uses the client device 10 to take a moving image of the product and to have a conversation with the avatar.

The client device 10 identifies product information related to products to be evaluated.
The client device 10 analyzes the image of the user's face included in the moving image when the identification of the product information is successful.
The client device 10 calculates a reaction score, which is an index of the user's reaction to the product, based on the analysis result.

In this way, when the product information can be specified, the client device 10 calculates the reaction score of the user for the product corresponding to the product information based on the analysis result of the face image of the user who is talking with the avatar. calculate.
This makes it possible to obtain an objective evaluation index for the product.

(3-2) Information processing Information processing according to the third embodiment will be described. FIG. 15 is a sequence diagram of information processing according to the third embodiment. 16 is a diagram showing an example of a screen displayed in the information processing of FIG. 15. FIG.

As shown in FIG. 15, the client device 10 executes a login request (S100) as in the first embodiment (FIG. 6).
After step S100, the server 30 performs login authentication (S300) as in the first embodiment.

After step S300, the client device 10 presents the avatar (S120).
Specifically, processor 12 displays screen P120 on display 15 (FIG. 16).

Screen P120 includes a display object A120 in addition to the same objects as screen P101 (FIG. 7).
An avatar image IMG120 is displayed on the display object A120. The avatar presents a question to the user (for example, at least one of text and voice of the question "What kind of product?").

After step S120, the client device 10 acquires video data (S121).
Specifically, when the user operates the button object B101 and inputs an answer (for example, at least one of text and voice regarding the answer) to the avatar's question, the processor 12 generates video data. and store the video data and information about the answer in the storage device 11 . The video data includes moving images captured by the camera 16 and audio input by the audio input/output device 17 .

After step S121, the client device 10 executes steps S102 to S106 as in the first embodiment (FIG. 6).
After step S106, the server 30 executes step S301 as in the first embodiment.

A person's emotions appear more strongly during a conversation with another person. According to the third embodiment, the reaction score is calculated based on the change in facial expression of the user who is having a conversation with the avatar about the product, even if no other user is present. As a result, it is possible to improve the accuracy of the user's objective evaluation index for the product.

(4) Modification A modification of the present embodiment will be described.

(4-1) Modification 1
Modification 1 will be described. Modification 1 is an example of calculating a reaction score in consideration of changes in facial expression and voice.

When applying modification 1 to the first embodiment, the processor 12 extracts the audio feature amount included in the video data in step S104.
The processor 12 applies the action unit information and the voice feature amount to the emotion estimation formula to estimate emotion information related to the user's emotion for each image frame forming the video data.

According to Modification 1, since the reaction score is calculated in consideration of changes in facial expression and voice, it is possible to improve the accuracy of the reaction score compared to the case where only face image analysis is used.

Modification 1 is also applicable to the second and third embodiments.

(4-2) Modification 2
Modification 2 will be described. Modification 2 is an example of calculating a reaction score in consideration of changes in facial expressions and biometric information.

In Modified Example 2, the user is wearing a measurement device capable of measuring biological information. The biometric information includes, for example, at least one of the following.
・Heart rate ・EEG ・Body temperature ・Stress hormone levels

When applying modification 2 to the first embodiment, the processor 12 acquires biological information from the measuring device in step S104.
The processor 12 applies the action unit information and the biometric information to the emotion estimation formula to estimate emotion information related to the user's emotion for each image frame forming the video data.

According to Modification 2, since the reaction score is calculated in consideration of changes in facial expression and biometric information, it is possible to improve the accuracy of the reaction score compared to the case where only face image analysis is used.

Modification 2 is also applicable to the second and third embodiments.

(4-3) Modification 3
Modification 3 will be described. Modification 3 is an example of calculating a reaction score in consideration of changes in facial expression and inclination of the face.

When applying modification 3 to the first embodiment, the processor 12 generates tilt information about the amount of change in tilt of the face in step S103 in addition to the action unit information.
In step S104, the processor 12 applies the action unit information and the tilt information to the emotion estimation formula to estimate emotion information related to the user's emotion for each image frame forming the video data.

According to Modified Example 3, since the reaction score is calculated in consideration of changes in facial expression and inclination of the face, it is possible to improve the accuracy of the reaction score compared to the case where only face image analysis is used.

Modification 3 is also applicable to the second and third embodiments.

(4-4) Modification 4
Modification 4 will be described. Modification 4 is an example of calculating a reaction score based on a still image including a user's face image.

When the processor 12 of Modification 4 generates video data in step S101, in step S103, the processor 12 extracts an image frame that satisfies a predetermined condition from the image frames forming the video data.

An image frame that satisfies a predetermined condition is, for example, any of the following.
- An image frame at the time when the button object B102 is operated - An image frame with the largest facial expression change amount among the image frames constituting the video data - An image frame designated by the user (among the image frames constituting the video data) For example, the image frame of the image that the user felt "most expresses his feelings")

The processor 12 estimates emotion information related to the user's emotion by applying feature amount analysis to image frames that satisfy a predetermined condition.

Note that in Modification 4, step S101 can be omitted. In this case, the client device 10 obtains a still image including the image of the user's face instead of step S101.

According to Modification 4, the reaction score is calculated based on the still image analysis result. As a result, even from a still image (that is, an expression at a certain point in time), it is possible to obtain an index of the user's objective evaluation of the product.

(5) Summary of this embodiment This embodiment will be summarized.

The first aspect of this embodiment is
A means for acquiring facial expression information related to a user's facial expression (for example, a processor 12 that executes step S101);
A means for specifying product information about the product (for example, a processor 12 that executes step S102);
If product information is specified, means for analyzing changes in facial expression based on facial expression information (for example, processor 12 that executes step S103);
Means (e.g., processor 12 executing step S104) for calculating a reaction score, which is an indicator of the user's reaction to the product, based on the results of the analysis;
An information processing device (for example, the client device 10).

According to the first aspect, the reaction score is calculated based on changes in the user's facial expression. As a result, it is possible to obtain an index of the user's objective evaluation of the product.

A second aspect of the present embodiment is
the acquisition means acquires at least one of a moving image including a user's face image and a user's facial myocardium;
It is an information processing device.

The third aspect of this embodiment is
The acquisition means further acquires a moving image including an image of the product,
The specifying means specifies the product information based on the image of the product included in the acquired moving image.
It is an information processing device.

The fourth aspect of this embodiment is
A means (for example, a voice input/output device 17) for acquiring voice including product information uttered by at least one user,
The identifying means identifies the product information based on the acquired voice,
It is an information processing device.

The fifth aspect of this embodiment is
The analyzing means further analyzes the voice,
It is an information processing device.

According to the fifth aspect, the reaction score is calculated in consideration of the analysis result of the user's voice in addition to the image of the user's face. Thereby, the accuracy of the reaction score can be improved.

A sixth aspect of the present embodiment is
a means for acquiring biometric information about a user's biometric body;
The calculating means calculates a reaction score based on the analysis result and the acquired biometric information;
It is an information processing device.

According to the sixth aspect, the reaction score is calculated in consideration of the biometric information of the user in addition to the image of the user's face. Thereby, the accuracy of the reaction score can be improved.

The seventh aspect of this embodiment is
comprising means for determining the tilt of the face of the user looking at the product based on the results of the analysis;
the computing means refers to the slope and computes a response score;
It is an information processing device.

According to the seventh aspect, the reaction score is calculated in consideration of the inclination of the user's face in addition to the image of the user's face. Thereby, the accuracy of the reaction score can be improved.

The eighth aspect of the present embodiment is
means for presenting to the user at least one of an avatar image of the virtual user, a text message containing a question for the user, and a voice message containing a question for the user (e.g., the processor 12 executing step S120);
The analyzing means analyzes the face included in the moving image at the time of presentation,
It is an information processing device.

According to the eighth aspect, the reaction score is calculated based on the change in facial expression of the user who is having a conversation with the avatar about the product even if no other user is present. As a result, it is possible to improve the accuracy of the user's objective evaluation index for the product.

A ninth aspect of the present embodiment is
Means for acquiring a first moving image including a first image of the face of the first user who uses the product and a second moving image including a second image of the face of the second user who interacts with the first user (for example , a processor 32) for executing step S310;
A means for specifying product information about the product (for example, a processor 32 that executes step S311),
means for analyzing the first image when product information is identified (e.g., processor 32 executing step S312);
Means (e.g., processor 32 executing step S314) for calculating a reaction score, which is an indicator of the first user's reaction to the product, based on the results of the analysis;
An information processing device (for example, server 30).

According to the ninth aspect, the reaction score is calculated based on the change in the facial expression of user A who is having a conversation with user B about products. As a result, it is possible to improve the accuracy of the user A's objective evaluation index for the product.

A tenth aspect of the present embodiment is
the means for analyzing analyzes the first image and the second image of the face of the second user;
It is an information processing device.

According to the tenth aspect, reaction scores are calculated based on changes in facial expressions of a plurality of users. As a result, it is possible to improve the accuracy of the user's objective evaluation index for the product.

The eleventh aspect of this embodiment is
The first moving image includes an image of the product,
The identifying means identifies the product information based on the image of the product included in the acquired first moving image.
It is an information processing device.

The twelfth aspect of this embodiment is
the acquiring means acquires a first voice including product information uttered by the first user and a second voice of the second user;
The identifying means identifies the product information based on the acquired first voice,
It is an information processing device.

The thirteenth aspect of this embodiment is
the means for analyzing further analyzes the first sound and the second sound;
It is an information processing device.

According to the thirteenth aspect, a reaction score is calculated based on facial expressions and voices of a plurality of users. As a result, it is possible to improve the accuracy of the user's objective evaluation index for the product.

The fourteenth aspect of this embodiment is
comprising means for acquiring first biometric information relating to the biometrics of the first user and second biometric information of the second user;
the calculating means calculates a reaction score based on the analysis result and the obtained first and second biological information;
It is an information processing device.

According to the fourteenth aspect, a reaction score is calculated based on facial expressions and biometric information of a plurality of users. As a result, it is possible to improve the accuracy of the user's objective evaluation index for the product.

The fifteenth aspect of this embodiment is
a means for determining the inclination of the face of the first user looking at the product and the inclination of the face of the second user based on the result of the analysis;
the calculating means refers to the slope and calculates a response score;
It is an information processing device.

According to the fifteenth aspect, the reaction score is calculated based on the facial expressions and facial tilts of a plurality of users. As a result, it is possible to improve the accuracy of the user's objective evaluation index for the product.

The sixteenth aspect of this embodiment is
The means for calculating calculates a response score by applying a first weighting factor to the analysis result of the face image of the first user and applying a second weighting factor to the analysis result of the face image of the second user. ,
the first weighting factor is greater than the second weighting factor;
It is an information processing device.

According to the sixteenth aspect, when calculating reaction scores based on changes in facial expressions of a plurality of users, emphasis is placed on the facial expression of the user who owns the product. As a result, it is possible to improve the accuracy of the user's objective evaluation index for the product.

The seventeenth aspect of this embodiment is
A first client device 10-1 used by a first user who owns a product, a second client device 10-2 used by a second user who interacts with the first user, and a first client device 10-1 and a second client device 10-1. an information processing system 1 comprising a server 30 that executes a video call sequence (S310) with a client device 10-2,
The first client device 10-1 comprises means (for example, the processor 12 that executes step S310) for acquiring a first moving image including a first image of the face of the first user,
The second client device 10-2 comprises means for acquiring a second moving image including a second image of the face of the second user (eg, the processor 12 executing step S310),
The server 30
A means for acquiring the first moving image and the second moving image (for example, the processor 12 that performs step S310),
A means for specifying product information about the product (for example, a processor 12 that executes step S311),
If the product information is specified, means for analyzing the face image included in the first moving image (for example, a processor 12 that executes step S312);
The information processing system 1 includes means (for example, the processor 12 executing step S314) for calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result.

The eighteenth aspect of the present embodiment is
comprising means for acquiring a still image including an image of the user's face;
Equipped with means for identifying product information about the product,
provided with means for analyzing the still image when product information is specified;
means for calculating a reaction score, which is an indicator of the user's reaction to the product, based on the analysis results;
It is an information processing device.

According to the eighteenth aspect, the reaction score is calculated based on the user's facial expression at a certain time. As a result, even from a still image, it is possible to obtain an index of the user's objective evaluation of the product.

The nineteenth aspect of the present embodiment is
It is a program for causing a computer (for example, processor 12) to function as each means described above.

(6) Other modified examples

The storage device 11 may be connected to the client device 10 via the network NW. Storage device 31 may be connected to server 30 via network NW.

Each step of the information processing described above can be executed by either the client device 10 or the server 30 .

The display objects A102a and A102b of the screen P102 (FIG. 8) and the display objects A112a and A112b of the screen P112 (FIG. 12) can be omitted.
When omitting the display objects A102a and A102b, step S106 (FIG. 6) is also omitted. In this case, the client device 10 transmits update request data to the server 30 after step S105 without accepting a user instruction.
If display objects A112a and A112b are omitted, steps S110 and S112 (FIG. 10) are also omitted. In this case, the server 30 executes step S315 without communicating with the client device 10 after step S314.
This can prevent affecting the measurement (eg, impairing the objectivity of the emotional information and the reaction score) by presenting the emotional information and the reaction score during the measurement.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Also, the above embodiments can be modified and modified in various ways without departing from the gist of the present invention. Also, the above embodiments and modifications can be combined.

1 : Information processing system 10 : Client device 11 : Storage device 12 : Processor 13 : Input/output interface 14 : Communication interface 15 : Display 16 : Camera 17 : Audio input/output device 30 : Server 31 : Storage device 32 : Processor 33 : Input Output interface 34: communication interface

Claims (12)

means for acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; prepared,
comprising means for specifying product information related to the product;
comprising means for analyzing the first image when the product information is specified;
means for calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
The information processing apparatus, wherein the analyzing means analyzes the first image and the second image of the face of the second user. The first moving image includes an image of the product,
2. The information processing apparatus according to claim 1, wherein said specifying means specifies said product information based on an image of a product included in said acquired first moving image. a means for acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; prepared,
comprising means for specifying product information related to the product;
comprising means for analyzing the first image when the product information is specified;
means for calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
The acquiring means acquires a first voice containing product information uttered by the first user and a second voice of the second user,
The specifying means specifies the product information based on the acquired first voice,
the means for analyzing further analyzes the first sound and the second sound;
Information processing equipment. a means for acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; prepared,
comprising means for specifying product information related to the product;
comprising means for analyzing the first image when the product information is specified;
means for calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
comprising means for acquiring first biometric information related to the biometrics of the first user and second biometric information of the second user;
the calculating means calculates the reaction score based on the analysis result and the obtained first and second biological information;
Information processing equipment. means for acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; prepared,
comprising means for specifying product information related to the product;
comprising means for analyzing the first image when the product information is specified;
means for calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
means for determining the inclination of the face of the first user looking at the product and the inclination of the face of the second user based on the analysis result;
The information processing apparatus, wherein the calculating means calculates the reaction score by referring to the inclination. means for acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; prepared,
comprising means for specifying product information related to the product;
comprising means for analyzing the first image when the product information is specified;
means for calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
The calculating means applies a first weighting factor to an analysis result of the face image of the first user and a second weighting factor to an analysis result of the face image of the second user to obtain the reaction calculate the score,
the first weighting factor is greater than the second weighting factor;
Information processing equipment. the computer
Acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; run ,
performing the step of identifying merchandise information about the merchandise;
if the product information is identified, analyzing the first image;
performing a step of calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
The information processing method, wherein the analyzing step analyzes the first image and a second image of the face of the second user. the computer
Acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; run ,
performing the step of identifying merchandise information about the merchandise;
if the product information is identified, analyzing the first image;
performing a step of calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
The acquiring step acquires a first voice containing product information uttered by the first user and a second voice of the second user,
The identifying step identifies the product information based on the acquired first voice,
the analyzing step further analyzes the first audio and the second audio;
Information processing methods. the computer
Acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; run ,
performing the step of identifying merchandise information about the merchandise;
if the product information is identified, analyzing the first image;
performing a step of calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
performing a step of acquiring first biometric information related to the biometrics of the first user and second biometric information of the second user;
The calculating step calculates the reaction score based on the result of the analysis and the obtained first and second biological information.
Information processing methods. the computer
Acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; run ,
performing the step of identifying merchandise information about the merchandise;
if the product information is identified, analyzing the first image;
performing a step of calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
performing a step of determining the tilt of the face of the first user looking at the product and the tilt of the face of the second user, based on the result of the analysis;
The information processing method, wherein the calculating step refers to the slope and calculates the reaction score. the computer
Acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user who interacts with the first user; run ,
performing the step of identifying merchandise information about the merchandise;
if the product information is identified, analyzing the first image;
performing a step of calculating a reaction score, which is an index of the first user's reaction to the product, based on the analysis result;
The calculating step applies a first weighting factor to an analysis result of the facial image of the first user and a second weighting factor to an analysis result of the second user's facial image, thereby calculate the score,
the first weighting factor is greater than the second weighting factor;
Information processing methods. A program for causing a computer to function as each means according to any one of claims 1 to 6.

Images