JPWO2019146405A1 - Information processing equipment, information processing systems, and programs for evaluating the reaction of monitors to products using facial expression analysis technology. - Google Patents

Abstract

The information processing device includes means for acquiring facial expression information related to the user's facial expression, includes means for specifying product information related to the product, and when the product information is specified, provides means for analyzing changes in facial expressions based on the facial expression information. A means for calculating a reaction score, which is an index of a user's reaction to a product, is provided based on the result of analysis.

Description

The present invention relates to an information processing device, an information processing system, and a program.

In general, market research on products uses a questionnaire to consumers to aggregate the evaluation of products. By obtaining responses from a large number of consumers, it can be expected that the accuracy of product evaluation will improve.

Japanese Unexamined Patent Publication No. 2003-308415 discloses a technique for easily acquiring many questionnaire results by using a system including a user terminal and a server.

In general, it is preferable that the evaluation of a product has objectivity.
However, the questionnaire results of JP-A-2003-308415 reflect the subjectivity of consumers. Therefore, the evaluation obtained from the questionnaire results does not have objectivity for the product.

An object of the present invention is to obtain an index for objective evaluation of a product.

One aspect of the present invention is
Equipped with a means to acquire facial expression information related to the user's facial expression
With means to identify product information about the product
When the product information is specified, a means for analyzing the change in the facial expression based on the facial expression information is provided.
A means for calculating a reaction score, which is an index of the user's reaction to the product, based on the result of the analysis.
It is an information processing device.

According to the present invention, it is possible to obtain an index of an objective evaluation of a product by a user.

It is the schematic of the information processing system of 1st Embodiment. It is a block diagram which shows the structure of the information processing system of 1st Embodiment. It is explanatory drawing of the outline of 1st Embodiment. It is a figure which shows the data structure of the user information database of 1st Embodiment. It is a figure which shows the data structure of the product information database of 1st Embodiment. It is a sequence diagram of information processing of 1st Embodiment. It is a figure which shows the screen example displayed in the information processing of FIG. It is a figure which shows the screen example displayed in the information processing of FIG. It is explanatory drawing of the outline of 2nd Embodiment. It is a sequence diagram of information processing of the second embodiment. It is a figure which shows the screen example displayed in the information processing of FIG. It is a figure which shows the screen example displayed in the information processing of FIG. It is a figure which shows the screen example displayed in the information processing of the modification 1 of the 2nd Embodiment. It is explanatory drawing of the outline of 3rd Embodiment. It is a sequence diagram of information processing of the third embodiment. It is a figure which shows the screen example displayed in the information processing of FIG.

Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. In addition, in the drawing for demonstrating the embodiment, the same components are in principle the same reference numerals, and the repeated description thereof will be omitted.

(1) First Embodiment The first embodiment will be described.

(1-1) Configuration of Information Processing System The configuration of the information processing system of the first embodiment will be described. FIG. 1 is a schematic view of the information processing system of the first embodiment. FIG. 2 is a block diagram showing a configuration of the information processing system of the first embodiment.

As shown in FIGS. 1 and 2, the information processing system 1 includes a client device 10 and a server 30.
The client device 10 and the server 30 are connected via a network (for example, the Internet or an intranet) NW.

The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a web server.

(1-1-1) Configuration of Client Device The configuration of the client device 10 will be described.

As shown in FIG. 2, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, a communication interface 14, a display 15, a camera 16, and an audio input / output device 17.

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
・ OS (Operating System) program ・ Application (for example, web browser) program that executes information processing

The data includes, for example, the following data.
-Database referenced in information processing-Data obtained by executing information processing (that is, the execution result of information processing)

The processor 12 is configured to realize the function of the client device 10 by activating the program stored in the storage device 11. The processor 12 is an example of a computer.

The input / output interface 13 is configured to acquire a user's instruction from an input device connected to the client device 10 and output information to an output device connected to the client device 10.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.

The communication interface 14 is configured to control communication between the client device 10 and the server 30.

The display 15 is configured to display an image. The display 15 is, for example, any of the following.
・ Liquid crystal display ・ Organic EL (Electro Luminescence) display

The camera 16 is configured to capture an image and generate image data of the captured image. The camera 16 has, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The audio input / output device 17 is configured to input / output audio. The voice input / output device 17 is a combination of a microphone for inputting voice and a speaker for outputting voice.

(1-1-2) Server Configuration The configuration of the server 30 will be described.

As shown in FIG. 2, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface 34.

The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of a ROM, a RAM, and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
・ OS program ・ Application program that executes information processing

The data includes, for example, the following data.
・ Database referred to in information processing ・ Execution result of information processing

The processor 32 is configured to realize the function of the server 30 by activating the program stored in the storage device 31. The processor 32 is an example of a computer.

The input / output interface 33 is configured to acquire a user's instruction from an input device connected to the server 30 and output information to an output device connected to the server 30.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display.

The communication interface 34 is configured to control communication between the server 30 and the client device 10.

(1-2) Outline of First Embodiment The outline of the first embodiment will be described. FIG. 3 is an explanatory diagram of an outline of the first embodiment.

As shown in FIG. 3, the client device 10 acquires facial expression information (for example, a moving image or a facial expression EMG) related to the user's facial expression.

The client device 10 specifies product information about the product to be evaluated.
When the client device 10 succeeds in specifying the product information, the client device 10 calculates the facial expression change index, which is an index related to the change in the user's facial expression, by analyzing the facial expression information.
The client device 10 calculates a reaction score, which is an index related to the user's reaction to the product, based on the facial expression change index obtained by the analysis.

In this way, when the product information can be specified, the client device 10 calculates the user's reaction score to the product based on the change in the user's facial expression.
As a result, it is possible to obtain an index for objective evaluation of the product.

(1-3) Database The database of the first embodiment will be described. The following database is stored in the storage device 31.

(1-3-1) User Information Database The user information database of the first embodiment will be described. FIG. 4 is a diagram showing a data structure of the user information database of the first embodiment.

The user information database of FIG. 4 stores user information about the user.
The user information database includes a "user ID" field, a "user name" field, and a "user attribute" field. Each field is associated with each other.

A user ID that identifies a user is stored in the "User ID" field.

Information about the user's name (eg, text) is stored in the Username field.

The "user attribute" field stores user attribute information related to the user's attribute.
The "User Attributes" field includes a "Gender" field and an "Age" field.

The Gender field stores information about the user's gender.

The Age field stores information about the user's age.

(1-3-2) Product Information Database The product information database of the first embodiment will be described. FIG. 5 is a diagram showing a data structure of the product information database of the first embodiment.

The product information database of FIG. 5 stores product information related to the product.
The product information database includes a "product ID" field, a "product name" field, a "product image" field, a "feature pattern" field, a "product attribute" field, and a "reaction score" field. Each field is associated with each other.

A product ID that identifies a product is stored in the "product ID" field.

The "Product Name" field stores information about the product name (eg, text).

The "product image" field stores image information of the product (for example, at least one of the product body, the product package, and the usage state of the product).

The "feature pattern" field stores information about the feature pattern on the image of the product.

Product attribute information related to the product attribute is stored in the "product attribute" field.
The "Product Attributes" field includes a "Type" field and a "Manufacturer" field.

The "type" field stores information about the type of product (eg, lipstick or lotion).

The "Manufacturer" field stores information about the manufacturer of the product (eg, text indicating the manufacturer's name).

In the "reaction score" field, a numerical value (hereinafter referred to as "reaction score") indicating the level of the user's reaction to the product is stored.
The Response Score field includes an Overall Average field and an Attribute Average field.

The "overall average" field stores the average value of the reaction scores of all users.

In the "attribute average" field, the average value of the reaction score for each user attribute is stored.
The Attribute Mean field includes a Gender field and an Age field.

The "Gender" field stores the average response score for each gender.

The "age" field stores the average response score for each age.

(1-4) Information Processing The information processing of the first embodiment will be described. FIG. 6 is a sequence diagram of information processing of the first embodiment. 7 to 8 are diagrams showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 6, the client device 10 executes a login request (S100).
Specifically, the processor 12 displays the screen P100 (FIG. 7) on the display 15.

The screen P100 includes a button object B100 and a field object F100.
The field object F100 is an object that accepts input of a user ID.
The button object B100 is an object that receives a user instruction for login.

When the user inputs the user ID into the field object F100 and operates the button object B100, the processor 12 transmits the login request data to the server 30.
The login request data includes the user ID entered in the field object F100.

After step S100, the server 30 executes login authentication (S300).
Specifically, when the user ID included in the login request data is included in the user information database (FIG. 4), the processor 32 transmits the login response data for authenticating the login to the client device 10.

After step S300, the client device 10 executes the acquisition of facial expression information (S101).

As a first example of step S101, when the processor 12 receives the login response data, the processor 12 displays the screen P101 (FIG. 7) on the display 15.

The screen P101 includes a display object A101 and a button object B101.
The image captured by the camera 16 is displayed on the display object A101. The image captured by the camera 16 includes an image IMG101a of the user's face and an image IMG101b of the product.
The button object B101 is an object that receives a user instruction for acquiring video data.

When the user operates the button object B101, the processor 12 generates video data (an example of "facial expression information") and stores the video data in the storage device 11. The video data includes a moving image captured by the camera 16 and audio input by the audio input / output device 17.

As a second example of step S101, the client device 10 is connected to the measuring device. The measuring device is configured to, for example, acquire a facial muscle signal relating to the movement of the facial muscle from an electrode attached to the user's face. The processor 12 acquires the facial muscle signal from the measuring device.

After step S101, the client device 10 executes the product identification (S102).

As a first example of step S102, the processor 12 identifies a product ID of a product based on a user instruction (for example, product information for a field object included in a screen displayed on the display 15).

As a second example of step S102, the processor 12 extracts the feature amount of the image IMG101b by applying the feature amount analysis to the image IMG101b of the product included in the video data acquired in step S101.
The processor 12 identifies the product ID of the product represented by the image IMG101b by comparing the extracted feature amount with the information in the "feature pattern" field of the product information database (FIG. 5).

As a third example of step S102, when the user emits a voice of product information (for example, a product name), the voice input / output device 17 inputs the voice emitted by the user.
The processor 12 identifies the product name by analyzing the voice input by the voice input / output device 17.
The processor 12 refers to the product information database (FIG. 5) and identifies the product ID associated with the specified product name.

After step S102, the client device 10 executes facial expression analysis (S103) on the facial expression information obtained in step S101.

As a first example of step S103, the processor 12 applies the facial action coding system (hereinafter referred to as “FACS” (Facial Action Coding System)) theory to the facial image IMG101a included in the video data acquired in step S101. By performing the based image analysis, action unit information regarding the level of change in movement is generated for each part of the face. This action unit information is an example of a facial expression change index indicating a change in the user's facial expression with respect to the product.

As a second example of step S103, the processor 12 generates a facial expression EMG (an example of a "facial expression change index") from the facial expression muscle signal acquired in step S101.

After step S103, the client device 10 executes emotion estimation (S104) based on the facial expression change index obtained in step S103.

As a first example of step S104, the processor 12 applies the action unit information of each part to the emotion estimation formula for estimating emotions, so that the user's emotions are related to each image frame constituting the video data. Estimate emotional information.
The emotional information includes, for example, the following information.
・ Information on the type of emotion (for example, joy, sadness, anger, surprise, fear, or irritation) ・ Information on the intensity of emotion

As a second example of step S104, the storage device 11 stores reference information indicating the relationship between the facial EMG and the user's emotion. The processor 12 estimates emotional information about the user's emotions by comparing the facial EMG with the reference information stored in the storage device 11.

As a third example of step S104, the processor 12 applies the action unit information and the facial expression to the emotion information estimated from the action unit information and the emotion information estimated from the facial EMG. Estimate emotional information based on both EMGs.

After step S104, the client device 10 executes the calculation of the reaction score (S105) based on the facial expression change index obtained in step S104.
Specifically, the processor 12 refers to the product information database (FIG. 5) to specify information regarding the product name associated with the product ID specified in step S102.
The processor 12 calculates the reaction score by applying the emotion information estimated in step S104 (that is, the emotion information obtained from the change in the user's facial expression) to the reaction score calculation formula.
The processor 12 displays the emotion information estimated in step S104 and the screen P102 (FIG. 8) based on the reaction score calculated in step S105 on the display 15.

The screen P102 includes display objects A101, A102a, and A102b, and a button object B102.
The display object A101 is the same as the screen P101 (FIG. 7).
The graph image IMG102 is displayed on the display object A102a. The graph image IMG102 is a transition (that is, a user for a product) of emotion information (EMOTION LEVEL) for each image frame (F) constituting video data for each type of emotion (for example, joy (HAPPY) and sadness (SAD)). (Time-series changes in emotions) are shown.
The product information specified in step S102 and the reaction score calculated in step S105 are displayed on the display object A102b.
The button object B102 is an object that receives a user instruction for executing the update request (S106).

After step S105, the client device 10 executes the update request (S106).
Specifically, when the user operates the button object B102, the processor 12 transmits update request data to the server 30.
The update request data includes the following information.
-Product ID specified in step S102
-Reaction score calculated in step S105

After step S106, the server 30 executes the database update (S301).
Specifically, the processor 32 refers to the user information database (FIG. 4) and identifies the user attribute information associated with the user ID included in the login request data.
The processor 32 refers to the product information database (FIG. 5) and identifies a record (hereinafter referred to as “update target record”) associated with the product ID included in the update request data.
The processor 32 updates the value in the "reaction score" field of the update target record as follows.
-In the "Overall average" field, the average value is updated based on the reaction score included in the update request data.
-In the "attribute average" field, the average value is updated based on the reaction score included in the update request data and the specified user attribute information.

According to the present embodiment, the reaction score is calculated based on the change in the facial expression of the user. As a result, it is possible to obtain an index of the user's objective evaluation of the product.

(2) Second Embodiment The second embodiment will be described. The second embodiment is an example of acquiring the user's reaction score to the product based on the change in the user's facial expression during the conversation.

(2-1) Outline of the Second Embodiment The outline of the second embodiment will be described. FIG. 9 is an explanatory diagram of an outline of the second embodiment.

As shown in FIG. 9, the client device 10-1 transmits a moving image including the face of the user A to the server 30. The client device 10-2 transmits a moving image including the face of the user B to the server 30. As a result, the user A of the client device 10-1 and the user B of the client device 10-2 make a video call via the server 30.

When the user A who owns the product takes a moving image including his / her face using the client device 10-1, the server 30 identifies the product information regarding the product to be evaluated.
When the server 30 succeeds in specifying the product information, the server 30 analyzes the image of the face of the user A included in the moving image.
Based on the result of the analysis, the server 30 calculates the reaction score, which is an index regarding the reaction of the user A to the product.

In this way, the server 30 calculates the reaction score of the user A to the product based on the result of the analysis of the face image of the user A during the conversation between the user A who owns the product and the user B.

As a result, it is possible to obtain an index of objective evaluation of the product of the user A who is talking with the user B about the product.

(2-2) Information Processing The information processing of the second embodiment will be described. FIG. 10 is a sequence diagram of information processing of the second embodiment. 11 to 12 are diagrams showing an example of a screen displayed in the information processing of FIG. 10.

As shown in FIG. 10, the client devices 10-1 and 10-2 execute the login request (S100) as in the first embodiment (FIG. 6).
After step S100, the server 30 executes login authentication (S300) as in the first embodiment.

After step S300, client devices 10-1 and 10-2 initiate a video call sequence (S310) via the server 30.
Specifically, the processor 12 of the client device 10-1 transmits a moving image including the face of the user A to the server 30. The processor 32 transmits this moving image to the client device 10-2.
The processor 12 of the client device 10-2 transmits a moving image including the face of the user B to the server 30. The processor 32 transmits this moving image to the client device 10-1.

The processor 12 of the client device 10-1 displays the screen P110-1 (FIG. 11A) on the display 15.
The screen P110-1 includes display objects A110-1a to A110-1b.
The image captured by the camera 16 of the client device 10-1 is displayed on the display object A110-1a. The image captured by the camera 16 includes an image IMG110-1a of the face of the user A and an image IMG110-1b of the product.
The moving image transmitted from the client device 10-2 is displayed on the display object A110-1b. This moving image includes an image IMG110-1c of the face of user B of the client device 10-2.

The processor 12 of the client device 10-2 displays the screen P110-2 (FIG. 11B) on the display 15.
The screen P110-2 includes display objects A110-2a to A110-2b.
The image captured by the camera 16 of the client device 10-2 is displayed on the display object A110-2a. The image captured by the camera 16 includes the image IMG110-2a of the face of the user B.
The moving image transmitted from the client device 10-1 is displayed on the display object A110-2b. This moving image includes an image IMG110-2b of the face of the user A of the client device 10-1 and an image IMG110-2c of the product.

After step S310, the server 30 executes the product identification (S311).

As a first example of step S311 the processor 12 transmits a user instruction (for example, product information for a field object included in the screen displayed on the display 15) to the server 30.
The processor 32 identifies the product ID of the product based on the user instruction transmitted by the processor 12.

As a second example of step S311, the processor 32 extracts the feature amount of the image IMG110-1b by applying the feature amount analysis to the image IMG110-1b of the product included in the video data acquired in step S310. To do.
The processor 32 identifies the product ID of the product represented by the image IMG110-1b by comparing the extracted feature amount with the information in the "feature pattern" field of the product information database (FIG. 5).

As a third example of step S311, when the user emits a voice of product information (for example, a product name), the voice input / output device 17 inputs the voice emitted by the user.
The processor 12 transmits the voice input by the voice input / output device 17 to the server 30.
The processor 32 identifies the product name by analyzing the voice transmitted by the processor 12.
The processor 32 identifies the product ID associated with the specified product name with reference to the product information database (FIG. 5).

After step S311 the server 30 executes face image analysis (S312).
Specifically, the processor 32 performs image analysis based on the FACS theory on the face image IMG110-1a included in the video data acquired in step S310, so that the user A can perform image analysis for each facial part. Generates action unit information about the level of change in motion.

After step S312, the server 30 performs emotion estimation (S313).
Specifically, the processor 32 estimates the emotion information related to the emotion of the user A for each image frame constituting the video data by applying the action unit information of each part of the user A to the emotion estimation formula.

After step S313, the server 30 performs a reaction score calculation (S314).
Specifically, the processor 32 refers to the product information database (FIG. 5) to specify information about the product name associated with the product ID identified in step S311.
The processor 32 applies the emotion information of the user A estimated in step S313 (that is, the emotion information corresponding to all the image frames constituting the video data) to the calculation formula of the reaction score, so that the reaction score of the user A To calculate.
The processor 32 transmits the calculated response score information to the client devices 10-1 and 10-2.

After step S314, client devices 10-1 and 10-2 perform response score presentation (S110).
Specifically, the processor 12 displays the screen P112 (FIG. 12) based on the reaction score transmitted from the server 30 in step S314 on the display 15.

The screen P112 includes display objects A112a to 112b and button objects B112 in addition to the same objects (display objects A110-1a and A110-1b) as the screen P110-1 (FIG. 11).
The graph image IMG112a is displayed on the display object A112a. The graph image IMG112a is a transition (that is, EMOTION LEVEL) of emotion information (EMOTION LEVEL) for each image frame (F) constituting video data for each type of emotion (for example, joy (HAPPY) and sadness (SAD)) of user A. The time-series change of user A's feelings toward the product) is shown.
The reaction score of the user A is displayed on the display object A112b.
The button object B112 is an object that receives a user instruction for ending the shooting of a moving image.

After step S110, the client device 10-1 executes the update request (S112).
Specifically, when the user A operates the button object B112, the processor 12 transmits the update request data to the server 30.

After step S112, the server 30 executes the database update (S315).
Specifically, the processor 32 refers to the user information database (FIG. 4) and identifies the user attribute information associated with the user ID included in the login request data.
The processor 32 refers to the product information database (FIG. 5) and identifies a record (hereinafter referred to as “update target record”) associated with the product ID specified in step S311.
The processor 32 updates the value in the "reaction score" field of the update target record as follows.
-In the "overall average" field, the average value is updated based on the reaction score calculated in step S314.
-In the "attribute average" field, the average value is updated based on the reaction score calculated in step S314 and the specified user attribute information.

A person's emotions are more pronounced during a conversation with another person. According to the second embodiment, the reaction score is calculated based on the change in the facial expression of the user A who talks with the user B about the product. As a result, the accuracy of the index of the objective evaluation of the user A for the product can be improved.

(2-3) Modified Example of Second Embodiment A modified example of the second embodiment will be described.

(2-3-1) Modification 1 of the second embodiment
A modification 1 of the second embodiment will be described. In the first modification of the second embodiment, the reaction scores of both users in conversation are calculated. FIG. 13 is a diagram showing an example of a screen displayed in the information processing of the first modification of the second embodiment.

The processor 32 of the first modification of the second embodiment calculates the reaction score of the user A and the reaction score of the user B in step S312.

In step S110, the processor 12 of the first modification of the second embodiment displays the screen P114 (FIG. 13) on the display 15.

The screen P114 includes display objects A114a to A114b in addition to the same objects as the screen P112 (FIG. 12).
The graph image IMG114a is displayed on the display object A114a. The graph image IMG114a is a transition (that is, EMOTION LEVEL) of emotion information (EMOTION LEVEL) for each image frame (F) constituting video data for each type of emotion (for example, joy (HAPPY) and sadness (SAD)) of user B. The time-series change of user A's feelings toward the product) is shown.
The reaction score of the user B is displayed on the display object A114b.

(2-3-2) Modification 2 of the second embodiment
A modification 2 of the second embodiment will be described. In the second modification of the second embodiment, the reaction score is calculated by giving priority to the facial expression of the user who owns the product among the users in conversation.

In step S312, the processor 32 of the second embodiment of the second embodiment identifies a user corresponding to any of the following (that is, a user who has a relatively high interest in the product) as the first user, and the first user. Users other than are specified as the second user.
-User who specified the product information-User of the client device 10 whose video data includes the image of the product-User who first emitted the sound of the product information

The emotion estimation formula stored in the storage device 31 includes a first weighting coefficient for the first user and a second weighting coefficient for the second user. The first weighting coefficient is larger than the second weighting coefficient. That is, in this emotion estimation formula, the reaction score is calculated by giving more importance to the facial expression of the first user than the second user.

In step S312, the processor 32 generates the first action unit information of the first user and the second action unit information of the second user.

In step S313, the processor 32 estimates the emotion information for each image frame constituting the video data by applying the first action unit information and the second action unit information to the emotion estimation formula. In this case, the first action unit information is applied to the first weighting coefficient. The second action unit information is applied to the second weighting factor.

According to the second modification of the second embodiment, when the reaction score is calculated based on the changes in the facial expressions of a plurality of users, the facial expressions of the user A who owns the product are emphasized. As a result, the accuracy of the index of the user's objective evaluation of the product can be improved.

(3) Third Embodiment The third embodiment will be described. The third embodiment is an example of acquiring a user's reaction score to a product based on a change in the user's facial expression during a conversation with a virtual person (hereinafter referred to as "avatar").

(3-1) Outline of the Third Embodiment The outline of the third embodiment will be described. FIG. 14 is an explanatory diagram of an outline of the third embodiment.

As shown in FIG. 14, the user of the client device 10 takes a moving image of the product using the client device 10 and has a conversation with the avatar.

The client device 10 specifies product information about the product to be evaluated.
When the client device 10 succeeds in specifying the product information, the client device 10 analyzes the image of the user's face included in the moving image.
The client device 10 calculates a reaction score, which is an index of the user's reaction to the product, based on the result of the analysis.

In this way, when the product information can be specified, the client device 10 calculates the user's reaction score to the product corresponding to the product information based on the result of the analysis of the image of the face of the user who talks with the avatar. calculate.
As a result, it is possible to obtain an index for objective evaluation of the product.

(3-2) Information Processing The information processing of the third embodiment will be described. FIG. 15 is a sequence diagram of information processing according to the third embodiment. FIG. 16 is a diagram showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 15, the client device 10 executes the login request (S100) as in the first embodiment (FIG. 6).
After step S100, the server 30 executes login authentication (S300) as in the first embodiment.

After step S300, the client device 10 executes the presentation of the avatar (S120).
Specifically, the processor 12 displays the screen P120 on the display 15 (FIG. 16).

The screen P120 includes a display object A120 in addition to the same objects as the screen P101 (FIG. 7).
The image IMG120 of the avatar is displayed on the display object A120. The avatar presents a question to the user (eg, at least one of the text and voice of the question "What product?").

After step S120, the client device 10 executes the acquisition of video data (S121).
Specifically, when the user operates the button object B101 and inputs an answer (for example, at least one of text and voice regarding the answer) to the question of the avatar, the processor 12 generates video data. In addition, the video data and the information regarding the answer are stored in the storage device 11. The video data includes a moving image captured by the camera 16 and audio input by the audio input / output device 17.

After step S121, the client device 10 executes steps S102 to S106 in the same manner as in the first embodiment (FIG. 6).
After step S106, the server 30 executes step S301 as in the first embodiment.

A person's emotions are more pronounced during a conversation with another person. According to the third embodiment, the reaction score is calculated based on the change in the facial expression of the user who talks about the product with the avatar even in the absence of another user. As a result, the accuracy of the index of the user's objective evaluation of the product can be improved.

(4) Modification Example A modification of the present embodiment will be described.

(4-1) Modification 1
A modification 1 will be described. Modification 1 is an example of calculating a reaction score in consideration of changes in facial expressions and voice.

When the first embodiment is applied to the first embodiment, the processor 12 extracts the audio features included in the video data in step S104.
The processor 12 estimates the emotion information about the user's emotion for each image frame constituting the video data by applying the action unit information and the voice feature amount to the emotion estimation formula.

According to the first modification, since the reaction score is calculated in consideration of the change in facial expression and the voice, the accuracy of the reaction score can be improved as compared with the case where only the face image analysis is used.

The first modification is also applicable to the second embodiment and the third embodiment.

(4-2) Modification 2
A modification 2 will be described. Modification 2 is an example of calculating a reaction score in consideration of changes in facial expressions and biological information.

In the second modification, the user is equipped with a measuring device capable of measuring biological information. The biological information includes, for example, at least one of the following.
・ Heart rate ・ Brain wave ・ Body temperature ・ Stress hormone level

When the second modification is applied to the first embodiment, the processor 12 acquires biometric information from the measuring device in step S104.
By applying the action unit information and the biological information to the emotion estimation formula, the processor 12 estimates the emotion information related to the user's emotion for each image frame constituting the video data.

According to the second modification, since the reaction score is calculated in consideration of the change in facial expression and the biological information, the accuracy of the reaction score can be improved as compared with the case where only the face image analysis is used.

The second modification is also applicable to the second embodiment and the third embodiment.

(4-3) Modification 3
Modification 3 will be described. Modification 3 is an example of calculating the reaction score in consideration of the change in facial expression and the inclination of the face.

When the modification 3 is applied to the first embodiment, the processor 12 generates tilt information regarding the amount of change in the tilt of the face in addition to the action unit information in step S103.
In step S104, the processor 12 applies the action unit information and the tilt information to the emotion estimation formula to estimate the emotion information related to the user's emotion for each image frame constituting the video data.

According to the third modification, since the reaction score is calculated in consideration of the change in facial expression and the inclination of the face, the accuracy of the reaction score can be improved as compared with the case where only the face image analysis is used.

The third modification is also applicable to the second embodiment and the third embodiment.

(4-4) Modification 4
A modification 4 will be described. Modification 4 is an example of calculating the reaction score based on a still image including an image of the user's face.

When the processor 12 of the modification 4 generates the video data in step S101, the processor 12 extracts an image frame satisfying a predetermined condition from the image frames constituting the video data in step S103.

The image frame satisfying the predetermined condition is, for example, any of the following.
-Image frame at the time when the button object B102 is operated-Image frame having the largest change in facial expression among the image frames constituting the video data-Image frame specified by the user among the image frames constituting the video data ( For example, an image frame of an image that the user feels "the most expressive of his feelings")

The processor 12 estimates emotional information regarding the user's emotions by applying the feature amount analysis to the image frame satisfying a predetermined condition.

In the modified example 4, step S101 can be omitted. In this case, the client device 10 changes to step S101 and acquires a still image including an image of the user's face.

According to the fourth modification, the reaction score is calculated based on the analysis result of the still image. As a result, it is possible to obtain an index of the user's objective evaluation of the product even from the still image (that is, the facial expression at a certain point in time).

(5) Summary of the present embodiment A summary of the present embodiment will be given.

The first aspect of this embodiment is
A means for acquiring facial expression information regarding the user's facial expression (for example, a processor 12 that executes step S101) is provided.
A means for identifying product information about a product (for example, a processor 12 that executes step S102) is provided.
When the product information is specified, a means for analyzing the change in facial expression based on the facial expression information (for example, the processor 12 that executes step S103) is provided.
A means (for example, a processor 12 that executes step S104) for calculating a reaction score, which is an index of a user's reaction to a product, is provided based on the result of analysis.
An information processing device (for example, a client device 10).

According to the first aspect, the reaction score is calculated based on the change in the facial expression of the user. As a result, it is possible to obtain an index of the user's objective evaluation of the product.

The second aspect of this embodiment is
The means for acquiring is to acquire at least one of a moving image including an image of the user's face and a user's facial EMG.
It is an information processing device.

The third aspect of this embodiment is
The means to acquire is to further acquire a moving image including an image of the product,
The means for identifying is to identify the product information based on the image of the product included in the acquired moving image.
It is an information processing device.

The fourth aspect of this embodiment is
A means for acquiring voice including product information emitted by at least one user (for example, a voice input / output device 17) is provided.
The means of identification is to identify the product information based on the acquired voice.
It is an information processing device.

A fifth aspect of this embodiment is
The means of analysis is to further analyze the voice,
It is an information processing device.

According to the fifth aspect, the reaction score is calculated in consideration of the analysis result of the user's voice in addition to the image of the user's face. Thereby, the accuracy of the reaction score can be improved.

The sixth aspect of this embodiment is
Equipped with a means to acquire biological information about the user's living body
The means of calculation is to calculate the reaction score based on the result of analysis and the acquired biological information.
It is an information processing device.

According to the sixth aspect, the reaction score is calculated in consideration of the user's biological information in addition to the image of the user's face. Thereby, the accuracy of the reaction score can be improved.

The seventh aspect of this embodiment is
A means for determining the inclination of the face of the user who is viewing the product based on the result of the analysis is provided.
The means of calculation is to calculate the reaction score by referring to the slope.
It is an information processing device.

According to the seventh aspect, the reaction score is calculated in consideration of the inclination of the user's face in addition to the image of the user's face. Thereby, the accuracy of the reaction score can be improved.

The eighth aspect of this embodiment is
A means for presenting to the user at least one of a virtual user's avatar image, a text message containing a question to the user, and a voice message containing a question to the user (eg, a processor 12 performing step S120).
The means of analysis is to analyze the face included in the moving image at the time of presentation.
It is an information processing device.

According to the eighth aspect, the reaction score is calculated based on the change in the facial expression of the user who talks about the product with the avatar even in the absence of another user. As a result, the accuracy of the index of the user's objective evaluation of the product can be improved.

The ninth aspect of this embodiment is
Means for acquiring a first moving image including a first image of the face of a first user using a product and a second moving image including a second image of the face of a second user interacting with the first user (for example). A processor 32) that executes step S310.
A means for identifying product information about the product (for example, a processor 32 that executes step S311) is provided.
When the product information is specified, a means for analyzing the first image (for example, a processor 32 for executing step S312) is provided.
A means (for example, a processor 32 that executes step S314) for calculating a reaction score, which is an index regarding the reaction of the first user to the product, based on the result of the analysis is provided.
An information processing device (for example, a server 30).

According to the ninth aspect, the reaction score is calculated based on the change in the facial expression of the user A who talks with the user B about the product. As a result, the accuracy of the index of the objective evaluation of the user A for the product can be improved.

The tenth aspect of this embodiment is
The means for analysis is to analyze the first image and the second image of the face of the second user.
It is an information processing device.

According to the tenth aspect, the reaction score is calculated based on the change in facial expressions of a plurality of users. As a result, the accuracy of the index of the user's objective evaluation of the product can be improved.

The eleventh aspect of this embodiment is
The first moving image includes the image of the product and
The identification means specifies the product information based on the image of the product included in the acquired first moving image.
It is an information processing device.

The twelfth aspect of this embodiment is
The means for acquiring is to acquire the first voice including the product information emitted by the first user and the second voice of the second user.
The identification means identifies the product information based on the acquired first voice.
It is an information processing device.

The thirteenth aspect of this embodiment is
The means for analysis further analyzes the first voice and the second voice.
It is an information processing device.

According to the thirteenth aspect, the reaction score is calculated based on the facial expressions and voices of a plurality of users. As a result, the accuracy of the index of the user's objective evaluation of the product can be improved.

The fourteenth aspect of this embodiment is
A means for acquiring the first biological information regarding the living body of the first user and the second biological information of the second user is provided.
The means for calculation is to calculate the reaction score based on the result of the analysis and the acquired first biological information and second biological information.
It is an information processing device.

According to the fourteenth aspect, the reaction score is calculated based on the facial expressions and biological information of a plurality of users. As a result, the accuracy of the index of the user's objective evaluation of the product can be improved.

The fifteenth aspect of this embodiment is
A means for determining the inclination of the face of the first user who is viewing the product and the inclination of the face of the second user based on the result of the analysis is provided.
The means of calculation is to calculate the reaction score by referring to the slope.
It is an information processing device.

According to the fifteenth aspect, the reaction score is calculated based on the facial expressions and facial inclinations of the plurality of users. As a result, the accuracy of the index of the user's objective evaluation of the product can be improved.

The sixteenth aspect of this embodiment is
The calculation means calculates the reaction score by applying the first weighting coefficient to the analysis result of the face image of the first user and applying the second weighting coefficient to the analysis result of the face image of the second user. ,
The first weighting factor is larger than the second weighting factor,
It is an information processing device.

According to the 16th aspect, when calculating the reaction score based on the changes in the facial expressions of a plurality of users, the facial expressions of the users who own the products are emphasized. As a result, the accuracy of the index of the user's objective evaluation of the product can be improved.

The seventeenth aspect of this embodiment is
The first client device 10-1 used by the first user who owns the product, the second client device 10-2 used by the second user interacting with the first user, the first client device 10-1 and the second. An information processing system 1 including a server 30 that executes a video call sequence (S310) with and from a client device 10-2.
The first client device 10-1 includes means for acquiring a first moving image including a first image of the face of the first user (for example, a processor 12 that executes step S310).
The second client device 10-2 includes means for acquiring a second moving image including a second image of the face of the second user (for example, a processor 12 that executes step S310).
The server 30
A means for acquiring a first moving image and a second moving image (for example, a processor 12 for executing step S310) is provided.
A means for identifying product information about a product (for example, a processor 12 that executes step S311) is provided.
When the product information is specified, a means for analyzing the face image included in the first moving image (for example, the processor 12 for executing step S312) is provided.
The information processing system 1 includes means for calculating a reaction score (for example, a processor 12 that executes step S314), which is an index related to a reaction of a first user to a product based on the result of analysis.

The eighteenth aspect of this embodiment is
Equipped with a means to acquire a still image including an image of the user's face,
With means to identify product information about the product
When the product information is specified, a means for analyzing the still image is provided.
A means for calculating a reaction score, which is an index of the user's reaction to the product, based on the result of the analysis.
It is an information processing device.

According to the eighteenth aspect, the reaction score is calculated based on the facial expression of the user at a certain time point. As a result, it is possible to obtain an index of the user's objective evaluation of the product even from the still image.

The nineteenth aspect of this embodiment is
It is a program for making a computer (for example, a processor 12) function as each of the above means.

(6) Other modifications

The storage device 11 may be connected to the client device 10 via the network NW. The storage device 31 may be connected to the server 30 via the network NW.

Each step of the above information processing can be executed by either the client device 10 or the server 30.

The display objects A102a and A102b on the screen P102 (FIG. 8) and the display objects A112a and A112b on the screen P112 (FIG. 12) can be omitted.
When the display objects A102a and A102b are omitted, step S106 (FIG. 6) is also omitted. In this case, after step S105, the client device 10 transmits the update request data to the server 30 without accepting the user instruction.
When the display objects A112a and A112b are omitted, steps S110 and S112 (FIG. 10) are also omitted. In this case, after step S314, the server 30 executes step S315 without communicating with the client device 10.
This makes it possible to prevent the influence on the measurement (for example, damage to the objectivity of the emotional information and the reaction score) by showing the emotional information and the reaction score during the measurement.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Further, the above-described embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Moreover, the above-described embodiment and modification can be combined.

1: Information processing system 10: Client device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Display 16: Camera 17: Voice input / output device 30: Server 31: Storage device 32: Processor 33: On Output interface 34: Communication interface

Claims (19)

Equipped with a means to acquire facial expression information related to the user's facial expression
With means to identify product information about the product
When the product information is specified, a means for analyzing the change in the facial expression based on the facial expression information is provided.
A means for calculating a reaction score, which is an index of the user's reaction to the product, based on the result of the analysis.
Information processing device. The acquisition means acquires at least one of a moving image including an image of the user's face and a facial expression EMG of the user.
The information processing device according to claim 1. The acquisition means further acquires a moving image including an image of the product, and obtains the moving image.
The identifying means identifies the product information based on the image of the product included in the acquired moving image.
The information processing device according to claim 1 or 2. A means for acquiring a voice including product information issued by at least one user,
The identifying means identifies the product information based on the acquired voice.
The information processing device according to any one of claims 1 to 3. The means for analysis further analyzes the voice.
The information processing device according to claim 4. A means for acquiring biological information about the user's living body is provided.
The calculation means calculates the reaction score based on the result of the analysis and the acquired biological information.
The information processing device according to any one of claims 1 to 5. A means for determining the inclination of the face of the user who is viewing the product based on the result of the analysis is provided.
The calculating means calculates the reaction score with reference to the slope.
The information processing device according to any one of claims 1 to 6. A means for presenting at least one of a virtual user's avatar image, a text message containing a question to the user, and a voice message containing a question to the user to the user.
The analysis means analyzes the face included in the moving image at the time of the presentation.
The information processing device according to any one of claims 2 to 7. A means for acquiring a first moving image including a first image of the face of a first user who owns a product and a second moving image including a second image of the face of a second user interacting with the first user. Prepare,
Provided with means for identifying product information related to the product
When the product information is specified, a means for analyzing the first image is provided.
A means for calculating a reaction score, which is an index of a first user's reaction to the product, based on the result of the analysis.
Information processing device. The means for analysis analyzes the first image and the second image of the face of the second user.
The information processing device according to claim 9. The first moving image includes an image of the product.
The identifying means identifies the product information based on the image of the product included in the acquired first moving image.
The information processing device according to claim 9 or 10. The acquisition means acquires a first voice including product information emitted by the first user and a second voice of the second user.
The information processing device according to any one of claims 9 to 11, wherein the identifying means specifies the product information based on the acquired first voice. The means for analysis further analyzes the first voice and the second voice.
The information processing device according to claim 12. A means for acquiring the first biological information regarding the living body of the first user and the second biological information of the second user is provided.
The calculation means calculates the reaction score based on the result of the analysis and the acquired first biological information and second biological information.
The information processing device according to any one of claims 9 to 13. A means for determining the inclination of the face of the first user who is viewing the product and the inclination of the face of the second user based on the result of the analysis are provided.
The calculating means calculates the reaction score with reference to the slope.
The information processing device according to any one of claims 9 to 14. The calculation means applies the first weighting coefficient to the analysis result of the face image of the first user, and applies the second weighting coefficient to the analysis result of the face image of the second user. Calculate the score and
The first weighting coefficient is larger than the second weighting coefficient.
The information processing device according to any one of claims 9 to 15. Between the first client device used by the first user who owns the product, the second client device used by the second user interacting with the first user, and the first client device and the second client device. An information processing system that includes a server that executes a video call sequence.
The first client device includes means for acquiring a first moving image including a first image of the face of the first user.
The second client device includes means for acquiring a second moving image including a second image of the face of the second user.
The server
A means for acquiring the first moving image and the second moving image is provided.
Provided with means for identifying product information related to the product
When the product information is specified, a means for analyzing the face image included in the first moving image is provided.
A means for calculating a reaction score, which is an index related to the reaction of the first user to the product, based on the result of the analysis.
Information processing system. Equipped with a means to acquire a still image including an image of the user's face,
With means to identify product information about the product
When the product information is specified, a means for analyzing the still image is provided.
A means for calculating a reaction score, which is an index of the user's reaction to the product, based on the result of the analysis.
Information processing device. A program for operating a computer as each means according to any one of claims 1 to 18.