TW201945898A - Information processing device and program - Google Patents

Abstract

This information processing device is provided with: a means for presenting an image including a face to a user; a means for acquiring gaze information relating to movement of the gaze of the user; a means for calculating unevenness in the gaze of the user in the image, on the basis of the gaze information; a means for determining recommendation information relating to the image, on the basis of a gaze distribution; and a means for presenting the recommendation information.

Description

Information processing device and program

The invention relates to an information processing device and a program.

In recent years, with the increase in the lineup of cosmetics, there is a tendency for persons in charge of cosmetics sales shops or consumers to choose cosmetics that meet consumer preferences.

Therefore, a technique for generating a makeup simulation image by executing a makeup simulation using a face image of a consumer on a computer is becoming more and more popular (for example, Japanese Re-watch Publication No. 2015-029371).

[Problems to be solved by the invention]

However, even if human beings feel that they are in line with their own preferences at the time, they may feel that they are not in accordance with their preferences. It is because human preferences depend on human subjectivity, and human subjectivity is not specific.

For example, even if a makeup simulation image selected by the consumer by using the technique of the re-published publication No. 2015-029371 selects a makeup simulation image that conforms to his or her preferences, there is a feeling that the selected makeup simulation image does not match The situation of your own preference, or the makeup simulation image that feels unselected is more in line with your preference.

In this way, the person's assessment of the observed face reflects the person's subjectivity. Therefore, it is difficult to prompt people for a potential assessment of the face being observed.

The purpose of the present invention is to prompt the user to a potential assessment of the face the user observes.
[Technical means to solve the problem]

An aspect of the present invention is an information processing device having a mechanism for prompting a user with an image including a face;
An organization that obtains sight information related to the sight movement of the user;
Based on the sight information, a mechanism for calculating the sight distribution of users in the image;
Organizations that determine the recommended information related to the image based on the line of sight distribution; and organizations that suggest the information.
[Effect of the invention]

According to the present invention, a user can be prompted for a potential assessment of a face that the user observes.

Hereinafter, one embodiment of the present invention will be described in detail based on the drawings. In addition, in the drawings for explaining the embodiment, the same constituent elements are denoted by the same symbols in principle, and redundant descriptions are omitted.

(1) First Embodiment A first embodiment of the present invention will be described.

(1-1) Structure of Information Processing System The structure of the information processing system will be described. 1 to 2 are block diagrams showing the configuration of the information processing system of the first embodiment.

As shown in FIGS. 1 to 2, the information processing system 1 includes a client device 10, an eye tracker 20, and a server 30.
The client device 10 and the server 30 are connected via a network (such as the Internet or an intranet) NW.
The eye tracker 20 is connected to the client device 10.

The client device 10 is an example of an information processing device that sends a request to the server 30. The client device 10 is, for example, a smart phone, a tablet terminal, or a personal computer.
The user U can give a user instruction to the client device 10.

The eye tracker 20 is configured to detect the line of sight movement of the user U and generate an eye tracking signal related to the line of sight movement ("an example of line of sight information"). Specifically, the eye tracker 20 measures the coordinates showing the position of the user's line of sight at specific time intervals.
The eye tracker 20 sends an eye tracking signal to the client device 10.

The server 30 is an example of an information processing device that provides a response corresponding to a request sent by the client device 10 to the client device 10. The server 30 is, for example, a web server.

(1-1-1) Configuration of Client Device Referring to FIG. 1, the configuration of the client device 10 will be described.

As shown in FIG. 1, the client device 10 includes a memory device 11, a processor 12, an input / output interface 13, a communication interface 14, a display 15, and a camera 16.

The memory device 11 is configured to memorize programs and data. The memory device 11 is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a memory (such as a flash memory or a hard disk).

The programs include, for example, the following programs.
· OS (Operating System) programs · Applications that perform information processing (such as makeup simulation applications) programs

The data includes, for example, the following data.
· Data base for reference in information processing · Data obtained by performing information processing (ie, results of performing information processing)

The processor 12 is configured to implement a function of the client device 10 by activating a program stored in the memory device 11. The processor 12 is an example of a computer.

The input / output interface 13 is configured to obtain a user's instruction from an input device connected to the client device 10, output information to an output device connected to the client device 10, and obtain an eye tracking signal from the eye tracker 20.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, the display 15.

The communication interface 14 is configured to control communication between the client device 10 and the server 30.

The display 15 is configured to display an image generated by the processor 12.

The camera 16 is configured to capture an image (for example, a face image of a user of the client device 10).

(1-1-2) Configuration of Server The configuration of the server 30 will be described with reference to FIG. 1.

As shown in FIG. 1, the server 30 includes a memory device 31, a processor 32, an input / output interface 33, and a communication interface 34.

The memory device 31 is configured to memorize programs and data. The memory device 31 is a combination of, for example, ROM, RAM, and storage (such as a flash memory or a hard disk).

The programs include, for example, the following programs.
· OS program · Application program for information processing

The data includes, for example, the following data.
· Database for reference in information processing · Results of information processing

The processor 32 is configured to implement a function of the server 30 by activating a program stored in the memory device 31. The processor 32 is an example of a computer.

The input / output interface 33 is configured to obtain a user's instruction from an input device connected to the server 30 and output information to an output device connected to the server 30.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display.

The communication interface 34 is configured to control communication between the server 30 and the client device 10.

(1-2) Outline of the First Embodiment The outline of the first embodiment will be described. FIG. 3 is an explanatory diagram of the outline of the first embodiment.

As shown in FIG. 3, in the first embodiment, a simulation image SIMG including a face of a person (for example, user U) is presented. In the simulated image SIMG, as shown in the mirror image, the right eye RE of the user U is disposed to the right, and the left eye LE of the user U is disposed to the left.
Next, an eye-tracking signal related to the line-of-sight movement of the user U during the observation of the simulated image SIMG is generated.
Next, based on the eye-tracking signal, the line-of-sight distribution of the first region GR1 located to the right (that is, including the right eye RE) in the coordinate space (hereinafter referred to as "image space") of the pixels constituting the simulated image SIMG is calculated. , And the sight line distribution of the second area GR2 located on the left side (including the left eye LE).
Then, based on the line-of-sight distribution of the first region GR1 and the line-of-sight distribution of the second region GR2, suggestion information related to the potential evaluation of the simulated image SIMG by the user U is provided.

Thus, in the first embodiment, based on the line-of-sight distribution (ie, line-of-sight shift) of the user U in the image space of one simulated image SIMG, the prompt reflects the potential evaluation of the simulated image SIMG by the user U. Suggested information.

(1-3) Database The database of the first embodiment will be described. The following database is stored in the memory device 31.

(1-3-1) User information database The user information database of the first embodiment will be described. FIG. 4 is a diagram showing a data structure of a user information database in the first embodiment.

The user information database in FIG. 4 stores user information related to the user.
The user information database includes a "user ID" field, a "user name" field, a "user image" field, and a "user attribute" field. The fields are related to each other.

A user ID (an example of "user identification information") identifying a user is stored in the "user ID" field.

Information (such as text) related to the user's name is stored in the "User name" field.

The "user image" field stores image data including an image of the user's face.

The "user attribute" field includes a "gender" field, an "age" field, and a "occupation" field.

Store information related to the user ’s gender in the "Gender" field.

Store age-related information in the "Age" field.

Store information related to the user's occupation in the "Occupation" field.

(1-3-2) Makeup pattern information master table The makeup pattern information master table of the first embodiment will be described. FIG. 5 is a diagram showing a data structure of a makeup pattern information master table according to the first embodiment.

The makeup pattern information master table in FIG. 5 stores makeup pattern information related to the makeup pattern.
The makeup pattern information master table includes a "pattern ID" field, a "pattern name" field, and a "item" field.

A makeup pattern ID (an example of "makeup pattern identification information") for identifying the makeup pattern is stored in the "pattern ID" field.

Information (such as text) related to the pattern name of the makeup pattern is stored in the "Pattern name" field.

The item information related to the makeup items used in the makeup simulation is stored in the "Item" field. The "Item" field contains a "Category" field, an "Item ID" field, and a "Parameter" field. The "category" field, the "item ID" field, and the "parameter" field are related to each other.

In the "Category" field, store information related to the category of makeup items (such as powder, lipstick, and blush).

An item ID (an example of "makeup item identification information") identifying a makeup item is stored in the "item ID" field.

In the "Parameter" field, the makeup parameters related to the effect when using the makeup items are stored. The makeup parameters are, for example, image processing parameters (for example, color conversion parameters used to convert color information of pixels of the original image) applied to the original image (as an example, the face image of the user U).

(1-3-3) Simulation log information database The simulation log information database of the first embodiment will be described. FIG. 6 is a diagram showing a data structure of the analog log information database of the first embodiment.

The simulation log information database in FIG. 6 stores information related to the results of the makeup simulation in time series.
The simulation log information database includes a "Simulation ID" field, a "Simulation execution day" field, a "Simulation condition" field, an "Eye tracking signal" field, and a "Sight distribution" field.
The simulation log information database is associated with the user ID.

In the "Simulation ID" field, a simulation ID (an example of "Simulation Identification Information") identifying the makeup simulation is stored.

Information related to the execution day of the makeup simulation is stored in the "Simulation execution day" field.

Information related to the simulation conditions of makeup simulation is stored in the "Simulation Conditions" field. The "Simulation conditions" field includes a "original image" field and a "pattern" field.

The "original image" field stores image data of the original image (for example, the face image of the user U) that is the subject of the makeup simulation.

The makeup pattern ID of the makeup pattern applied to the image data in the "original image" field is stored in the "pattern" field.

An eye tracking signal generated by the eye tracker 20 is stored in the “eye tracking signal” field. The "eye tracking signal" field includes a "measurement coordinate" field and a "measurement time" field. The "measurement coordinates" field and the "measurement time" field are related to each other.

In the "Measurement Coordinates" field, measurement coordinates showing the position of the line of sight of the user in the image space are stored.

Measured time information related to the timing of the eye tracking signal generated by the eye tracker 20 is stored in the "Measured time" field. The measurement time information is, for example, the time that has elapsed since the eye tracker 20 started generating the eye tracking signal.
By combining the coordinates of the "measurement coordinates" field and the information of the "measurement time" field, the sight of a specific user U moves.

A line-of-sight distribution derived from the information of the "eye tracking signal" field is stored in the "line-of-sight distribution" field (an example of "line-of-sight parameter"). The line-of-sight distribution shows the line-of-sight distribution of the user in the image space of the makeup simulation image. The "line-of-sight distribution" field includes an "R distribution" field and an "L distribution" field.

In the image space in which the makeup simulation image is stored in the "R distribution" field, for the makeup simulation image, the user's line of sight in the first area to the right (that is, to the right of the observer) is distributed. rate.

In the image space in which the makeup simulation image is stored in the "L distribution" field, for the makeup simulation image, the user's line of sight in the second area to the left (that is, to the left of the observer) is distributed. rate.

(1-4) Information processing The information processing of the first embodiment will be described. FIG. 7 is a flowchart of information processing in the first embodiment. 8 to 10 are diagrams showing examples of screens displayed in the information processing of FIG. 7. FIG. 11 is a detailed flowchart of the calculation of the line-of-sight parameters of FIG. 7. FIG. 12 is an explanatory diagram of calculation of a line of sight parameter of FIG. 7.

As shown in FIG. 7, the client device 10 executes the determination of the original image (S100).
Specifically, the processor 12 displays a screen P100 (FIG. 8) on the display 15.

The picture P100 includes an image object IMG100.
The image object IMG100 contains an image of the face of the user U. The image object IMG100 is, for example, any of the following.
· Images stored in the memory device 11 in advance · Images reproduced from the image data in the "User Image" field of the user information database (Figure 4) · Photographed by the camera 16 connected to the input / output interface 13 Image

After step S100, the client device 10 performs generation of a makeup simulation image (S101).
Specifically, the processor 12 specifies an arbitrary record of the makeup pattern information master table (FIG. 5).
The processor 12 applies the specific recorded information of the "parameter" field to the simulated image SIMG determined in step S100, thereby generating a makeup simulated image.

After step S101, the client device 10 executes the prompt of the makeup simulation image (S102).
Specifically, the processor 12 displays a screen P101 (FIG. 8) on the display 15.

The screen P101 includes an image object IMG101.
The image object IMG101 is a makeup simulation image generated in step S101.

After step S102, the client device 10 performs calculation of the sight parameters (S103).

As shown in FIG. 11, the client device 10 obtains an eye tracking signal (S1030).
Specifically, the processor 12 obtains an eye tracking signal from the eye tracker 20.
The processor 12 associates the measurement coordinates with the measurement time based on the eye tracking signal and stores the measurement coordinates in the storage device 11.

After step S1030, the client device 10 performs image space segmentation (S1031).
Specifically, as shown in FIG. 12, the processor 12 calculates the coordinates of the reference line of the image space (for example, a line connecting the middle point of the line between the eyes and the center of the nose) of the IRL.
The processor 12 divides the image space of the simulated image SIMG into a first area A1 and a second area A2 with the reference line IRL as a boundary line.
The first area A1 is a rightward area (that is, an area including the right eye RE) with respect to the simulated image SIMG.
The second area A2 is an area to the left with respect to the simulated image SIMG (that is, an area including the left eye LE).

After step S1031, the client device 10 performs area coordinate specification (S1032).
Specifically, the processor 12 specifies the measurement coordinates included in the first area A1 among the measurement coordinates stored in the memory device 11 in step S1030 as the first area coordinate C1, and specifies the measurement coordinates included in the second area A2 as Coordinate C2 of the second area.

After step S1032, the client device 10 performs calculation of the line-of-sight distribution rate (S1033).
Specifically, the processor 12 uses Equation 1 to calculate the ratio of the user U ’s line of sight to the first area A1 (hereinafter referred to as the “first line of sight distribution rate”) E1 and the ratio of the user U ’s line of sight to the second area A2 (Hereinafter referred to as the "second sight distribution rate") E2.
[Number 1]
… (Number 1)
· E1 ... 1st line of sight distribution rate · E2 ... 2nd line of sight distribution rate · N (C1) ... Total number of coordinates C1 in the first area · N (C2) ... Total number of coordinates C2 in the second area

SS1 (steps S101 to S103) in FIG. 7 corresponds to the makeup simulation process of the first embodiment. That is, the makeup simulation is performed a plurality of times for each of the plurality of makeup patterns.

For example, in the second step S101, the processor 12 applies a makeup pattern different from the makeup pattern used in the first step S101 to the original image, thereby generating the same as that obtained in the first step S101. Makeup simulation images Different makeup simulation images.

In the second step S102, the processor 12 displays the screen P102 (FIG. 9) on the display 15.

The picture P102 includes an image object IMG102.
The image object IMG102 is a makeup simulation image generated in the second step S101.

That is, the processor 12 individually presents a plurality of makeup simulation images, and calculates a sight line distribution for each makeup simulation image. Thereby, the line-of-sight distribution for each makeup simulation image can be obtained.

After step S103, the client device 10 executes prompting of the sight parameters (S104).
Specifically, the processor 12 displays a screen P103 (FIG. 10) on the display 15.

The screen P103 includes display objects A103a to A103b, and image objects IMG101 and IMG102.
On the display object A103a, information related to the makeup pattern used to generate the image object IMG101 (such as a pattern name), and the first line-of-sight distribution rate E1 and the second line-of-sight distribution rate E2 for the image object IMG101 are displayed.
On the display object A103b, information related to the makeup pattern used to generate the image object IMG102 (for example, the name of the pattern) and the first line-of-sight distribution rate E1 and the second line-of-sight distribution rate E2 for the image object IMG102 are displayed.

After step S104, the client device 10 executes the determination of the recommended information (S105).

In the first example of step S105, the processor 12 calculates the line-of-sight distribution rate for the plurality of makeup simulation images calculated in step S103 (that is, the first line-of-sight distribution rate E1 of the display objects A103a to A103b displayed on the screen P103). And the second line-of-sight distribution rate E2), the makeup simulation image with the largest line-of-sight distribution rate E1 is selected.

In the second example of step S105, the processor 12 calculates the line-of-sight distribution rate for the plurality of makeup simulation images calculated in step S103 (that is, the first line-of-sight distribution rate E1 of the display objects A103a to A103b displayed on the screen P103). And the second line-of-sight distribution rate E2), at least one makeup simulation image is selected whose first line-of-sight distribution rate is a certain value or more.
In addition, when the first line-of-sight distribution rate E1 of all the makeup simulation images does not reach a specific value, no suggestion information is prompted.

In the document "Brady, N., Campbell, M., & Flaherty, M. (2005). Perceptual asymmetries are preserved in memory for highly familiar faces of self and friend. Brain & Cognition, 58, 334-342.", Records the research results of the familiarity reflected on the right half of the face. The face that one observes daily is the face that appears on the mirror. Therefore, when people observe the situation of their own face reflected on the mirror, they tend to feel more like themselves to the right of the observer.
Also, in the document "Guo, K., Tunnicliffe, D., & Roebuck, H. (2010). Human spontaneousgaze patterns in viewing of faces of different species. Perception, 39, 533-542." Describes humans or primates. This kind of research results have the tendency to observe the left half of the face to which the observer is looking when looking at people or animals.

The makeup simulation image with the largest first line-of-sight distribution rate E1 in the first example means the makeup simulation image with the most attention on the right eye side among the plurality of makeup simulation images. In other words, the makeup simulation image with the largest first line-of-sight distribution rate E1 can also be referred to as the image with the highest potential evaluation by user U (e.g., the image that cares most, the image that feels most like itself, the image that feels most affinity Images, or images that feel most impressive).

At least one makeup simulation image in which the first line-of-sight distribution rate E1 of the second example is equal to or more than a specific value means a makeup simulation image in which the right eye side of the plurality of makeup simulation images receives more or more attention. In other words, a makeup simulation image in which the first line-of-sight distribution rate E1 is a certain value or more can also be referred to as an image where the potential evaluation of the user U exceeds a certain reference (for example, an image with a higher degree of interest to a certain degree, a certain degree to a certain degree) (Feel like your own image, a certain degree of affinity, or a certain degree of impression).

In the example of the screen P103, since the first sight distribution ratio E1 of the image object IMG101 among the image objects IMG101 and IMG102 is the largest, the makeup simulation image corresponding to the image object IMG101 (that is, the application pattern name "MC1" (Makeup simulation images of makeup patterns) was decided as recommended information.

After step S105, the client device 10 executes the prompting of the recommended information (S106).
Specifically, the processor 12 displays a screen P104 (FIG. 10) on the display 15.

The screen P104 includes an image object IMG101, display objects A104a to A104b, and operation objects B104a to B104b.
A display name of a makeup pattern for generating a makeup simulation image is displayed on the display object A104a.
The display object A104b displays item information (for example, the category and item ID) of the item corresponding to the makeup pattern used to generate the makeup simulation image.
The operation object B104a is an object that receives a user instruction for updating the database on the server 30 with the recommended information (for example, the contents of the image object IMG101 and the display objects A104a to A104b).
The operation object B104b is an object that receives a user instruction to access a website that sells items displayed on the display object A104b. The URL (Uniform Resource Locator) of the website is assigned to the operation object B104b.

After step S106, the client device 10 executes an update request (S107).
Specifically, the processor 12 sends the update request data to the server 30.
The update request data contains the following information.
User ID of user U
Image data determined in step S100Makeup pattern ID for reference in step S101
· Information related to the execution day of step S103 · Measurement coordinates and measurement time stored in the memory device 11 in step S1030 · First sight line distribution rate E1 and second sight line distribution rate E2 calculated in step S1033

After step S107, the server 30 performs database update (S300).
Specifically, the processor 32 adds a new record to the simulation log information database associated with the user ID included in the update request data (FIG. 6).
The following information is stored in each field of the new record.
· In the "Simulation ID" field, a new simulation ID is stored.
· In the "Simulation execution day" field, information related to the execution day included in the update request data is stored.
· In the "original image" field, the image data contained in the update request data is stored.
· In the "Pattern ID" field, the makeup pattern ID contained in the update request data is stored.
· In the "Measurement Coordinates" column, the measurement coordinates contained in the update request data are stored.
· In the "Measurement time" field, the measurement time included in the update request data is stored.
· In the "R distribution" field, the first line-of-sight distribution rate E1 included in the update request data is stored.
· In the "L distribution" field, the second line-of-sight distribution rate E2 included in the update request data is stored.

According to the first embodiment, based on the deviation of the line of sight of the facial image (for example, makeup simulation image) presented by the user U, suggestion information that matches the preference of the user U (for example, the user in multiple makeup simulation images) U has a higher potential assessment of makeup simulation images). In this way, the user U can be prompted for information with a higher potential evaluation of the facial image observed by the user U.

(2) Second Embodiment A second embodiment of the present invention will be described. The second embodiment is an example in which advice information is presented based on a line of sight shift in the mirror image of the face of the user.

(2-1) Configuration of Client Device The configuration of the client device 10 according to the second embodiment will be described. FIG. 13 is an external view of a client device according to a second embodiment.

As shown in FIG. 13, the client device 10 further includes a half mirror 17. The half mirror 17 is disposed on the display 15. Since the half mirror 17 reflects external light and transmits light emitted from the display portion 15, the user U can observe the mirror MI and the simulated image SIMG at the same time.

(2-2) Information processing The information processing of the second embodiment will be described. FIG. 14 is a flowchart of information processing in the second embodiment. FIG. 15 is a detailed flowchart of the calculation of the line-of-sight parameters of FIG. 14. FIG. 16 is an explanatory diagram of the calculation of the line-of-sight parameters of FIG. 14. FIG. 17 is a diagram showing an example of a screen displayed in the information processing of FIG. 14. FIG.

As shown in FIG. 14, the client device 10 performs calculation of the sight line parameters (S110).

As shown in FIG. 15, the client device 10 executes photographing of the user image (S1100) after step S1030 (FIG. 11).
Specifically, the camera 16 captures an image of the face of the user U who is observing the mirror image MI reflected on the half mirror 17.
The processor 12 generates image data of an image obtained by the camera 16.

After step S1100, the client device 10 performs the calculation of the mirror coordinates (S1101).
Specifically, the processor 12 calculates the coordinates of each part of the face of the user U (for example, contour, eyes, nose, and mouth) in the image space by analyzing the feature amounts of the image data generated in step S1100 And dimensions.
The processor 12 calculates the distance between the half-mirror 17 and the user U based on the coordinates and size of each part of the face and the position of the camera 16 in the client device 10.
Based on the calculated distance and the coordinates in the image space, the processor 12 calculates the coordinates (hereinafter referred to as "mirror coordinates") of each part of the face in the coordinate space (hereinafter referred to as "mirror space") of the mirror MI.

After step S1101, the client device 10 performs partitioning of the mirror space (S1102).
Specifically, as shown in FIG. 16, the processor 12 calculates a reference line of the mirrored space based on the mirror coordinates of each part calculated in step S1101 (for example, a line connecting the middle point of the line connecting the eyes and the center of the nose). ) The coordinates of MRL.
The processor 12 divides the mirror space into a first area A1 and a second area A2 using the reference line IRL as a boundary line.
The first area A1 is an area in the mirror MI that is located to the right of the user U (that is, an area including the right eye RE).
The second area A2 is an area located on the left side of the user U (that is, an area including the left eye LE) in the mirror MI.

After step S1102, the client device 10 executes steps S1032 to S1033 (FIG. 11).

After step S110, the client device 10 executes the determination of the recommended information (S111).
Specifically, the processor 12 applies the first line-of-sight distribution rate E1 and the second line-of-sight distribution rate E2 calculated in step S110 to Equation 2, thereby calculating a score S related to the user U's evaluation of the mirror MI.
[Number 2]
(Number 2)
· S ... score · a ·· Coefficient of the first line of sight distribution rate · b ... Coefficient of the second line of sight distribution rate

After step S111, the suggestion information prompt is executed (S112).
Specifically, the processor 12 displays a screen P110 (FIG. 17A) on the display 15.
The screen P110 includes a display object A110.
The score S calculated in step S111 is displayed on the display object A110.

When the mirror MI changes (for example, the effect of makeup on the face of the user U changes), the processing results of steps S110 to S112 also change.
As a result, the processor 12 displays a screen P111 (FIG. 17B) on the display 15.
The screen P111 includes a display object A111.
The display object A111 displays the score S calculated in step S111. This score S is different from the score S displayed on the display object A110.

According to the second embodiment, based on the user's deviation of the line of sight of the mirror MI of the face of the user U, suggestion information (for example, the score S related to the evaluation of the mirror MI of the user U on the mirror MI) is presented. Thereby, the user can be prompted for a potential evaluation of the mirror MI of the face the user observes.

For example, while the user U observes the mirror image MI reflected on the half mirror 17 and uses a plurality of makeup items for makeup, the score S represents the potential evaluation of the user U on the makeup using each makeup item. That is, the score S indicates the response of the user U to each makeup item.

For example, when the user U observes the makeup of the mirror MI that is reflected on the half mirror 17, the score S indicates the potential of the user to the mirror MI (such as the face in the makeup) of the user U from the start to the end of the makeup. Evaluation. That is, the score S represents the transition of the user U's response to the makeup effect of the user U from the start of makeup to the end.

FIG. 13 shows an example in which the client device 10 includes a half mirror 17, but the second embodiment is not limited to this. The half-mirror 17 can be a mirror that completely reflects light, and the display 15 is disposed outside the user-side device 10. In this case, the user U observes a mirror image of his face reflected in the mirror. The screens P110 to P111 are displayed on a display arranged outside the client device 10.

(3) Third Embodiment A third embodiment will be described. The third embodiment is an example of prompting a user for potential evaluation of a virtual avatar image (hereinafter referred to as a "virtual avatar image") acting as an avatar of the user in a computer space.

Information processing in the third embodiment will be described. FIG. 18 is a flowchart of information processing in the third embodiment. 19 to 20 are diagrams showing examples of screens displayed in the information processing of FIG. 18.
The information processing in FIG. 18 is executed before or during a computer game when the third embodiment is applied to a computer game, for example.

As shown in FIG. 18, the client device 10 executes the prompt of the avatar image (S120).
Specifically, the processor 12 displays a screen P120 (FIG. 19) on the display 15.

The picture P120 includes an image object IMG120 and a message object M120.
The image object IMG120 is an avatar image.
The message object M120 is a message for guiding the sight of the user U to the image object IMG101.

After step S120, the client device 10 executes step S103 (FIG. 7).

SS3 (steps S120 and S103) of FIG. 18 is performed a plurality of times on each of the plurality of avatar images.

For example, in the second step S120, the processor 12 displays a screen 121 (FIG. 19) on the display 15.

The screen P121 includes an image object IMG121 and a message object M120.
The image object IMG121 is a avatar image different from the avatar image presented in the first step S120.

Thereby, the line of sight distribution of the plurality of virtual avatar images is obtained.

After step S103, the client device 10 executes the determination of the recommended information (S121).
Specifically, the processor 12 applies the first line-of-sight distribution rate E1 and the second line-of-sight distribution rate E2 of each avatar image to Equation 3, thereby calculating the self-identification index Si (an example of recommended information) ). The self-identification index Si is an index related to the degree to which the user U feels that the avatar image is "an avatar of himself."
[Number 3]
(Number 3)
· Si… Score · α… Coefficient of the first line of sight distributionβ • Coefficient of the second line of sight distribution

After step S121, the prompt of the recommended information is executed (S122).
Specifically, the processor 12 displays a screen P122 (FIG. 20) on the display 15.

The screen P122 includes display objects A122a to A122b, image objects IMG120 to IMG121, and operation objects B122a to B122b.
The avatar name corresponding to the avatar image corresponding to the image object IMG120, the first sight distribution ratio E1 and the second sight distribution ratio E2 to the image object IMG120, and the self-identification index of the image object IMG120 are displayed on the display object A122a. Si.
The avatar name corresponding to the avatar image corresponding to the image object IMG121, the first line-of-sight distribution rate E1 and the second line-of-sight distribution rate E2 to the image object IMG121, and the self-identification index of the image object IMG121 are displayed on the display object A122b. Si.

After step S122, the client device 10 executes an update request (S123).
Specifically, the processor 12 displays a screen P123 on the display 15.

The screen P123 includes a display object A123, operation objects B123a to B123b, and an image object IMG123.
The display object A123 displays information related to the avatar image with the highest self-identification index Si calculated in step S121 (the avatar name, the first sight distribution rate E1, the second sight distribution rate E2, and the self-identification index Si ).
The image object IMG123 is an avatar image with the highest self-identity index Si.
The operation object B123a is an object that receives a user's instruction to determine the avatar image used as the image object IMG123.
The operation object B123b is an object that receives a user instruction to reject the avatar image used as the image object IMG123.

When the user operates the operation object B123a, the processor 12 sends the update request data to the server 30.
The update request data contains the following information.
User ID
Image data of the avatar image assigned to the image object IMG123

After step S123, the server 30 performs database update (S300).
Specifically, the processor 32 refers to the user information database (FIG. 4), and specifies a code associated with the user ID included in the update request data.
The processor 32 stores the image data included in the update request data in a specific encoded "user image" field.
Thereby, the user U can use the avatar image with the highest self-identification index Si.

According to the third embodiment, the user can be prompted for a potential evaluation of the avatar image. In this way, the user can easily select the avatar with the highest potential evaluation of himself (e.g., the avatar that cares most, the avatar that feels most like himself (i.e., can be regarded as himself), the avatar that feels most affinity, or The avatar that feels most impressive).

(4) Fourth Embodiment A fourth embodiment will be described. The fourth embodiment is an example of a movement pattern based on a line of sight, which prompts a user for potential evaluation of an image.

(4-1) Outline of Fourth Embodiment The outline of the fourth embodiment will be described. Fig. 23 is an explanatory diagram of the outline of the fourth embodiment.

As shown in FIG. 23, in the fourth embodiment, similar to the first embodiment (FIG. 3), a simulated image SIMG including a face of a person (for example, user U) is presented.
Next, an eye-tracking signal related to the movement of the line of sight of the user U during the observation of the simulated image SIMG is generated.
Next, based on the eye-tracking signal, a line-of-sight pattern (an example of a "line-of-sight pattern") related to a pattern in which the user U's line of sight moves is calculated in the coordinate space (hereinafter referred to as "image space") of pixels constituting the simulated image SIMG .
Then, based on the line-of-sight pattern, suggestion information related to the potential evaluation of the simulated image SIMG by the user U is provided.

In this way, in the fourth embodiment, based on the sight pattern of the user U in the image space of the simulated image SIMG, the suggestion information reflecting the potential evaluation of the simulated image SIMG by the user U is presented.

(4-2) Information processing The information processing of the fourth embodiment will be described. FIG. 24 is a flowchart showing the calculation processing of the line-of-sight parameters in the fourth embodiment. 25 and 26 are explanatory diagrams of calculation of the line-of-sight parameters of FIG. 24.

The information processing of the fourth embodiment is performed in the same manner as the first embodiment (FIG. 7).

As shown in FIG. 24, after the client device 10 performs the calculation of the sight line parameters after step S1030 (S1034).
Specifically, the processor 12 calculates the sight line parameters of at least one of the following instead of the sight line distribution.
· Dwelling time when the line of sight stays in a certain size area (hereinafter referred to as "staying area") · Number of times the line of sight stays in the staying area · Moving range of the sight line · The moving order of the sight line · The moving area of the sight line

As a first example, the processor 12 calculates the dwell time Ts using Equation 4.
[Number 4]
(Number 4)
･ Ts ... dwell time ･ n ... the number of measurement coordinates included in the dwell area ･ t ... the interval of the measurement time is shown in FIG. 25A. The user U ’s sight EM of the image FIMG of the face stays in the dwell area 1 for example 5 After 10 seconds, stay in stop zone 2 for 10 seconds. In this case, the dwell time Ts1 of the dwell area 1 is 5 seconds, the dwell time Ts2 of the dwell area 2 is 10 seconds, and the total dwell time ΣTs is 15 seconds.
The processor 12 associates the measured time with the dwell time and stores the measured time in the memory device 11.

As a second example, the processor 12 calculates the number of stays Ns using Equation 5.
[Number 5]
(Number 5)
･ Ns ... Number of stays ･ ns ... Number of times included in the stay area within a specific time In the example of FIG. 25A, the number of stays is two.
The processor 12 stores the measurement time and the number of stays in association with each other and memorizes them in the storage device 11.

As a third example, the processor 12 calculates a rectangular area defined by the X coordinate and the Y coordinate (hereinafter referred to as "end point coordinates") located at the end points of the measurement coordinates as the moving range.
In the example of FIG. 25B, the movement range is a rectangular area Z defined by the endpoint coordinates {(X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2)}.
The processor 12 stores the coordinates of the endpoints of the predetermined rectangular area Z in the storage device 11.

As a fourth example, the processor 12 calculates the movement order according to the following method.
The processor 12 divides the pixel area of the image FIMG into pixel areas (e.g., left eye, right eye, nose, and mouth) pixel areas (e.g., left eye, right eye, nose, and mouth), And mouth pixel area).
The processor 12 calculates the order in which the line of sight EM passes through each pixel region (hereinafter referred to as "moving order").
In the example of FIG. 25C, the movement order is ｛the right-eye pixel area, the nose-pixel area, the mouth-pixel area, and the left-eye pixel area｝.
The processor 12 stores the movement sequence in the storage device 11.

As a fifth example, the processor 12 calculates a moving area based on any of the following.
The number of measurement coordinates in the XY space (however, repeated measurement coordinates are counted as 1 coordinate) (FIG. 26A).
When the pixel area of the image FIMG is divided into areas SEC divided by a specific area, the total number of areas SEC including measurement coordinates is included (Figure 26B)
The processor 12 stores the moving area in the storage device 11.

A sight pattern evaluation model is stored in the memory device 11. The line-of-sight pattern evaluation model is a model that inputs a line-of-sight pattern and outputs an evaluation of the face.

After step S103, the processor 12 of the client device 10 refers to the sight line pattern evaluation model stored in the memory device 11 in step S105, and calculates an evaluation corresponding to the sight line pattern calculated in step S103.

According to the fourth embodiment, when the line-of-sight pattern is used, the user U can also be prompted to evaluate the potential of the image FIMG.

In the first example, when the user U observes the image FIMG and performs makeup using an arbitrary makeup item, the makeup time is calculated while the makeup is performed.
As an example, when the measurement time is earlier (for example, the period from the start of the measurement to the elapse of a specific time of the entire measurement time (as an example, the time of 1/10 of the entire measurement time)) and the residence time is long, Means that the user U immediately feels against the effect of makeup immediately after starting makeup. In this case, the processor 12 determines that the evaluation of the makeup item by the user U is low, and prompts suggestion information based on the evaluation (for example, the message that the user U feels against the effect of the makeup effect).
As another example, when the measurement time is late (for example, the period from the end of the measurement to a specific time back to the entire measurement time (as an example, a time which is 1/10 of the entire measurement time) and the residence time is longer, It means that the user U does not feel a sense of discomfort in the makeup. In this case, the processor 12 determines that the evaluation of the makeup item by the user U is high, and prompts suggestion information based on the evaluation (for example, the message that the user U is satisfied with the effect of the makeup).

In the second example, when the user U observes the image FIMG while applying makeup using an arbitrary makeup item, the makeup is performed and the number of stays is calculated.
As an example, when the measurement time is early and the number of stays is large, it means that the user U immediately feels against the effect of makeup immediately after starting makeup. In this case, the processor 12 determines that the evaluation of the makeup item by the user U is low, and prompts suggestion information based on the evaluation (for example, the message that the user U feels contrary to the effect of the makeup).
As another example, when the measurement time is slow and the number of stays is large, it means that the user U feels a sense of disagreement on the makeup effect before the end of makeup, but does not feel a sense of disagreement on the makeup effect after the end of makeup. In this case, the processor 12 determines that the evaluation of the makeup item by the user U is high, and prompts suggestion information based on the evaluation (for example, the message that the user U is satisfied with the effect of the makeup).

In the third example, when the user U observes the image FIMG while applying makeup using an arbitrary makeup item, the makeup is performed and the moving range is calculated.
Makeup evaluation experts are known to look at the entire face. In addition, it is known that when people observe the faces of other people, they observe the entire body, whereas, when they observe their own faces, they tend to observe parts. That is, observing the entire face means objectively observing one's own face. The processor 12 indicates the consistency between the observer's own assessment and the assessment of others as recommended information by calculating the movement range.

In the fourth example, for example, after the makeup is finished, in order to evaluate the makeup effect, the moving order of the line of sight EM when the user U observes the image FIMG is calculated.
In general, the movement order indicates the order in which the user U pays attention to the parts after the makeup is performed. This order means the priority of the makeup characteristics for the user U. The processor 12 specifies a characteristic having a higher priority for the user U based on the movement sequence, thereby prompting makeup items suitable for the characteristic having a higher priority as recommended information.

In the fifth example, when the user U is applying makeup using an arbitrary makeup item while observing the image FIMG, the makeup is performed and the moving area is calculated.
The moving area is the same as the moving range of the third example, indicating the degree of consistency between the self-assessment and the others' assessment. The processor 12 calculates the moving area, and prompts the consistency between the self-assessment and the assessment of others as suggested information.

The fourth embodiment is also applicable to the second embodiment. When the fourth embodiment is applied to the case of the second embodiment, the user U can also be prompted to evaluate the potential MI of the mirror MI when the sight parameters other than the sight distribution are used.

(5) Modification Example A modification example of this embodiment will be described.

(5-1) Modification 1
A first modification will be described. Variation 1 is an example that guides the user's line of sight in the makeup simulation image (S102) of FIG. 7. FIG. 21 is a diagram showing a screen example of the first modification.
Modification 1 can also be applied to any of the first to third embodiments.

In step S102 of the first modification, the processor 12 displays a screen P101a (FIG. 21) on the display 15.

The screen P101a is different from the screen P101 (FIG. 8) in that the image object IMG101a is displayed. The image object IMG101a is an object for guiding the sight of the user U.
The image object IMG101a is displayed in the second area A2 of the image object IMG101.

It is known that the left field of view is generally associated with the entire process, and the right field of view is associated with the local process. According to the modification 1, since the image object IMG101a for guiding the sight is displayed in the second area A2, when the user U observes his face, the entire processing becomes an advantage, and a more objective self-assessment can be realized. One of the purposes of makeup is to make an impression on others. Reduce the difference between self-assessment and others' assessments by improving the objectivity of self-assessment. As a result, when self-evaluation is improved by the makeup of one's own face, the evaluation of the makeup from others also increases.

In addition, the difference between self-assessment and other people's assessment is smaller than the case where the sight is not guided. Therefore, numerical results can be used to show the effect of guiding the sight (improving objectivity and reducing the degree of deviation between others' assessment and self-assessment). For example, the moving range or moving area can be used to calculate the consistency between self-assessment and other people's assessments, and the calculation results can be used to show the effect when guiding the eyes.

(5-2) Modification 2
A modification 2 will be described. Variation 2 is an example that prompts the user for potential evaluation of an image. FIG. 22 is a diagram showing an example of a screen displayed in the information processing of Modification 2. FIG.
Modification 2 can also be applied to any of the first to fourth embodiments.

The information processing of the modification 2 is the same as that of the first embodiment (FIG. 7). However, step S104 (FIG. 7) is omitted.

After step S103, the processor 12 of the client device 10 applies the line-of-sight distribution rate calculated in step S103 (i.e., the first line-of-sight distribution rate E1 and the second line-of-sight distribution rate E2 in FIG. 12) in step S105. In order to calculate the score S related to the evaluation of the simulated image SIMG by the user U.

After step S105, the processor 12 displays the screen P130 (FIG. 22) on the display 15 instead of the screen P104 (FIG. 10) in step S106.

The screen P130 includes display objects A103a, A104b, and A130, and an image object IMG101.
The score calculated in step S105 is displayed on the display object A130.

According to the modification 2, the score S for one makeup simulation image is displayed on the display 15. In this way, the user U can be prompted for a potential assessment of one image.

(5-3) Modification 3
A modification 3 will be described. Modification 3 is an example in which the present embodiment is applied to a makeup simulation image and a face image other than the avatar.

As a first example, this embodiment can also be applied to a facial image after an arbitrary hairstyle is applied.
In this case, based on the sight shift of the face image presented to the user U (for example, the face image after applying any hairstyle), suggestion information (for example, multiple hairstyles) that matches the preference of the user U is prompted (Simulation image of hairstyle with higher potential evaluation of user U in the simulation image). In this way, the user U can be prompted for information with a higher potential assessment of the facial image observed by the user U.

As a second example, this embodiment can also be applied to a facial image after performing any operation.
In this case, based on the sight shift of the face image presented to the user U (for example, the face image after performing any operation), suggestion information (for example, a plurality of operations) that matches the preference of the user U is presented Surgical simulation image with higher potential assessment of user U in the simulation image). In this way, the user U can be prompted for information with a higher potential assessment of the facial image observed by the user U.

(6) Summary of this embodiment This embodiment is summarized.

The first aspect of this embodiment is an information processing device (for example, the client device 10), which includes: a mechanism for prompting the user with an image including a face (for example, the processor 12 that executes the processing of step S102);
A mechanism for obtaining the sight information related to the movement of the sight of the user (for example, the processor 12 executing the processing of step S1030);
A mechanism for calculating a user's line of sight offset in the image based on the line of sight information (for example, the processor 12 executing the processing of step S1033);
An organization that determines recommended information related to the image based on the line-of-sight shift (for example, the processor 12 that performs the processing of step S105); and an organization that prompts for the recommended information (for example, the processor 12 that performs the processing of step S106).

According to the first aspect, based on the user U's line-of-sight shift of the facial image, the decision information to be presented to the user U is determined. In this way, the user U can be prompted to evaluate the potential of the face that the user U observes.

The second aspect of this embodiment is an information processing device in which the calculation mechanism divides the image space of the image into a first region and a second region with a specific boundary line; and calculates a first line of sight in the first region Distribution; and the second line-of-sight distribution in the second region.

According to the second aspect, the advice information to be presented to the user U is determined based on the line-of-sight distribution in each of the two regions (the first region and the second region) constituting the image space.
In this way, suggestion information corresponding to the deviation of the line of sight distribution can be prompted.

The third aspect of this embodiment is an information processing device, in which the presenting mechanism individually presents a plurality of images different from each other;
The calculation mechanism calculates the first line of sight distribution and the second line of sight distribution for each image;
The deciding agency decides at least one of the plurality of images as recommendation information.

According to the third aspect, advice information is determined for each of the plurality of images based on the line of sight distribution in each of the two regions (the first region and the second region) constituting the image space.
In this way, at least one of the plurality of images can be prompted as recommended information.

The fourth aspect of this embodiment is an information processing device, wherein the first area is a rightward area relative to the image;
The second area is the area to the left relative to the image;
The recommended information is at least one image in which the ratio of the first line of sight distribution in the plurality of images is a certain value or more.

According to the fourth aspect, an image in which the ratio of the line-of-sight distribution with respect to the area to the right of the image is greater than a specific value is suggested as the recommended information. Thereby, it is possible to easily specify an image with a higher potential evaluation of the user U.

The fifth aspect of this embodiment is an information processing device, wherein the image includes a plurality of makeup simulation images after applying different makeup to the user's face;
The first region is a region to the right with respect to each makeup simulation image;
The second area is the area to the left relative to each makeup simulation image;
The recommended information is the makeup simulation image with the largest proportion of first sight distribution among the plurality of makeup simulation images.

According to the fifth aspect, among the plurality of makeup simulation images, it is suggested that the ratio of the sight line distribution to the area to the right of each makeup simulation image is greater than or equal to a specific makeup simulation image. Thereby, it is possible to easily specify a makeup simulation image with a higher potential evaluation by the user.

The sixth aspect of this embodiment is an information processing device, wherein the image is a makeup simulation image after applying makeup to the user's face;
The first area is the area to the right relative to the makeup simulation image;
The second area is the area to the left relative to the makeup simulation image;
The recommended information is information corresponding to the proportion of the first line of sight distribution in the makeup simulation image.

According to the sixth aspect, suggestion information (for example, a score) corresponding to the proportion of the line-of-sight distribution with respect to the area to the right of the makeup simulation image is presented. Thereby, the potential evaluation of the makeup simulation image by the user U can be known.

The seventh aspect of this embodiment is an information processing device, wherein the image includes a plurality of avatar images different from each other;
The first region is a region to the right relative to each avatar image;
The second area is the area to the left relative to each avatar image;
The recommended information is the avatar image with the largest proportion of first sight distribution among the plurality of avatar images.

According to the seventh aspect, among the plurality of virtual avatar images, it is suggested that the ratio of the line of sight distribution to the area to the right of each virtual avatar image is greater than or equal to the specific avatar image. Thereby, the avatar image with high potential evaluation of the user U can be easily specified.

An eighth aspect of this embodiment is an information processing device, wherein the image is a virtual avatar image;
The first area is the area to the right relative to the avatar image;
The second area is the area to the left relative to the avatar image;
The recommended information is information corresponding to the proportion of the first line of sight distribution in the avatar image.

According to the eighth aspect, suggestion information (for example, a score) corresponding to the ratio of the line of sight distribution to the area to the right of the avatar image is presented. In this way, the potential evaluation of the avatar image by the user U can be known.

The ninth aspect of this embodiment is an information processing device, wherein the image includes a plurality of images after applying different hairstyles to the user's face, and after performing different operations on the user's face At least one of the images.

According to the ninth aspect, the suggestion information is displayed based on displaying at least one of a face after applying an arbitrary hairstyle and a face after surgery. Thereby, the potential evaluation of the face by the user U can be known.

The tenth aspect of this embodiment is an information processing device (such as the client device 10), which includes:
A mechanism for obtaining the sight information related to the sight movement of the user's face mirroring of the user (for example, the processor 12 executing the processing of step S1030);
A mechanism for calculating a user's line-of-sight offset in the image based on the line-of-sight information (for example, the processor 12 executing the processing of step S1033);
On the basis of the line-of-sight shift, a mechanism for determining recommended information related to mirroring (for example, the processor 12 performing the processing of step S111); and a mechanism for prompting for the recommended information (for example, the processor 12 performing the processing of step S112).

According to the tenth aspect, the recommendation information is determined based on the user's line-of-sight shift of the face mirror. In this way, the user U can be prompted to evaluate the potential of the face of the user U who is reflected in the mirror.

The eleventh aspect of this embodiment is an information processing device, in which the calculation mechanism divides the mirrored mirror space into a first region and a second region with a specific boundary line; and calculates a first line of sight distribution in the first region; And the second line of sight distribution in the second area.

According to the eleventh aspect, based on the line-of-sight distribution in each of the two regions (the first region and the second region) constituting the mirror space, the recommended information to be presented to the user is determined.
In this way, suggestion information corresponding to the deviation of the line of sight distribution can be prompted.

The twelfth aspect of this embodiment is an information processing device, wherein the first area includes a right eye;
Area 2 contains the left eye;
The recommended information is information corresponding to the proportion of the first line of sight distribution, and is information related to the user's evaluation of the mirror image.

According to the twelfth aspect, suggestion information (for example, a score) corresponding to the ratio of the line of sight distribution with respect to the area to the right of the mirror image is presented. In this way, it is possible to know the potential evaluation of the user U on the mirror image (such as the mirror image of his own face after the makeup is performed).

The thirteenth aspect of this embodiment is an information processing device (such as the client device 10), which includes:
A mechanism for prompting the user's face image (for example, the processor 12 executing the processing of step S102);
A mechanism for obtaining the sight information related to the movement of the sight of the user (for example, the processor 12 executing the processing of step S1030);
A mechanism for calculating a line-of-sight pattern related to the line-of-sight movement pattern of the user in the image based on the line-of-sight information (for example, the processor 12 executing the process of step S1034);
Based on the calculated line-of-sight pattern, a mechanism for suggesting recommended information to be presented to the user is determined; and a mechanism for suggesting the recommended information is determined (for example, the processor 12 that performs the process of step S106).

According to the thirteenth aspect, based on the line-of-sight pattern of the face image of the user U, the advice information to be presented to the user is determined. In this way, the user U can be prompted to evaluate the potential of the face that the user U observes.

The fourteenth aspect of this embodiment is an information processing device, in which the calculating mechanism calculates the time that the user's line of sight stays in a specific area of the staying area, that is, the staying time, as the line of sight pattern.

The fifteenth aspect of this embodiment is an information processing device, in which the calculating mechanism calculates the number of stays of the user in the stay area in a specific range, that is, the stays number, as the sight pattern.

The sixteenth aspect of this embodiment is an information processing device, in which the calculation mechanism calculates the area of the user's line of sight distribution as the line of sight pattern.

The seventeenth aspect of this embodiment is an information processing device, in which the calculation mechanism calculates the order in which the user's line of sight moves as the line of sight pattern.

The eighteenth aspect of this embodiment is an information processing device, in which the calculating mechanism calculates the area where the user's line of sight moves as the line of sight pattern.

The nineteenth aspect of this embodiment is a program for causing a computer (for example, the processor 12) to function as each of the mechanisms described in any of the above.

(7) Other changes

The memory device 11 can be connected to the client device 10 via the network NW. The memory device 31 can be connected to the server 30 via a network NW.

Each step of the above information processing can be performed by any one of the client device 10 and the server 30.

In the above embodiment, the makeup simulation image is exemplified as an example of the simulation image SIMG. However, this embodiment can also be applied when the simulated image SIMG is at least one of the following.
Simulation images of the face after treatment (such as plastic surgery) Simulation images of the face including dyed hair

As mentioned above, although embodiment of this invention was described in detail, the range of this invention is not limited to the said embodiment. Moreover, the said embodiment can be variously improved or changed in the range which does not deviate from the meaning of this invention. The above-mentioned embodiments and modifications can be combined.

1‧‧‧ Information Processing System

10‧‧‧Client Device

11‧‧‧Memory device

12‧‧‧ processor

13‧‧‧I / O interface

14‧‧‧ communication interface

15‧‧‧ Display

16‧‧‧ Camera

17‧‧‧ half mirror

20‧‧‧ Eye Tracker

30‧‧‧Server

31‧‧‧Memory device

32‧‧‧ processor

33‧‧‧I / O interface

34‧‧‧ communication interface

A1‧‧‧Area 1

A2‧‧‧ Zone 2

A103a‧‧‧Display object

A103b‧‧‧ Display object

A104a‧‧‧Display object

A104b‧‧‧Display object

A110‧‧‧Display object

A111‧‧‧Display object

A122a‧‧‧Display object

A122b‧‧‧ Display object

A123a‧‧‧Display object

A123b‧‧‧ Display Object

A130‧‧‧Display object

B104a‧‧‧Operation object

B104b‧‧‧Operation object

B123a‧‧‧Operation object

B123b‧‧‧Operation object

E1‧‧‧The first line of sight distribution rate

E2‧‧‧Second line of sight distribution

EM‧‧‧Sight

FIMG‧‧‧Image

IMG‧‧‧Image Object

IMG100‧‧‧Image Object

IMG101‧‧‧Image Object

IMG101a‧‧‧Image Object

IMG102‧‧‧Image Object

IMG120‧‧‧Image Object

IMG121‧‧‧Image Object

IMG123‧‧‧Image Object

IRL‧‧‧Baseline

LE‧‧‧Left Eye

M101‧‧‧Message Object

M120‧‧‧Message Object

MI‧‧‧Mirror

MRL‧‧‧Baseline

NW‧‧‧Internet

P100 ～ P104‧‧‧Screen

P101a‧‧‧screen

P110‧‧‧screen

P111‧‧‧screen

P120 ～ P123‧‧‧Screen

P130‧‧‧screen

RE‧‧‧ Right Eye

SEC‧‧‧Area

SIMG‧‧‧Simulation image

S100 ～ S107‧‧‧step

S110 ～ S112‧‧‧step

S120 ～ S123‧‧‧step

S300‧‧‧step

S1030 ～ S1034‧‧‧step

S1100 ～ S1102‧‧‧step

SS1‧‧‧makeup simulation

SS3‧‧‧step

U‧‧‧User

(X1, Y2) ‧‧‧ endpoint coordinates

(X1, Y1) ‧‧‧ endpoint coordinates

(X2, Y1) ‧‧‧ endpoint coordinates

(X2, Y2) ‧‧‧ endpoint coordinates

X‧‧‧ coordinates

Y‧‧‧coordinates

Z‧‧‧ rectangular area

FIG. 1 is a block diagram showing the configuration of the information processing system of the first embodiment.

Fig. 2 is a block diagram showing the structure of the information processing system of the first embodiment.

FIG. 3 is an explanatory diagram of the outline of the first embodiment.

FIG. 4 is a diagram showing a data structure of a user information database in the first embodiment.

FIG. 5 is a diagram showing a data structure of a makeup pattern information master table according to the first embodiment.

FIG. 6 is a diagram showing a data structure of the analog log information database of the first embodiment.

FIG. 7 is a flowchart of information processing in the first embodiment.

FIG. 8 is a diagram showing an example of a screen displayed in the information processing of FIG. 7. FIG.

FIG. 9 is a diagram showing an example of a screen displayed in the information processing of FIG. 7. FIG.

FIG. 10 is a diagram showing an example of a screen displayed in the information processing of FIG. 7. FIG.

FIG. 11 is a detailed flowchart of the line-of-sight distribution calculation of FIG. 7.

FIG. 12 is an explanatory diagram of the calculation of the line-of-sight distribution of FIG. 7. FIG.

FIG. 13 is an external view of a client device according to a second embodiment.

FIG. 14 is a flowchart of information processing in the second embodiment.

FIG. 15 is a detailed flowchart of the line-of-sight distribution calculation of FIG. 14.

FIG. 16 is an explanatory diagram of the calculation of the line-of-sight distribution of FIG. 14. FIG.

17A and 17B are diagrams showing examples of screens displayed in the information processing of FIG. 14.

FIG. 18 is a flowchart of information processing in the third embodiment.

FIG. 19 is a diagram showing an example of a screen displayed in the information processing of FIG. 18. FIG.

FIG. 20 is a diagram showing an example of a screen displayed in the information processing of FIG. 18. FIG.

FIG. 21 is a diagram showing a screen example of the first modification.

FIG. 22 is a diagram showing an example of a screen displayed in the information processing of Modification 2. FIG.

Fig. 23 is an explanatory diagram of the outline of the fourth embodiment.

FIG. 24 is a flowchart of calculation processing of the line-of-sight parameters in the fourth embodiment.

25A to 25C are explanatory diagrams for calculating the line-of-sight parameters of FIG.

26A and 26B are diagrams for explaining the calculation of the line-of-sight parameters of FIG. 24.

Claims (19)

An information processing device including: A mechanism that prompts the user with images of faces; An organization that obtains sight information related to the sight movement of the above users; A mechanism for calculating the sight shift of the user in the image based on the sight information; An agency that determines recommended information related to the above-mentioned images based on the above-mentioned line of sight shift; and Agencies that suggest the above information. If the information processing device of claim 1, wherein The above calculation mechanism divides the image space of the above image into a first region and a second region with a specific boundary line; and calculates The first line of sight distribution in the above first area; and The second line of sight distribution in the second area. If the information processing device of claim 2, wherein The above-mentioned prompting mechanism individually presents a plurality of images different from each other; The above calculation mechanism calculates the first line-of-sight distribution and the second line-of-sight distribution for each image; The decision making mechanism determines at least one of the plurality of images as the above-mentioned recommended information. Such as the information processing device of item 3, wherein The first area is an area facing the right side of the image; The second area is an area facing the left side of the image; The above-mentioned advice information is at least one image of the plurality of images in which the ratio of the first line of sight distribution is a certain value or more. If the information processing device of claim 3 or 4, wherein The above image includes a plurality of makeup simulation images after applying different makeup to the face of the user; The first region is a region on the right side facing each makeup simulation image; The second area is an area to the left of each makeup simulation image; The above advice information is the makeup simulation image in which the proportion of the first line of sight distribution is the largest among the plurality of makeup simulation images. If the information processing device of claim 3 or 4, wherein The image includes a plurality of makeup simulation images after the user's face is subjected to makeup; The first area is an area facing the right side of each of the makeup simulation images; The second area is an area facing the left side of each of the makeup simulation images; The above-mentioned recommended information is information corresponding to a ratio of the first line-of-sight distribution in the makeup simulation image. If the information processing device of claim 3 or 4, wherein The above-mentioned prompting mechanism individually prompts a plurality of avatar images different from each other; The first area is the area on the right facing each avatar image; The second area is the area on the left facing each avatar image; The above recommended information is the avatar image with the largest proportion of the first line of sight among the plurality of avatar images. If the information processing device of claim 3 or 4, wherein The above image is a virtual avatar image; The first area is located to the right of the avatar image; The second area is located on the left side facing the virtual avatar image; The recommended information is information corresponding to a ratio of the first line of sight distribution in the avatar image. The information processing device of any one of claims 3 to 8, wherein The image includes at least one of a plurality of images after applying different hairstyles to the face of the user, and images after performing different operations on the face of the user. An information processing device including: A mechanism for obtaining gaze information related to the gaze movement of the user's face mirroring of the user; A mechanism for calculating the sight shift of the user in the mirror image based on the sight information; An agency that determines the recommended information related to the above-mentioned mirroring based on the above-mentioned line of sight shift; and Agencies that suggest the above information. The information processing device of claim 10, wherein The above calculation mechanism divides the above-mentioned mirror image space into a first region and a second region with a specific boundary line; and calculates The first line of sight distribution in the above first area; and The second line of sight distribution in the second area. The information processing device as claimed in item 11, wherein The first area is located to the right facing the mirror image; The second area is located on the left side facing the mirror image; The above-mentioned recommended information is information corresponding to the proportion of the first line-of-sight distribution, and is information related to the evaluation of the image by the user. An information processing device including: A mechanism to prompt the user's face image; An organization that obtains sight information related to the sight movement of the above users; A mechanism for calculating a line of sight pattern related to the pattern of the line of sight movement of the user in the image based on the line of sight information; Based on the calculated line-of-sight pattern, determine the organization that should advise the above user of the recommended information; and Agencies suggesting recommended information for the above decision. The information processing device of claim 13, wherein The calculation mechanism described above calculates the time that the user's line of sight stays in a specific range of staying area, that is, the staying time, as the line of sight pattern. If the information processing device of claim 13 or 14, wherein The above calculation mechanism calculates the number of times that the user's line of sight stays in a specific range of staying area, that is, the number of stays, as the line of sight pattern. The information processing device according to any one of claims 13 to 15, wherein The calculation means calculates the area of the line of sight distribution of the user as the line of sight pattern. The information processing device according to any one of claims 13 to 16, wherein The calculation means calculates the order in which the line of sight of the user moves as the line of sight pattern. The information processing device of any one of claims 13 to 17, wherein The calculating mechanism calculates the area where the user's line of sight moves as the line of sight pattern. A program for causing a computer to function as a mechanism according to any one of claims 1 to 18.