JPWO2020116069A1 - Information processing equipment and programs - Google Patents

Abstract

The information processing device includes means for acquiring moving image information of the user's face, means for extracting motion information related to the movement of the user's face based on the moving image information, and the user based on the motion information. It is provided with a means for identifying the habit of facial movement.

Description

The present invention relates to an information processing device and a program.

In general, facial expression is one of the factors that determine the impression of others. For example, a frowning expression gives a negative impression to others. On the other hand, facial expression habits may occur due to lifestyle habits. For example, a person who uses a personal computer for a long time may have a facial expression habit of unconsciously frowning. People with this facial habit unintentionally give a negative impression to others, even if they have positive emotions. In addition, facial expression habits over many years can contribute to the appearance of skin characteristics such as wrinkles or sagging.
However, people usually do not know their original facial habits because they often make facial expressions when looking at their faces in the mirror. In recent years, interest in cosmetology has increased, and there has been an increasing need for facial habits and the need to improve skin characteristics that may appear due to the effects of facial habits. For that purpose, it is first necessary to know one's facial expression habits.

As a technique for analyzing the movement of a part of a human face, for example, there is one described in Japanese Patent Application Laid-Open No. 2008-071179. In Japanese Unexamined Patent Publication No. 2008-071179, information indicating the state of each part of a human face obtained from a plurality of images taken continuously in time series is recognized as the movement of each part of an existing human face. It is described that it is determined based on the movement pattern of each registered part.

In Japanese Unexamined Patent Publication No. 2008-071179, it is determined whether or not the movement of the facial part of the target person in the face image corresponds to the natural human-like movement, and whether or not the person in the face image actually exists. Since it is a judgment, the facial expression habit of the target person is not judged.

An object of the present invention is to determine a user's facial expression habit.

One aspect of the present invention is
Equipped with a means to acquire moving image information of the user's face
A means for extracting motion information related to the movement of the user's face based on the moving image information is provided.
A means for identifying the movement habit of the user's face based on the movement information is provided.
It is an information processing device.

According to the present invention, the facial expression habit of the user can be determined.

It is a block diagram which shows the structure of the information processing system of this embodiment. It is explanatory drawing of the outline of this embodiment. It is a figure which shows the data structure of the motion instruction information database of this embodiment. It is a figure which shows the data structure of the facial expression habit information database of this embodiment. It is a sequence diagram of the facial expression habit determination process of this embodiment. It is a figure which shows the screen example displayed in the information processing of FIG. It is a figure which shows the data structure of the personal characteristic-related information database of the modification 1. It is a figure which shows the data structure of the skin condition-related information database of the modification 1. It is a sequence diagram of the facial expression habit determination process of the modification 1. It is a figure which shows the screen example displayed in the information processing of FIG. It is a figure which shows the data structure of the facial expression habit information database of the modification 2. It is a sequence diagram of the facial expression habit determination process of the modification 2. It is a figure which shows the screen example displayed in the information processing of FIG. It is a figure which shows the data structure of the facial expression habit information database of the modification 3. It is a sequence diagram of the facial expression habit determination process of the modification 3. It is a figure which shows the screen example displayed in the information processing of FIG. It is a figure which shows the screen example which is displayed in the information processing of the modification 4. It is a figure which shows the data structure of the facial expression habit information database of the modification 5. It is a figure which shows the screen example which is displayed in the information processing of the modification 5.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In addition, in the drawing for demonstrating the embodiment, the same components are in principle the same reference numerals, and the repeated description thereof will be omitted.

(1) Configuration of information processing system The configuration of the information processing system will be described. FIG. 1 is a block diagram showing a configuration of an information processing system of the present embodiment.

As shown in FIG. 1, the information processing system 1 includes a client device 10 and a server 30.
The client device 10 and the server 30 are connected via a network (for example, the Internet or an intranet) NW.

The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a web server.

(1-1) Configuration of Client Device The configuration of the client device 10 will be described.

As shown in FIG. 1, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, a communication interface 14, and a camera 15.

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
・ OS (Operating System) program ・ Application (for example, web browser) program that executes information processing

The data includes, for example, the following data.
-Database referenced in information processing-Data obtained by executing information processing (that is, the execution result of information processing)

The processor 12 is configured to realize the function of the client device 10 by activating the program stored in the storage device 11. The processor 12 is an example of a computer.

The input / output interface 13 is connected to the camera 15.
The input / output interface 13 is configured to acquire a user's instruction from an input device connected to the client device 10 and output information to an output device connected to the client device 10.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display.

The communication interface 14 is configured to control communication between the client device 10 and the server 30.

The camera 15 is configured to capture a moving image (for example, a moving image of a user's face).

(1-2) Server Configuration The configuration of the server 30 will be described.

As shown in FIG. 1, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface 34.

The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of a ROM, a RAM, and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
・ OS program ・ Application program that executes information processing

The data includes, for example, the following data.
・ Database referred to in information processing ・ Execution result of information processing

The processor 32 is configured to realize the function of the server 30 by activating the program stored in the storage device 31. The processor 32 is an example of a computer.

The input / output interface 33 is configured to acquire a user's instruction from an input device connected to the server 30 and output information to an output device connected to the server 30.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display.

The communication interface 34 is configured to control communication between the server 30 and the client device 10.

(2) Outline of the Embodiment The outline of the present embodiment will be described. FIG. 2 is an explanatory diagram of an outline of the present embodiment.

As shown in FIG. 2, the server 30 acquires the moving image information of the user's face via the client device 10.
The server 30 extracts motion information related to the movement of the user's face based on the motion image information of the user's face.
The server 30 identifies the habit of the user's face movement based on the movement information regarding the user's face movement.

According to this embodiment, the facial expression habit of the user is determined. As a result, the user can know his / her facial expression habit.

In the present embodiment, "face movement habit" and "facial expression habit" are synonymous and are used interchangeably.

(3) Database The database of the present embodiment will be described. The following database is stored in the storage device 31.

(3-1) Motion instruction information database The motion instruction information database of the present embodiment will be described. FIG. 3 is a diagram showing a data structure of the motion instruction information database of the present embodiment.

The movement instruction information database of FIG. 3 stores movement instruction information related to a message prompting the movement of the face presented to the user. The motion instruction information database includes a "motion instruction ID" field and a "message content" field. Each field is associated with each other.

A movement instruction ID is stored in the "movement instruction ID" field. The movement instruction ID is an example of the movement instruction identification information for identifying the movement instruction.

The "message content" field stores information (for example, text) about a message prompting the user to move the face. In this embodiment, it is assumed that three records are stored in order to simplify the explanation.

(3-2) Facial expression habit information database The facial expression habit information database of the present embodiment will be described. FIG. 4 is a diagram showing a data structure of the facial expression habit information database of the present embodiment.

The facial expression habit information database of FIG. 4 stores facial expression habit information related to the habit of movement of a person's face (hereinafter referred to as "facial expression habit"). The facial expression habit information database includes a "facial expression habit ID" field, a "feature amount" field, a "facial expression habit name" field, and an "advice" field. Each field is associated with each other.

The facial expression habit ID is stored in the "facial expression habit ID" field. The facial expression habit ID is an example of facial expression habit identification information for identifying facial expression habits.

The "feature amount" field stores information about the feature amount of each part of the face in the moving image of the face for the corresponding facial expression habit. The feature amount is, for example, the amount of change at the feature point of each part of the face (for example, the amount of change in the relative position at the feature point of each part of the face (for example, motion vector) and the color at the feature point of each part of the face). It is indicated by at least one of the changes in taste). The feature amount is modeled based on moving images of faces obtained from a plurality of people. The "feature amount" field includes a plurality of subfields (each "movement instruction ID" field). Even with the same facial expression habit, if the movement instruction ID is different, the feature amount is also different.

In the "facial expression habit name" field, information (for example, text) regarding the name of the facial expression habit (hereinafter referred to as "facial expression habit name") is stored.

The "advice" field stores advice information (eg, text or image data) for dealing with the corresponding facial habits and at least one of the skin features and facial features that may appear under the influence of the facial habits. The skin feature that can be manifested by the effects of facial habits is, for example, at least one of wrinkles, sagging, and dullness. Facial features that can appear under the influence of facial habits are, for example, changes in facial muscles. In addition, "to deal with" means, for example, improvement, prevention, restraint of progress, or slowing of progress.

(4) Information processing The information processing of the present embodiment will be described. FIG. 5 is a sequence diagram of the facial expression habit determination process of the present embodiment. FIG. 6 is a diagram showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 5, the client device 10 executes the reception of the determination condition (S100).
Specifically, the processor 12 displays the screen P100 (FIG. 6) on the display.

The screen P100 includes display objects A100a to A100d and operation objects B100a to B100c.
The display objects A100a to A100c include a message prompting the user to move his / her face.
The display object A100d includes a moving image taken by the camera 15 connected to the client device 10.
The operation objects B100a to B100c are objects that receive user instructions for executing the shooting of moving images. The operation objects B100a to B100c correspond to the display objects A100a to A100c, respectively. Corresponding movement instruction IDs are assigned to the operation objects B100a to B100c.

When the user operates the operation object B100a while pointing the camera 15 at his / her face, the processor 12 generates the moving image data of the moving image included in the display object A100d. Specifically, the user operates the operation object B100a while moving each part of the face according to the message content displayed on the display object A100a.
The processor 12 includes moving image data of a moving image captured by the camera 15 (that is, a moving image of the user's face in response to a message displayed on the display object A100a), and a movement instruction ID assigned to the operation object B100a. , Are associated with each other and stored in the storage device 11.

When the user operates the operation object B100b while pointing the camera 15 at his / her face, the processor 12 generates the moving image data of the moving image included in the display object A100d. Specifically, the user operates the operation object B100b while moving each part of the face according to the message content displayed on the display object A100b.
The processor 12 includes moving image data of a moving image captured by the camera 15 (that is, a moving image of the user's face in response to a message displayed on the display object A100b), and a movement instruction ID assigned to the operation object B100b. , Are associated with each other and stored in the storage device 11.

When the user operates the operation object B100c while pointing the camera 15 at his / her face, the processor 12 generates the moving image data of the moving image included in the display object A100d. Specifically, the user operates the operation object B100c while moving each part of the face according to the message content displayed on the display object A100c.
The processor 12 includes moving image data of a moving image captured by the camera 15 (that is, a moving image of the user's face in response to a message displayed on the display object A100c), and a movement instruction ID assigned to the operation object B100c. , Are associated with each other and stored in the storage device 11.

The processor 12 displays the screen P101 (FIG. 6) on the display.

The screen P101 includes a display object A101 and an operation object B101.
The display object A101 includes a message notifying the user that the shooting of the moving image is completed.
The operation object B101 is an object that receives a user instruction for executing the transmission of the generated moving image data.

After step S100, the client device 10 executes the determination request (S101).
Specifically, when the user operates the operation object B101, the processor 12 transmits the determination request data to the server 30.
The judgment request data includes the following information.
A combination of the moving image data stored in the storage device 11 in step S100 and the movement instruction ID.

After step S101, the server 30 executes the identification of facial expression habits (S300).
Specifically, the processor 32 calculates the feature amount of each part of the face from the moving image data included in the determination request data.
The processor 32 refers to the facial expression habit information database (FIG. 4), and obtains each feature amount corresponding to the movement instruction ID included in the judgment request data and a feature amount calculated from the moving image data included in the judgment request data. By comparing, it is determined whether or not the movement of the face in the moving image data corresponds to the facial habit. Do this for each facial habit. For example, the moving image data associated with the movement instruction ID "MES001" is compared with each feature amount in the "feature amount" field associated with the movement instruction ID "MES001".
The processor 32 refers to the facial expression habit information database (FIG. 4), and identifies the information regarding the facial expression habit name of the facial expression habit determined to be applicable and the corresponding advice information.

After step S300, the server 30 executes the determination response (S301).
Specifically, the processor 32 transmits the determination response data to the client device 10.
The judgment response data includes the following information.
-Information on facial expression habit names specified in step S300-Advice information specified in step S300

After step S301, the client device 10 executes the presentation of the determination result (S102).
Specifically, the processor 12 displays the screen P102 (FIG. 6) on the display based on the determination response data.

The screen P102 includes a display object A102.
The display object A102 includes a facial expression habit name and corresponding advice information. The advice information can be, for example, at least one of the following:
-Facial expression habits and information on cosmetics for dealing with at least one of the skin features and facial features that may appear due to the effects of facial expression habits (for example, product name, application site, application amount, product image, etc.)
-Information on facial habits and beauty equipment for dealing with at least one of the skin features and facial features that may appear due to the effects of facial habits (eg, product name, usage, product image, etc.)
-Information on facial habits and beauty treatments to deal with at least one of the skin and facial features that may be affected by facial habits (eg, type of treatment (massage, esthetics, yoga, etc.), method, etc.)
-Facial expression habits and information on exercises to deal with at least one of the skin features and facial features that may appear due to the effects of facial expression habits (for example, how to perform facial expression exercises by the user).
• Other advice information for dealing with facial habits and at least one of the skin and facial features that may be affected by facial habits.

According to the present embodiment, the facial expression habit is determined by giving the moving image of the face to the client device 10. As a result, the user can know his / her facial expression habit. In addition, advice information for dealing with at least one of the facial habits and the skin features and facial features that may appear due to the influence of the facial habits is presented. Thereby, the user can take measures to more appropriately deal with the facial expression habit and at least one of the skin features and facial features that may appear due to the influence of the facial expression habit.

Further, by storing the moving image data of the movement of the user's face according to the displayed message and the corresponding movement instruction ID in association with each other, the feature amount corresponding to the message content of the movement instruction is referred to in step S300. can do. Therefore, the facial expression habit can be identified with high accuracy.

(5) Modification Example A modification of the present embodiment will be described.

(5-1) Modification 1
Modification 1 will be described. The first modification is an example in which the user's facial expression habit is estimated based on the user information and information on the inferred facial expression habit is presented.

(5-1-1) Personal Characteristic Related Information Database The personal characteristic related information database of Modification 1 will be described. FIG. 7 is a diagram showing a data structure of the personal characteristic-related information database of the first modification.

The personal characteristic-related information database of FIG. 7 stores information on the relationship between personal characteristics and facial expression habits. The personal trait related information database includes a "personal trait ID" field, a "personal trait type" field, a "facial expression habit name" field, and an "advice" field.

In the "personal characteristic ID" field, a personal characteristic ID that identifies the individual characteristic is stored. The personal characteristic ID is an example of personal characteristic identification information that identifies an individual characteristic.

The Personal Characteristic Type field stores information about the type of personal characteristic. The "individual trait" includes at least one of internal traits (eg, personality traits, values traits, cosmetological consciousness) and behavioral traits (eg, lifestyle, habits).

The "facial expression habit name" field stores information (for example, text) regarding the name of the facial expression habit (facial expression habit name) that is presumed to be possessed by the person corresponding to the type of personal characteristic in the record. Facial habits inferred from the type of personal trait are based on statistical data obtained from multiple individuals.

The "advice" field stores advice information (eg, text or image data) for dealing with the corresponding facial habits and at least one of the skin features and facial features that may appear under the influence of the facial habits.

(5-1-2) Skin Condition-Related Information Database The skin condition-related information database of Modification 1 will be described. FIG. 8 is a diagram showing a data structure of the skin condition-related information database of the first modification.

The skin condition-related information database of FIG. 8 stores information on the relationship between the skin condition (hereinafter referred to as “skin condition”) and facial expression habits. The skin condition related information database includes a "skin condition ID" field, a "skin condition type" field, a "facial expression habit name" field, and an "advice" field.

In the "skin condition ID" field, a skin condition ID that identifies the skin condition is stored. The skin condition ID is an example of skin condition identification information for identifying the skin condition.

The "Skin Condition Type" field stores information about the type of skin condition. "Skin condition" refers to the condition of human skin. The skin condition relates to, for example, wrinkles, viscoelasticity of the skin, water content of the skin, water evaporation of the skin, and the like.

The "facial expression habit name" field stores information (for example, text) regarding the name of the facial expression habit (facial expression habit name) presumed to be possessed by the person corresponding to the type of skin condition in the record. Facial habits inferred from the type of skin condition are based on statistical data obtained from multiple individuals.

The "advice" field stores advice information (eg, text or image data) for dealing with the corresponding facial habits and at least one of the skin features and facial features that may appear under the influence of the facial habits.

(5-1-3) Information Processing The information processing of Modification 1 will be described. FIG. 9 is a sequence diagram of the facial expression habit determination process of the first modification. FIG. 10 is a diagram showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 9, the client device 10 executes the reception of the determination condition (S110).
Specifically, the processor 12 displays the screen P100 (FIG. 6) on the display.

The user operates the operation objects B100a, B100b and B100c in the same manner as in FIG. As a result, moving image data of the moving image of the user's face corresponding to each of the display objects A100a to A100c is generated. A corresponding motion instruction ID is associated with each motion image data. Each moving image data is stored in the storage device 11.

The processor 12 displays the screen P201 (FIG. 10) on the display.

The screen P201 includes field objects F201a to F201b and operation objects B201.
The field object F201a is an object that receives a user instruction for designating information regarding personal characteristics. The information about the personal characteristic is, for example, information about the type of personal characteristic that the user is self-recognizing, information about the judgment result by the application for determining the personal characteristic, or a predetermined question for the server 30 to judge the personal characteristic. Information about the answer to.
The field object F201b is an object that receives a user instruction for designating information regarding the skin condition. The information on the skin condition includes, for example, information on the type of skin condition that the user is self-recognizing, information on the determination result by the application or measuring device for determining the skin condition, and a predetermined question for the server 30 to determine the skin condition. Information about the answer to the matter, or image data (still image data or moving image data) including the skin of the face.
The operation object B201 is an object that receives the user instructions for transmitting the generated moving image data and the user instructions input to the field objects F201a to F201b to the server 30.

After step S110, the client device 10 executes the determination request (S111).
Specifically, when the user inputs a user instruction to the field object F201a and the field object F201b and operates the operation object B201, the processor 12 transmits the determination request data to the server 30.
The judgment request data includes the following information.
-Combination of moving image data and movement instruction ID stored in the storage device 11 in step S110-Information on personal characteristics specified in field object F201a-Information on skin condition specified in field object F201b

After step S111, the server 30 executes the identification of facial expression habits (S300) as in FIG.

After step S300, the server 30 executes the estimation of the facial expression habit (S311).
Specifically, the processor 32 identifies the type of personal trait of the user based on the information about the personal trait contained in the determination request data.
The processor 32 refers to the personal trait association information database (FIG. 7) to identify the personal trait ID associated with the identified personal trait type.
The processor 32 refers to the personal trait-related information database (FIG. 7) to identify information about the facial expression habit name associated with the identified personal trait ID. The processor 32 identifies the information related to the facial expression habit name not specified in step S300 among the information related to the specified facial expression habit name. The facial expression habit related to the facial expression habit name indicates a facial expression habit that the user does not have at the time of step S110, but may have in the future (hereinafter referred to as “guessed facial expression habit”).
The processor 32 refers to the personal characteristic-related information database (FIG. 7) regarding the guessed facial expression habit, and identifies the advice information corresponding to the information regarding the specified facial expression habit name.
The processor 32 identifies the type of skin condition of the user based on the information about the skin condition included in the determination request data.
The processor 32 identifies the skin condition ID associated with the identified skin condition type with reference to the skin condition related information database (FIG. 8).
The processor 32 refers to the skin condition-related information database (FIG. 8) and identifies the information regarding the facial expression habit name associated with the specified skin condition ID. The processor 32 identifies the information related to the facial expression habit name not specified in step S300 among the information related to the specified facial expression habit name. The facial expression habit related to the facial expression habit name indicates a facial expression habit (guessed facial expression habit) that the user does not have at the time of step S110 but may have in the future.
The processor 32 refers to the skin condition-related information database (FIG. 8) with respect to the guessed facial expression habit, and identifies the advice information corresponding to the information regarding the specified facial expression habit name.

After step S311 the server 30 executes the determination response (S312).
Specifically, the processor 32 transmits the determination response data to the client device 10.
The judgment response data includes the following information.
-Information on the facial expression habit name specified in step S300-Advice information specified in step S300-Information on the facial expression habit name of the guessed facial expression habit specified in step S311-Advice information specified in step S311

After step S312, the client device 10 executes the presentation of the determination result (S112).
Specifically, the processor 12 displays the screen P202 (FIG. 10) on the display based on the determination response data.

The screen P202 includes display objects A202a to A202b.
The display object A202a includes the facial expression habit name identified in step S300 and the corresponding advice information.
The display object A202b includes the facial expression habit name of the guessed facial expression habit identified in step S311 and the corresponding advice information. The advice information can be, for example, at least one of the following:
-Facial expression habits and information on cosmetics for dealing with at least one of the skin features and facial features that may appear due to the effects of facial expression habits (for example, product name, application site, application amount, product image, etc.)
-Information on facial habits and beauty equipment for dealing with at least one of the skin features and facial features that may appear due to the effects of facial habits (eg, product name, usage, product image, etc.)
-Information on facial habits and beauty treatments to deal with at least one of the skin and facial features that may be affected by facial habits (eg, type of treatment (massage, esthetics, yoga, etc.), method, etc.)
-Facial expression habits and information on exercises to deal with at least one of the skin features and facial features that may appear due to the effects of facial expression habits (for example, how to perform facial expression exercises by the user).
-Facial expression habits and other advice information for dealing with at least one of the skin features and facial features that may appear due to the effects of facial expression habits According to the first modification, the user may have facial expressions in the future. You can know the habit. Therefore, the user takes measures based on the advice information and the like before the facial expression habit actually appears on the face, so that at least one of the skin feature and the facial feature that can appear due to the influence of the facial expression habit and the facial expression habit appears. The possibility can be reduced.

(5-2) Modification 2
Modification 2 will be described. The second modification is an example in which the facial expression habit and the advice information for coping with at least one of the skin features and the facial features that may appear due to the influence of the facial expression habit are selected based on the user information.

(5-2-1) Facial expression habit information database The facial expression habit information database of Modification 2 will be described. FIG. 11 is a diagram showing a data structure of the facial expression habit information database of the second modification.

The facial expression habit information database of FIG. 11 stores facial expression habit information related to facial expression habits. The facial expression habit information database includes a "facial expression habit ID" field, a "feature amount" field, a "personal characteristic type" field, an "advice" field, and a "facial expression habit name" field. Each field is associated with each other.

The “facial expression habit ID” field, the “feature amount” field, and the “facial expression habit name” field are the same as those described in FIG.

The Personal Characteristic Type field stores information about the type of personal characteristic.

The "advice" field stores advice information (eg, text or image data) for dealing with the corresponding facial habits and at least one of the skin features and facial features that may appear under the influence of the facial habits. A plurality of advice information is stored for one facial expression habit ID. Advice information is associated with information about the type of personal trait.

(5-2-2) Information processing The information processing of the modified example 2 will be described. FIG. 12 is a sequence diagram of the facial expression habit determination process of the second modification. FIG. 13 is a diagram showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 12, the client device 10 executes the reception of the determination condition (S120).
Specifically, the processor 12 displays the screen P100 (FIG. 6) on the display.

The user operates the operation objects B100a, B100b and B100c in the same manner as in FIG. As a result, moving image data of the moving image of the user's face corresponding to each of the display objects A100a to A100c is generated. A corresponding motion instruction ID is associated with each motion image data. Each moving image data is stored in the storage device 11.

The processor 12 displays the screen P301 (FIG. 13) on the display.

The screen P301 includes a field object F301 and an operation object B301.
The field object F301 is an object that receives a user instruction for designating information regarding personal characteristics. The information about the personal characteristic is, for example, information about the type of personal characteristic that the user is self-recognizing, information about the judgment result by the application for determining the personal characteristic, or a predetermined question for the server 30 to judge the personal characteristic. Information about the answer to.
The operation object B301 is an object that receives the user instruction for transmitting the generated moving image data and the user instruction input to the field object F301 to the server 30.

After step S120, the client device 10 executes the determination request (S121).
Specifically, when the user inputs a user instruction to the field object F201 and operates the operation object B301, the processor 12 transmits the determination request data to the server 30.
The judgment request data includes the following information.
-Combination of the moving image data and the movement instruction ID stored in the storage device 11 in step S120-Information on the personal characteristics specified in the field object F301.

After step S121, the server 30 executes facial expression habit identification (S320).
Specifically, the processor 32 refers to the facial expression habit information database (FIG. 11) and determines whether or not each facial expression habit is applicable to the movement of the face in the moving image data, as in FIG.
The processor 32 refers to the facial expression habit information database (FIG. 11) and identifies information regarding the facial expression habit name of the facial expression habit determined to be applicable.

After step S320, the server 30 executes the identification of the advice information (S321).
Specifically, the processor 32 identifies the type of personal trait of the user based on the information about the personal trait contained in the determination request data. The processor 32 refers to the facial expression habit information database (FIG. 11) to identify the advice information associated with the type of personal trait identified in the record of the facial expression habit name identified in S320.

After step S321, the server 30 executes the determination response (S322).
Specifically, the processor 32 transmits the determination response data to the client device 10.
The judgment response data includes the following information.
-Information on facial expression habit names specified in step S320-Advice information specified in step S321

After step S322, the client device 10 executes the presentation of the determination result (S122).
Specifically, the processor 12 displays the screen P302 (FIG. 13) on the display based on the determination response data.
The screen P302 includes the display object A302.
The display object A302 includes a facial expression habit name and corresponding advice information.
According to the second modification, the user can obtain advice information based on the user information. For example, for users who have the same facial habit of "keeping one thing patiently", advice information "use basic cosmetics A every day for half a year" is presented, while "trying various things". For the user who has the characteristic, the advice information that "facial expression device B and massage method C are alternately applied every other week for 3 months" is presented. Therefore, the user can take measures according to the advice information suitable for himself / herself.

(5-3) Modification 3
Modification 3 will be described. Modification 3 is an example of presenting estimation information of the user's skin condition.

(5-3-1) Facial expression habit information database The facial expression habit information database of Modification 3 will be described. FIG. 14 is a diagram showing a data structure of the facial expression habit information database of the modified example 3.

The facial expression habit information database of FIG. 14 stores facial expression habit information related to facial expression habits. The facial expression habit information database includes a "facial expression habit ID" field, a "feature amount" field, a "facial expression habit name" field, an "advice" field, and a "reference skin condition" field. Each field is associated with each other.

The “facial expression habit ID” field, the “feature amount” field, the “facial expression habit name” field, and the “advice” field are the same as those described in FIG.

The "reference skin condition" field stores information (eg, text) about the skin condition that is presumed to be possessed by the person having the facial expression habit in the record. The skin condition inferred from facial habits is based on statistical data obtained from multiple persons.

(5-3-2) Information processing The information processing of the modified example 3 will be described. FIG. 15 is a sequence diagram of the facial expression habit determination process of the modified example 3. FIG. 16 is a diagram showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 15, the client device 10 executes the reception of the determination condition (S100) and the determination request (S101) in the same manner as in FIG.

After step S101, the server 30 refers to the facial expression habit information database (FIG. 14) and executes the identification of the facial expression habit (S330) in the same manner as in FIG.

After step S330, the server 30 performs skin condition estimation (S331).
Specifically, the processor 32 refers to the facial expression habit information database (FIG. 14) and specifies the information related to the reference skin condition associated with the information related to the facial expression habit name specified in step S330 as the estimated skin state. ..

After step S331, the server 30 executes the determination response (S332).
Specifically, the processor 32 transmits the determination response data to the client device 10.
The judgment response data includes the following information.
-Information on facial expression habit names specified in step S330-Advice information specified in step S330-Information on estimated skin condition specified in step S331

After step S332, the client device 10 executes the presentation of the determination result (S132).
Specifically, the processor 12 displays the screen P402 (FIG. 16) on the display based on the determination response data.

The screen P402 includes the display object A402.
The display object A402 includes a facial expression habit name, corresponding advice information, and a guessed skin condition.
According to the third modification, the user can know the presumed skin condition. Therefore, the user can know the skin condition that may be the cause of the facial expression habit or the future skin condition, and the skin characteristics that can be more appropriately expressed by the facial expression habit and the influence of the facial expression habit. And at least one of facial features can be addressed.

(5-4) Modification 4
A modification 4 will be described. Modification 4 is an example in which one moving image data corresponds to a plurality of message contents.

(5-4-1) Information processing The information processing of the modified example 4 will be described. FIG. 17 is a diagram showing an example of a screen displayed in the information processing of the modified example 4.

As shown in FIG. 5, the client device 10 executes the reception of the determination condition (S100).
Specifically, the processor 12 displays the screen P500 (FIG. 17) on the display.

The screen P500 includes display objects A500a to A500b and operation objects B500.
The display object A500a includes a message prompting the user to move his / her face.
The display object A500b includes a moving image taken by the camera 15 connected to the client device 10.
The operation object B500 is an object that receives a user instruction for executing shooting of a moving image.

When the user operates the operation object B500, the processor 12 displays different message contents on the display object A100 every n seconds from the operation of the operation object B500 in step S100 (FIG. 5).
When the user moves each part of the face according to the content of the message, the processor 12 stores the moving image data captured by the camera 15 and the elapsed time n from the operation of the operation object B100 in the storage device 11 in association with each other. ..

The processor 12 displays the screen P101 (FIG. 6) on the display.
After step S100, the client device 10 executes the determination request (S101).
Specifically, when the user operates the operation object B101, the processor 12 transmits the determination request data to the server 30.
The judgment request data includes the following information.
A combination of the moving image data stored in the storage device 11 in step S100 and the elapsed time n.

After step S101, the server 30 executes the identification of facial expression habits (S300).
Specifically, the storage device 31 stores the elapsed time n and the message content in association with each other.
The processor 32 identifies the message content corresponding to the moving image data based on the combination of the moving image data included in the determination request data and the elapsed time n.
Similar to FIG. 5, the processor 32 compares the feature quantities and specifies the information regarding the facial expression habit name of the facial expression habit determined to be applicable and the corresponding advice information.

After step S300, the server 30 executes the determination response (S301) as in FIG.

After step S301, the client device 10 executes the presentation of the determination result (S102) in the same manner as in FIG.

According to the fourth modification, one moving image data can correspond to a plurality of message contents. Therefore, the user can specify the determination condition with a simpler operation.

(5-5) Modification 5
A modified example 5 will be described. Modification 5 is an example in which the advice information differs depending on the content of the message.

(5-5-1) Facial Expression Habit Information Database The facial expression habit information database of Modification 5 will be described. FIG. 18 is a diagram showing a data structure of the facial expression habit information database of the modified example 5.

The facial expression habit information database of FIG. 18 stores facial expression habit information related to facial expression habits. The facial expression habit information database includes a "facial expression habit ID" field, a "feature amount" field, a "facial expression habit name" field, and an "advice" field. Each field is associated with each other.
The facial expression habit information database is associated with the movement instruction ID.

The “facial expression habit ID” field, the “feature amount” field, and the “facial expression habit name” field are the same as those described in FIG.

The "advice" field stores advice information (eg, text or image data) for dealing with the corresponding facial habits and at least one of the skin features and facial features that may appear under the influence of the facial habits. In the modified example 5, even if the facial expression habit is the same, if the movement instruction ID is different, the advice information is also different.

(5-5-2) Information Processing The information processing of the modified example 5 will be described. FIG. 19 is a diagram showing an example of a screen displayed in the information processing of the modified example 5.

The client device 10 executes the reception of the determination condition (S100) and the determination request (S101) in the same manner as in FIG.

After step S101, the server 30 executes the identification of facial expression habits (S300).
Specifically, in the modification 5, the processor 32 refers to the facial habit information database (FIG. 18) associated with the movement instruction ID included in the determination request data, and includes each feature amount and the determination request data. By comparing with the feature amount calculated from the moving image data, it is determined whether or not the movement of the face in the moving image data corresponds to the facial habit.
The processor 32 refers to the facial expression habit information database (FIG. 18), and identifies the information regarding the facial expression habit name of the facial expression habit determined to be applicable and the corresponding advice information.

After step S300, the server 30 executes the determination response (S301) as in FIG.

After step S301, the client device 10 executes the presentation of the determination result (S102).
Specifically, the processor 12 displays the screen P602 (FIG. 19) on the display based on the determination response data.

The screen P602 includes a display object A602.
The display object A602 includes a facial expression habit name and corresponding advice information.

According to the fifth modification, the advice information in consideration of the message content is presented. Therefore, the user can know more accurate advice information.

(6) Summary of the present embodiment A summary of the present embodiment will be given.

The first aspect of this embodiment is
A means for acquiring moving image information of a user's face (for example, a processor 32 that executes the process of step S300) is provided.
A means for extracting motion information related to the movement of the user's face based on the motion image information (for example, a processor 32 that executes the process of step S300) is provided.
A means for identifying a user's facial movement habit based on motion information (for example, a processor 32 that executes the process of step S300) is provided.
An information processing device (for example, a server 30).

According to the first aspect, the user's facial movement habit (facial expression habit) is determined. As a result, the user can generate a result for knowing his / her facial expression habit.

The second aspect of this embodiment is
A means for presenting information about the identified movement habits (eg, a processor 32 that executes the process of step S301).
It is an information processing device.

According to the second aspect, the result for knowing the facial expression habit is presented to the user. As a result, the user can know his / her facial expression habit.

The third aspect of this embodiment is
A means for acquiring user information about a user (for example, a processor 32 that executes the process of step S300) is provided.
A means for estimating a user's facial movement habit based on user information (for example, a processor 32 that executes the process of step S311) is provided.
A means (for example, a processor 32 that executes the process of step S312) for presenting information about the inferred movement habit is provided.
It is an information processing device.

According to the third aspect, the user can know the facial expression habits that may be possessed in the future. Therefore, the user takes measures based on the advice information and the like before the facial expression habit actually appears on the face, so that at least one of the skin feature and the facial feature that can appear due to the influence of the facial expression habit and the facial expression habit appears. The possibility can be reduced.

The fourth aspect of this embodiment is
User information includes information about at least one of the user's personal characteristics and physical condition.
It is an information processing device.

According to the fourth aspect, the user can know the personal characteristics and the facial expression habits that may have in the future that correlate with at least one of the physical conditions.

A fifth aspect of this embodiment is
User information includes information about the condition of the user's skin or face,
It is an information processing device.

According to the fifth aspect, the user can know the facial expression habits that may be possessed in the future that correlate with the skin or facial condition.

The sixth aspect of this embodiment is
Information about movement habits includes advice information for dealing with at least one of the skin and facial features that may be manifested by movement habits and the effects of movement habits.
It is an information processing device.

According to the sixth aspect, the facial expression habit and the advice information for dealing with at least one of the skin characteristics and the facial features that may appear due to the influence of the facial expression habit are presented. Thereby, the user can take measures to more appropriately deal with the facial expression habit and at least one of the skin features and facial features that may appear due to the influence of the facial expression habit.

The seventh aspect of this embodiment is
A means for acquiring user information about a user (for example, a processor 32 that executes the process of step S320) is provided.
Based on the user information, a means for selecting advice information for dealing with at least one of the movement habit and the skin feature and the face feature that may appear due to the influence of the movement habit (for example, the process of step S321 is executed). Equipped with a processor 32)
Selected advice information is presented,
It is an information processing device.

According to the seventh aspect, advice information based on user information can be obtained. Therefore, the user can take measures according to the advice information suitable for himself / herself.

The eighth aspect of this embodiment is
Information about movement habits includes guessing information about the user's skin or facial condition.
It is an information processing device.

According to the eighth aspect, the user can know the presumed skin condition. Therefore, the user can know the skin condition that may be the cause of the facial expression habit or the future skin condition, and the skin characteristics that can be more appropriately expressed by the facial expression habit and the influence of the facial expression habit. And at least one of facial features can be addressed.

The ninth aspect of this embodiment is
The means for acquiring the moving image information is to acquire the moving image information of the facial movement in response to the message prompting the user to move the face.
A means for associating and storing information about a message with moving image information.
It is an information processing device.

According to the ninth aspect, the facial expression habit is specified in consideration of the message content. Therefore, the user can know a more accurate facial expression habit. In addition, advice information that takes into account the content of the message is presented. Therefore, the user can know more accurate advice information.

The tenth aspect of this embodiment is
A means for acquiring user information about a user (for example, a processor 32 that executes the process of step S300) is provided.
A means for estimating facial movement habits based on user information (for example, a processor 32 that executes the process of step S311) is provided.
A means (for example, a processor 32 that executes the process of step S312) for presenting information about the inferred movement habit is provided.
An information processing device (for example, a server 30).

According to the tenth aspect, the user can know the facial expression habits that may be possessed in the future. Therefore, the user takes measures based on the advice information and the like before the facial expression habit actually appears on the face, so that at least one of the skin feature and the facial feature that can appear due to the influence of the facial expression habit and the facial expression habit appears. The possibility can be reduced.

The eleventh aspect of this embodiment is
It is a program for making a computer (for example, a processor 32) function as each of the above means.

(7) Other modifications

The storage device 11 may be connected to the client device 10 via the network NW. The storage device 31 may be connected to the server 30 via the network NW.

Each step of the above information processing can be executed by either the client device 10 or the server 30.
One device may perform all information processing without going through the network.

In the present embodiment, an example is shown in which the information regarding the message prompting the movement of the face presented to the user is text, but the present embodiment is not limited to this. The information about the message may be a moving image. The moving image is, for example, an animated moving image of a face having various facial expressions, and the user operates the operation object B100 on the screen P100 while imitating the facial expression, and the moving image data of the moving image of the user's face is generated. May be done.

In the present embodiment, an example of determining a facial expression habit based on a moving image acquired from the camera 15 has been shown, but the present embodiment is not limited to this. The facial expression habit may be determined based on the moving image stored in the storage device 11 or the storage device 31.

In the present embodiment, an example is shown in which the screen P100 includes a display object A100d including a moving image captured by the camera 15, but the present embodiment is not limited to this. The display object A100d may not be included in the screen P100. In this case, since the user cannot see his / her own face at all when moving each part of the face, the original facial expression is likely to be obtained, and the facial expression habit can be determined more accurately.

In the present embodiment, an example in which the facial expression habit name and the advice information are presented as information on the facial expression habit is shown, but the present embodiment is not limited to this. The information regarding the presented facial expression habit may be, for example, either the facial expression habit name or the advice information.

In the first modification, the facial expression habit name and the advice information are presented as the information regarding the inferred facial expression habit, but the present embodiment is not limited to this. The information regarding the presented facial expression habit may be, for example, either the facial expression habit name or the advice information.

In the present embodiment and the first modification, the facial expression habit name presented is an example in which the context relating to the state (for example, “raising the eyebrows”) is shown, but the present embodiment is not limited to this. The information about the facial expression habit name presented may be, for example, in the context of a combination of condition and degree, facial features, and at least one of the situations. Specifically, the context of the combination of state and degree (eg, "strong eyebrow rise"), the context of the combination of state and facial area (eg, "wrinkles at the corners of the right eye"), and the state and situation. It may be at least one of the contexts of the combination with (for example, "the corner of the mouth is barely raised when making a smile"). In addition, the information on the presented facial expression habit may include information on the content of the message.

In the first modification, an example is shown in which the user information is information on the personal characteristics of the user and information on the skin condition of the user, but the present embodiment is not limited to this. The user information may be, for example, at least one of the personal characteristics of the user, the physical condition of the user, the skin condition of the user, the facial condition of the user, and the attributes of the user. "Face condition" refers to the condition of a person's face. The facial condition relates to, for example, facial blood flow, facial 3D shape, and the like. Examples of the "user attribute" include age, gender, occupation, and the like.

In the first modification, the skin condition is specified in the field object F201b, but the present embodiment is not limited to this. The skin condition may be specified by image analysis from the moving image data generated by the operation of the operation objects B100a to B100c on the screen P100 (FIG. 6).

In the second modification, an example is shown in which the user information is information related to the personal characteristics of the user, but the present embodiment is not limited to this. The user information may be, for example, at least one of the personal characteristics of the user, the physical condition of the user, the skin condition of the user, the facial condition of the user, and the attributes of the user.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Further, the above-described embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Moreover, the above-described embodiment and modification can be combined.

1: Information processing system 10: Client device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Camera 30: Server 31: Storage device 32: Processor 33: Input / output interface 34: Communication interface

Claims (10)

Equipped with a means to acquire moving image information of the user's face
A means for extracting motion information related to the movement of the user's face based on the moving image information is provided.
A means for identifying the movement habit of the user's face based on the movement information is provided.
Information processing device. Provided with means for presenting information about the identified movement habits.
The information processing device according to claim 1. A means for acquiring user information about the user is provided.
A means for estimating the facial movement habit of the user based on the user information is provided.
A means for presenting information about the inferred movement habit,
The information processing device according to claim 1 or 2. The user information includes information about at least one of the user's personal characteristics and physical condition.
The information processing device according to claim 3. The user information includes information on the skin or face condition of the user.
The information processing device according to claim 3 or 4. The information regarding the movement habits includes advice information for dealing with the movement habits and at least one of the skin features and facial features that may appear due to the influence of the movement habits.
The information processing device according to any one of claims 2 to 5. A means for acquiring user information about the user is provided.
A means for selecting advice information for dealing with at least one of the movement habit and the skin feature and the face feature that may appear due to the influence of the movement habit based on the user information.
The selected advice information is presented,
The information processing device according to claim 6. The information regarding the movement habit includes the estimation information of the skin or face condition of the user.
The information processing device according to any one of claims 2 to 7. The means for acquiring the moving image information acquires the moving image information of the facial movement in response to the message prompting the user to move the face.
A means for associating and storing information about the message with the moving image information is provided.
The information processing device according to any one of claims 1 to 8. A program for operating a computer as each means according to any one of claims 1 to 9.

Images