JP7406502B2 - Information processing device, program and information processing method - Google Patents

Description

The present invention relates to an information processing device , a program , and an information processing method .

Generally, facial expressions are one of the factors that determine impressions of others. For example, a frowning facial expression gives a negative impression to others. On the other hand, facial expressions may develop due to lifestyle habits. For example, people who use computers for long periods of time may unconsciously frown. Even if people with this facial expression habit have positive emotions, they unintentionally give others a negative impression. Furthermore, long-standing facial expression habits can contribute to the appearance of skin characteristics such as wrinkles or sagging.
However, because people usually make facial expressions when looking at themselves in the mirror, they are unaware of their natural facial expression habits. In recent years, interest in beauty has increased, and there has been an increasing need to improve facial expressions and skin characteristics that may be affected by facial expressions. To do this, it is first necessary to know your own facial expressions.

As a technique for analyzing the movement of parts of a person's face, there is a technique described in, for example, Japanese Patent Laid-Open No. 2008-071179. In Japanese Unexamined Patent Publication No. 2008-071179, information indicating the state of each part of a person's face obtained from multiple images taken consecutively in chronological order is recognized as the movement of each part of a real person's face. It is described that it is determined based on registered movement patterns of each part.

In Japanese Patent Application Laid-Open No. 2008-071179, it is determined whether or not the movements of the facial parts of a target person in a facial image correspond to natural human movements, thereby determining whether or not the person in the facial image is a real person. Since this is a determination, the facial expressions of the target person are not determined.

An object of the present invention is to determine a user's facial expression habits.

One aspect of the present invention is
Equipped with means for acquiring video image information of the user's face,
comprising means for extracting movement information regarding the movement of the user's face based on the moving image information,
comprising means for identifying facial movement habits of the user based on the movement information;
It is an information processing device.

According to the present invention, a user's facial expression habits can be determined.

FIG. 1 is a block diagram showing the configuration of an information processing system according to the present embodiment. It is an explanatory diagram of an outline of this embodiment. FIG. 3 is a diagram showing a data structure of a movement instruction information database according to the present embodiment. FIG. 3 is a diagram showing the data structure of the facial expression habit information database according to the present embodiment. FIG. 2 is a sequence diagram of facial expression habit determination processing according to the present embodiment. 6 is a diagram showing an example of a screen displayed in the information processing of FIG. 5. FIG. 7 is a diagram illustrating a data structure of a personal characteristic related information database of Modification 1. FIG. 7 is a diagram illustrating a data structure of a skin condition related information database according to modification 1. FIG. FIG. 7 is a sequence diagram of facial expression habit determination processing according to modification example 1; 10 is a diagram showing an example of a screen displayed in the information processing of FIG. 9. FIG. FIG. 7 is a diagram showing a data structure of a facial expression habit information database in modification example 2. FIG. 7 is a sequence diagram of facial expression habit determination processing according to modification example 2; 13 is a diagram showing an example of a screen displayed in the information processing of FIG. 12. FIG. 12 is a diagram showing a data structure of a facial expression habit information database according to modification 3. FIG. FIG. 12 is a sequence diagram of facial expression habit determination processing according to modification example 3; 16 is a diagram showing an example of a screen displayed in the information processing of FIG. 15. FIG. 12 is a diagram illustrating an example of a screen displayed in information processing of Modification 4. FIG. 11 is a diagram showing a data structure of a facial expression habit information database according to modification 5. FIG. 12 is a diagram illustrating an example of a screen displayed in information processing according to modification 5. FIG.

Hereinafter, one embodiment of the present invention will be described in detail based on the drawings. In addition, in the drawings for explaining the embodiments, the same components are generally designated by the same reference numerals, and repeated explanations thereof will be omitted.

(1) Configuration of information processing system The configuration of the information processing system will be explained. FIG. 1 is a block diagram showing the configuration of an information processing system according to this embodiment.

As shown in FIG. 1, the information processing system 1 includes a client device 10 and a server 30.
The client device 10 and the server 30 are connected via a network (eg, the Internet or an intranet) NW.

The client device 10 is an example of an information processing device that sends a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. Server 30 is, for example, a web server.

(1-1) Configuration of Client Device The configuration of the client device 10 will be explained.

As shown in FIG. 1, the client device 10 includes a storage device 11, a processor 12, an input/output interface 13, a communication interface 14, and a camera 15.

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of ROM (Read Only Memory), RAM (Random Access Memory), and storage (for example, flash memory or hard disk).

The programs include, for example, the following programs.
・OS (Operating System) program ・Application program that executes information processing (e.g. web browser)

The data includes, for example, the following data.
・Databases referenced in information processing ・Data obtained by executing information processing (that is, execution results of information processing)

The processor 12 is configured to implement the functions of the client device 10 by activating a program stored in the storage device 11. Processor 12 is an example of a computer.

The input/output interface 13 is connected to the camera 15.
The input/output interface 13 is configured to obtain user instructions from an input device connected to the client device 10 and output information to an output device connected to the client device 10 .
The input device is, for example, a keyboard, pointing device, touch panel, or a combination thereof.
The output device is, for example, a display.

Communication interface 14 is configured to control communication between client device 10 and server 30.

The camera 15 is configured to capture a moving image (for example, a moving image of a user's face).

(1-2) Server Configuration The configuration of the server 30 will be explained.

As shown in FIG. 1, the server 30 includes a storage device 31, a processor 32, an input/output interface 33, and a communication interface 34.

The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of ROM, RAM, and storage (eg, flash memory or hard disk).

The programs include, for example, the following programs.
・OS program ・Application program that executes information processing

The data includes, for example, the following data.
・Databases referenced in information processing ・Execution results of information processing

The processor 32 is configured to implement the functions of the server 30 by activating a program stored in the storage device 31. Processor 32 is an example of a computer.

The input/output interface 33 is configured to obtain user instructions from an input device connected to the server 30 and output information to an output device connected to the server 30.
The input device is, for example, a keyboard, pointing device, touch panel, or a combination thereof.
The output device is, for example, a display.

Communication interface 34 is configured to control communication between server 30 and client device 10 .

(2) Overview of the embodiment An overview of the present embodiment will be described. FIG. 2 is an explanatory diagram outlining the present embodiment.

As shown in FIG. 2, the server 30 acquires moving image information of the user's face via the client device 10.
The server 30 extracts motion information regarding the movement of the user's face based on the moving image information of the user's face.
The server 30 identifies the user's facial movement habits based on movement information regarding the user's facial movements.

According to this embodiment, the user's facial expression habits are determined. This allows the user to know his own facial expressions.

In this embodiment, "facial movement habit" and "facial expression habit" have the same meaning and are used interchangeably.

(3) Database The database of this embodiment will be explained. The following database is stored in the storage device 31.

(3-1) Movement instruction information database The movement instruction information database of this embodiment will be explained. FIG. 3 is a diagram showing the data structure of the movement instruction information database of this embodiment.

The movement instruction information database shown in FIG. 3 stores movement instruction information related to a message to be presented to the user to encourage facial movement. The movement instruction information database includes a "movement instruction ID" field and a "message content" field. Each field is associated with each other.

A motion instruction ID is stored in the "movement instruction ID" field. The movement instruction ID is an example of movement instruction identification information that identifies a movement instruction.

The "message content" field stores information (for example, text) regarding a message to be presented to the user that encourages facial movement. Note that in this embodiment, in order to simplify the explanation, it is assumed that three records are stored.

(3-2) Facial expression information database The facial expression information database of this embodiment will be explained. FIG. 4 is a diagram showing the data structure of the facial expression habit information database of this embodiment.

The facial expression information database shown in FIG. 4 stores facial expression information related to the movement habits of a person's face (hereinafter referred to as "expression habits"). The facial expression information database includes a "facial quirk ID" field, a "feature amount" field, a "facial quirk name" field, and an "advice" field. Each field is associated with each other.

The “facial expression habit ID” field stores the facial expression habit ID. The facial expression habit ID is an example of facial expression habit identification information that identifies a facial expression habit.

The “feature amount” field stores information regarding the feature amount of each part of the face in the moving image of the face for the corresponding facial expression. The feature amount is, for example, the amount of change in the feature point of each part of the face (for example, the amount of change in relative position (e.g., motion vector) of the feature point of each part of the face, and the color at the feature point of each part of the face). (at least one of the amount of change in taste). The feature amounts are modeled based on moving images of faces obtained from multiple people. The “feature quantity” field includes a plurality of subfields (each “motion instruction ID” field). Even if the facial expression is the same, the feature amount will be different if the movement instruction ID is different.

The "Facial expression habit name" field stores information (for example, text) regarding the name of the facial expression habit (hereinafter referred to as "Facial expression habit name").

The "advice" field stores advice information (for example, text or image data) for dealing with at least one of the corresponding facial expression and the skin characteristics and facial features that may appear under the influence of the facial expression. Skin characteristics that may appear due to facial expressions include, for example, at least one of wrinkles, sagging, and dullness. Facial features that may appear due to the influence of facial expressions include, for example, changes in facial muscles. Note that "to deal with" refers to, for example, improving, preventing, inhibiting progression, or slowing progression.

(4) Information processing Information processing of this embodiment will be explained. FIG. 5 is a sequence diagram of the facial expression habit determination process of this embodiment. FIG. 6 is a diagram showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 5, the client device 10 executes acceptance of determination conditions (S100).
Specifically, processor 12 displays screen P100 (FIG. 6) on the display.

Screen P100 includes display objects A100a to A100d and operation objects B100a to B100c.
Display objects A100a to A100c include messages encouraging the user to move his or her face.
The display object A100d includes a moving image captured by the camera 15 connected to the client device 10.
The operation objects B100a to B100c are objects that accept user instructions for shooting moving images. The operation objects B100a to B100c correspond to the display objects A100a to A100c, respectively. Corresponding motion instruction IDs are assigned to the operation objects B100a to B100c.

When the user operates the operation object B100a while pointing the camera 15 toward his or her face, the processor 12 generates moving image data of the moving image included in the display object A100d. Specifically, the user operates the operation object B100a while moving each part of the face in accordance with the content of the message displayed on the display object A100a.
The processor 12 uses the moving image data of the moving image captured by the camera 15 (that is, the moving image of the user's face in response to the message displayed on the display object A 100a) and the movement instruction ID assigned to the operation object B 100a. , are stored in the storage device 11 in association with each other.

When the user operates the operation object B100b while pointing the camera 15 toward his or her face, the processor 12 generates moving image data of the moving image included in the display object A100d. Specifically, the user operates the operation object B 100b while moving each part of the face in accordance with the message content displayed on the display object A 100b.
The processor 12 uses the moving image data of the moving image captured by the camera 15 (that is, the moving image of the user's face in response to the message displayed on the display object A 100b) and the movement instruction ID assigned to the operation object B 100b. , are stored in the storage device 11 in association with each other.

When the user operates the operation object B100c while pointing the camera 15 toward his or her face, the processor 12 generates moving image data of the moving image included in the display object A100d. Specifically, the user operates the operation object B100c while moving each part of the face in accordance with the content of the message displayed on the display object A100c.
The processor 12 uses the moving image data of the moving image captured by the camera 15 (that is, the moving image of the user's face in response to the message displayed on the display object A 100c) and the movement instruction ID assigned to the operation object B 100c. , are stored in the storage device 11 in association with each other.

The processor 12 displays screen P101 (FIG. 6) on the display.

Screen P101 includes a display object A101 and an operation object B101.
The display object A101 includes a message informing the user that the shooting of the moving image has been completed.
The operation object B101 is an object that accepts a user instruction for transmitting generated moving image data.

After step S100, the client device 10 executes a determination request (S101).
Specifically, when the user operates the operation object B101, the processor 12 transmits determination request data to the server 30.
The judgment request data includes the following information.
- Combination of the moving image data and motion instruction ID stored in the storage device 11 in step S100

After step S101, the server 30 identifies facial expression habits (S300).
Specifically, the processor 32 calculates the feature amount of each part of the face from the moving image data included in the determination request data.
The processor 32 refers to the facial expression habit information database (FIG. 4) and calculates each feature amount corresponding to the movement instruction ID included in the determination request data and the feature amount calculated from the moving image data included in the determination request data. A comparison is made to determine whether or not the facial movement in the video data corresponds to the facial expression. Do this for each facial expression habit. For example, the moving image data associated with the motion instruction ID "MES001" is compared with each feature amount in the "feature amount" field associated with the motion instruction ID "MES001."
The processor 32 refers to the facial expression habit information database (FIG. 4) and specifies information regarding the name of the facial expression habit determined to be applicable and corresponding advice information.

After step S300, the server 30 executes a determination response (S301).
Specifically, the processor 32 transmits the determination response data to the client device 10.
The judgment response data includes the following information.
- Information regarding the facial expression name identified in step S300 - Advice information identified in step S300

After step S301, the client device 10 presents the determination result (S102).
Specifically, the processor 12 displays screen P102 (FIG. 6) on the display based on the determination response data.

Screen P102 includes display object A102.
The display object A102 includes a facial expression name and corresponding advice information. The advice information may be, for example, at least one of the following.
・Information regarding cosmetics to address at least one of facial expressions and skin characteristics and facial features that may appear due to facial expressions (e.g., product name, application site, amount of application, product image, etc.)
・Information regarding beauty appliances for addressing at least one of facial expressions and skin characteristics and facial features that may appear as a result of facial expressions (e.g., product name, usage instructions, product images, etc.)
・Information regarding beauty treatments to address at least one of facial expressions and skin characteristics and facial features that may be affected by facial expressions (e.g. type of treatment (massage, beauty salon, yoga, etc.), method, etc.)
・Information about facial expressions and exercises to deal with at least one of the skin characteristics and facial features that may appear as a result of facial expressions (for example, how to perform facial expression exercises by the user)
・Other advice information for dealing with facial expressions and at least one of the skin characteristics and facial features that may appear as a result of facial expressions

According to this embodiment, facial expression habits are determined by providing a moving image of a face to the client device 10. This allows the user to know his own facial expressions. Further, advice information for dealing with at least one of the facial expressions and the skin characteristics and facial features that may appear under the influence of the facial expressions is presented. Thereby, the user can take measures to more appropriately deal with facial expressions and at least one of the skin characteristics and facial features that may appear under the influence of facial expressions.

Further, by storing the moving image data of the user's facial movement in response to the displayed message in association with the corresponding movement instruction ID, the feature amount corresponding to the message content of the movement instruction is referenced in step S300. can do. Therefore, facial expression habits can be identified with high accuracy.

(5) Modification Example A modification example of this embodiment will be explained.

(5-1) Modification example 1
Modification 1 will be explained. Modification 1 is an example in which a user's facial expressions are estimated based on user information and information regarding the estimated facial expressions is presented.

(5-1-1) Personal characteristics related information database The personal characteristics related information database of Modification 1 will be explained. FIG. 7 is a diagram showing the data structure of the personal characteristics related information database of Modification 1.

The personal characteristic related information database in FIG. 7 stores information regarding the relationship between personal characteristics and facial expression habits. The personal characteristic related information database includes a "personal characteristic ID" field, a "personal characteristic type" field, a "facial expression name" field, and an "advice" field.

The "personal characteristic ID" field stores a personal characteristic ID that identifies personal characteristics. The personal characteristic ID is an example of personal characteristic identification information that identifies personal characteristics.

The "personal characteristic type" field stores information regarding the type of personal characteristic. "Personal characteristics" include at least one of internal characteristics (e.g., personality characteristics, value characteristics, beauty consciousness) and behavioral characteristics (e.g., lifestyle, habits).

The "facial quirk name" field stores information (for example, text) regarding the name of a facial quirk (facial quirk name) that is presumed to be possessed by a person corresponding to the type of personal characteristic in the record. Facial expression habits inferred from types of personal characteristics are based on statistical data obtained from multiple people.

The "advice" field stores advice information (for example, text or image data) for dealing with at least one of the corresponding facial expression and the skin characteristics and facial features that may appear under the influence of the facial expression.

(5-1-2) Skin Condition Related Information Database The skin condition related information database of Modified Example 1 will be explained. FIG. 8 is a diagram showing the data structure of the skin condition related information database of Modification 1.

The skin condition related information database in FIG. 8 stores information regarding the relationship between skin conditions (hereinafter referred to as "skin conditions") and facial expressions. The skin condition related information database includes a "skin condition ID" field, a "skin condition type" field, a "facial expression name" field, and an "advice" field.

The "skin condition ID" field stores a skin condition ID that identifies a skin condition. The skin condition ID is an example of skin condition identification information that identifies a skin condition.

The "skin condition type" field stores information regarding the type of skin condition. "Skin condition" refers to the condition of a person's skin. The skin condition relates to, for example, wrinkles, skin viscoelasticity, skin moisture content, skin moisture evaporation rate, and the like.

The "facial habit name" field stores information (for example, text) regarding the name of the facial habit (facial habit name) that is presumed to be possessed by a person corresponding to the type of skin condition in the record. Facial expressions inferred from skin condition types are based on statistical data obtained from multiple people.

The "advice" field stores advice information (for example, text or image data) for dealing with at least one of the corresponding facial expression and the skin characteristics and facial features that may appear under the influence of the facial expression.

(5-1-3) Information Processing Information processing of Modified Example 1 will be explained. FIG. 9 is a sequence diagram of facial expression habit determination processing according to modification 1. FIG. 10 is a diagram showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 9, the client device 10 executes acceptance of determination conditions (S110).
Specifically, processor 12 displays screen P100 (FIG. 6) on the display.

The user operates the operation objects B100a, B100b, and B100c similarly to FIG. 6. As a result, moving image data of a moving image of the user's face corresponding to each of the display objects A100a to A100c is generated. Each moving image data is associated with a corresponding motion instruction ID. Each moving image data is stored in the storage device 11.

The processor 12 displays screen P201 (FIG. 10) on the display.

Screen P201 includes field objects F201a to F201b and operation object B201.
Field object F201a is an object that accepts user instructions for specifying information regarding personal characteristics. The information regarding personal characteristics is, for example, information regarding the type of personal characteristics that the user self-recognizes, information regarding the determination result by an application that determines personal characteristics, or predetermined questions for the server 30 to determine personal characteristics. This is information regarding the answer to.
Field object F201b is an object that accepts user instructions for specifying information regarding skin conditions. The information regarding the skin condition includes, for example, information regarding the type of skin condition that the user self-recognizes, information regarding the determination result by an application or measuring device that determines the skin condition, and a predetermined question for the server 30 to determine the skin condition. This is information related to answers to questions, or image data (still image data or moving image data) that includes facial skin.
The operation object B201 is an object that accepts user instructions for transmitting generated moving image data and user instructions input into the field objects F201a to F201b to the server 30.

After step S110, the client device 10 executes a determination request (S111).
Specifically, when the user inputs user instructions to the field object F201a and field object F201b and operates the operation object B201, the processor 12 transmits determination request data to the server 30.
The judgment request data includes the following information.
・A combination of the moving image data and movement instruction ID stored in the storage device 11 in step S110 ・Information regarding personal characteristics specified in the field object F201a ・Information regarding the skin condition specified in the field object F201b

After step S111, the server 30 executes the identification of facial expression habits (S300) as in FIG. 5.

After step S300, the server 30 performs facial expression estimation (S311).
Specifically, the processor 32 identifies the type of the user's personal characteristics based on the information regarding the personal characteristics included in the determination request data.
The processor 32 refers to the personal characteristic related information database (FIG. 7) to identify the personal characteristic ID associated with the identified personal characteristic type.
The processor 32 refers to the personal characteristic related information database (FIG. 7) and specifies information regarding the facial expression name associated with the specified personal characteristic ID. The processor 32 identifies information regarding the facial expression quirk names that have not been specified in step S300, from among the information regarding the identified facial expression quirk names. The facial expression habit related to this facial expression habit name indicates a facial expression habit that the user does not have at the time of step S110 but may have in the future (hereinafter referred to as "estimated facial expression habit").
Regarding the estimated facial expression habit, the processor 32 refers to the personal characteristics related information database (FIG. 7) and specifies advice information corresponding to the information regarding the specified facial expression habit name.
The processor 32 identifies the type of skin condition of the user based on the information regarding the skin condition included in the determination request data.
The processor 32 refers to the skin condition related information database (FIG. 8) and identifies the skin condition ID associated with the identified skin condition type.
The processor 32 refers to the skin condition related information database (FIG. 8) and specifies information regarding the facial expression name associated with the specified skin condition ID. The processor 32 identifies information regarding the facial expression quirk names that have not been specified in step S300, from among the information regarding the identified facial expression quirk names. The facial expression habit associated with this facial expression habit name indicates a facial expression habit (estimated facial habit) that the user does not have at the time of step S110 but may have in the future.
Regarding the estimated facial expression habit, the processor 32 refers to the skin condition related information database (FIG. 8) and specifies advice information corresponding to information regarding the specified facial expression habit name.

After step S311, the server 30 executes a determination response (S312).
Specifically, the processor 32 transmits the determination response data to the client device 10.
The judgment response data includes the following information.
・Information regarding the facial expression habit name specified in step S300 ・Advice information specified in step S300 ・Information regarding the facial expression habit name of the guessed facial habit identified in step S311 ・Advice information specified in step S311

After step S312, the client device 10 presents the determination result (S112).
Specifically, the processor 12 displays screen P202 (FIG. 10) on the display based on the determination response data.

Screen P202 includes display objects A202a to A202b.
The display object A202a includes the name of the facial expression specified in step S300 and the corresponding advice information.
The display object A202b includes the name of the estimated facial habit identified in step S311 and the corresponding advice information. The advice information may be, for example, at least one of the following.
・Information regarding cosmetics to address at least one of facial expressions and skin characteristics and facial features that may appear due to facial expressions (e.g., product name, application site, amount of application, product image, etc.)
・Information regarding beauty appliances for addressing at least one of facial expressions and skin characteristics and facial features that may appear as a result of facial expressions (e.g., product name, usage instructions, product images, etc.)
・Information regarding beauty treatments to address at least one of facial expressions and skin characteristics and facial features that may be affected by facial expressions (e.g. type of treatment (massage, beauty salon, yoga, etc.), method, etc.)
・Information about facial expressions and exercises to deal with at least one of the skin characteristics and facial features that may appear as a result of facial expressions (for example, how to perform facial expression exercises by the user)
・Other advice information for dealing with at least one of facial expression habits and skin characteristics and facial features that may appear under the influence of facial expression habits According to modification example 1, the user can understand the facial expressions that the user is likely to have in the future. You can learn about habits. Therefore, by taking measures based on advice information etc. before the facial expression habit actually appears on the user's face, the facial expression habit and at least one of the skin characteristics and facial features that may appear under the influence of the facial expression habit will appear. possibility can be reduced.

(5-2) Modification example 2
Modification 2 will be explained. Modification 2 is an example in which advice information for dealing with at least one of a facial expression and at least one of skin characteristics and facial features that may appear due to the influence of the facial expression is selected based on user information.

(5-2-1) Facial expression information database The facial expression information database of Modification 2 will be explained. FIG. 11 is a diagram showing the data structure of the facial expression habit information database of Modification 2.

The facial expression information database shown in FIG. 11 stores facial expression information related to facial expression habits. The facial expression information database includes a "facial quirk ID" field, a "feature amount" field, a "personal characteristic type" field, an "advice" field, and a "facial quirk name" field. Each field is associated with each other.

The "facial expression habit ID" field, the "feature amount" field, and the "facial expression habit name" field are the same as those described in FIG. 4.

The "personal characteristic type" field stores information regarding the type of personal characteristic.

The "advice" field stores advice information (for example, text or image data) for dealing with at least one of the corresponding facial expression and the skin characteristics and facial features that may appear under the influence of the facial expression. A plurality of pieces of advice information are stored for one facial expression habit ID. The advice information is associated with information regarding types of personal characteristics.

(5-2-2) Information Processing Information processing in Modified Example 2 will be explained. FIG. 12 is a sequence diagram of facial expression habit determination processing according to modification 2. FIG. 13 is a diagram showing an example of a screen displayed in the information processing of FIG. 12.

As shown in FIG. 12, the client device 10 accepts the determination condition (S120).
Specifically, processor 12 displays screen P100 (FIG. 6) on the display.

The user operates the operation objects B100a, B100b, and B100c similarly to FIG. 6. As a result, moving image data of a moving image of the user's face corresponding to each of the display objects A100a to A100c is generated. Each moving image data is associated with a corresponding motion instruction ID. Each moving image data is stored in the storage device 11.

The processor 12 displays screen P301 (FIG. 13) on the display.

Screen P301 includes a field object F301 and an operation object B301.
Field object F301 is an object that accepts user instructions for specifying information regarding personal characteristics. The information regarding personal characteristics is, for example, information regarding the type of personal characteristics that the user self-recognizes, information regarding the determination result by an application that determines personal characteristics, or predetermined questions for the server 30 to determine personal characteristics. This is information regarding the answer to.
The operation object B301 is an object that accepts a user instruction for transmitting the generated moving image data and the user instruction input into the field object F301 to the server 30.

After step S120, the client device 10 executes a determination request (S121).
Specifically, when the user inputs a user instruction into the field object F201 and operates the operation object B301, the processor 12 transmits determination request data to the server 30.
The judgment request data includes the following information.
- Combination of the moving image data and movement instruction ID stored in the storage device 11 in step S120 - Information regarding the personal characteristics specified in the field object F301

After step S121, the server 30 identifies facial expression habits (S320).
Specifically, the processor 32 refers to the facial expression habit information database (FIG. 11), and similarly to FIG. 5, determines whether the facial movement in the video data corresponds to each facial expression habit.
The processor 32 refers to the facial expression information database (FIG. 11) and specifies information regarding the name of the facial expression determined to be applicable.

After step S320, the server 30 specifies advice information (S321).
Specifically, the processor 32 identifies the type of the user's personal characteristics based on the information regarding the personal characteristics included in the determination request data. The processor 32 refers to the facial expression habit information database (FIG. 11) and identifies advice information associated with the specified personal characteristic type in the record of the facial expression habit name specified in S320.

After step S321, the server 30 executes a determination response (S322).
Specifically, the processor 32 transmits the determination response data to the client device 10.
The judgment response data includes the following information.
・Information regarding the facial expression name specified in step S320 ・Advice information specified in step S321

After step S322, the client device 10 presents the determination result (S122).
Specifically, the processor 12 displays screen P302 (FIG. 13) on the display based on the determination response data.
Screen P302 includes display object A302.
Display object A302 includes a facial expression name and corresponding advice information.
According to the second modification, the user can obtain advice information based on user information. For example, regarding the same facial expression, a user who has the characteristic of ``persisting with one thing'' will be presented with advice information such as ``Use basic cosmetics A every day for six months,'' while the other user will be presented with advice such as ``Use basic cosmetics A every day for six months.'''' characteristic, advice information such as ``apply facial beauty device B and massage method C alternately every other week for 3 months'' is presented to the user. Therefore, the user can take measures according to the advice information suitable for the user.

(5-3) Modification example 3
Modification 3 will be explained. Modification 3 is an example in which estimated information about the user's skin condition is presented.

(5-3-1) Facial expression information database The facial expression information database of Modification 3 will be explained. FIG. 14 is a diagram illustrating the data structure of the facial expression information database of Modification 3.

The facial expression information database shown in FIG. 14 stores facial expression information related to facial expressions. The facial expression information database includes a "facial quirk ID" field, a "feature amount" field, a "facial quirk name" field, an "advice" field, and a "reference skin condition" field. Each field is associated with each other.

The "Facial expression habit ID" field, the "Feature value" field, the "Facial expression habit name" field, and the "Advice" field are the same as those described in FIG. 4.

The "reference skin condition" field stores information (for example, text) regarding the skin condition that is presumed to be possessed by the person who has the facial expression in the record. Skin conditions inferred from facial expressions are based on statistical data obtained from multiple people.

(5-3-2) Information Processing Information processing of Modified Example 3 will be explained. FIG. 15 is a sequence diagram of facial expression habit determination processing according to modification 3. FIG. 16 is a diagram showing an example of a screen displayed in the information processing of FIG. 15.

As shown in FIG. 15, the client device 10 receives a determination condition (S100) and executes a determination request (S101), similarly to FIG.

After step S101, the server 30 refers to the facial expression habit information database (FIG. 14) and executes the identification of facial expression habits (S330) in the same manner as in FIG.

After step S330, the server 30 performs skin condition estimation (S331).
Specifically, the processor 32 refers to the facial expression habit information database (FIG. 14) and specifies, as the estimated skin condition, the information regarding the reference skin condition associated with the information regarding the facial expression habit name identified in step S330. .

After step S331, the server 30 executes a determination response (S332).
Specifically, the processor 32 transmits the determination response data to the client device 10.
The judgment response data includes the following information.
- Information regarding the facial expression name specified in step S330 - Advice information specified in step S330 - Information regarding the estimated skin condition specified in step S331

After step S332, the client device 10 presents the determination result (S132).
Specifically, the processor 12 displays screen P402 (FIG. 16) on the display based on the determination response data.

Screen P402 includes display object A402.
Display object A402 includes a facial expression name, corresponding advice information, and estimated skin condition.
According to the third modification, the user can know the estimated skin condition. Therefore, the user can know the skin condition that may be the cause of the facial expression habit or the future skin condition, and more appropriately understand the facial expression habit and the skin characteristics that may appear due to the influence of the facial expression habit. and facial features.

(5-4) Modification example 4
Modification 4 will be explained. Modification 4 is an example in which one moving image data corresponds to a plurality of message contents.

(5-4-1) Information Processing Information processing of Modified Example 4 will be explained. FIG. 17 is a diagram showing an example of a screen displayed in the information processing of Modification 4.

As shown in FIG. 5, the client device 10 executes acceptance of determination conditions (S100).
Specifically, processor 12 displays screen P500 (FIG. 17) on the display.

Screen P500 includes display objects A500a to A500b and operation object B500.
Display object A500a includes a message urging the user to move his or her face.
The display object A500b includes a moving image captured by the camera 15 connected to the client device 10.
The operation object B500 is an object that accepts user instructions for shooting a moving image.

When the user operates the operation object B500, the processor 12 displays different message contents on the display object A100 every n seconds after the operation of the operation object B500 in step S100 (FIG. 5).
When the user moves each part of the face in accordance with the content of the message, the processor 12 associates the moving image data captured by the camera 15 with the elapsed time n since the operation of the operation object B100 and stores them in the storage device 11. .

The processor 12 displays screen P101 (FIG. 6) on the display.
After step S100, the client device 10 executes a determination request (S101).
Specifically, when the user operates the operation object B101, the processor 12 transmits determination request data to the server 30.
The judgment request data includes the following information.
- Combination of the moving image data stored in the storage device 11 in step S100 and the elapsed time n

After step S101, the server 30 identifies facial expression habits (S300).
Specifically, the storage device 31 stores the elapsed time n and message content in association with each other.
The processor 32 identifies the message content corresponding to the moving image data based on the combination of the moving image data and the elapsed time n included in the determination request data.
Similarly to FIG. 5, the processor 32 compares the feature amounts and specifies information regarding the name of the facial expression habit determined to be applicable and corresponding advice information.

After step S300, the server 30 executes a determination response (S301) similarly to FIG. 5.

After step S301, the client device 10 executes presentation of the determination result (S102) similarly to FIG. 5.

According to the fourth modification, one moving image data can correspond to a plurality of message contents. Therefore, the user can specify the determination conditions with a simpler operation.

(5-5) Modification example 5
Modification 5 will be explained. Modification 5 is an example in which advice information differs depending on message content.

(5-5-1) Facial expression information database The facial expression information database of modification 5 will be explained. FIG. 18 is a diagram showing the data structure of the facial expression habit information database of modification 5.

The facial expression information database shown in FIG. 18 stores facial expression information related to facial expressions. The facial expression information database includes a "facial quirk ID" field, a "feature amount" field, a "facial quirk name" field, and an "advice" field. Each field is associated with each other.
The facial expression information database is associated with the movement instruction ID.

The "facial expression habit ID" field, the "feature amount" field, and the "facial expression habit name" field are the same as those described in FIG. 4.

The "advice" field stores advice information (for example, text or image data) for dealing with at least one of the corresponding facial expression and the skin characteristics and facial features that may appear as a result of the facial expression. In modification 5, even if the facial expressions are the same, the advice information will be different if the movement instruction ID is different.

(5-5-2) Information Processing Information processing in Modified Example 5 will be described. FIG. 19 is a diagram illustrating an example of a screen displayed in information processing of modification 5.

The client device 10 receives a determination condition (S100) and executes a determination request (S101), as in FIG. 5.

After step S101, the server 30 identifies facial expression habits (S300).
Specifically, in Modification 5, the processor 32 refers to the facial expression information database (FIG. 18) associated with the movement instruction ID included in the determination request data, and determines each feature amount and the information included in the determination request data. It is compared with the feature amount calculated from the moving image data, and it is determined whether the facial movement in the moving image data corresponds to the facial expression.
The processor 32 refers to the facial expression habit information database (FIG. 18) and specifies information regarding the name of the facial expression habit determined to be applicable and corresponding advice information.

After step S300, the server 30 executes a determination response (S301) similarly to FIG. 5.

After step S301, the client device 10 presents the determination result (S102).
Specifically, the processor 12 displays screen P602 (FIG. 19) on the display based on the determination response data.

Screen P602 includes display object A602.
Display object A602 includes a facial expression name and corresponding advice information.

According to modification 5, advice information that takes the message content into consideration is presented. Therefore, the user can know more accurate advice information.

(6) Summary of this embodiment This embodiment will be summarized.

The first aspect of this embodiment is
comprising means for acquiring moving image information of the user's face (for example, a processor 32 that executes the process of step S300),
comprising means (for example, a processor 32 that executes the process of step S300) for extracting movement information regarding the movement of the user's face based on the moving image information,
comprising means (for example, a processor 32 that executes the process of step S300) for identifying the user's facial movement habits based on the movement information;
It is an information processing device (for example, server 30).

According to the first aspect, the user's facial movement habits (facial expression habits) are determined. Thereby, it is possible to generate results that allow the user to know his or her own facial expression habits.

The second aspect of this embodiment is
comprising means for presenting information regarding the identified movement habit (for example, a processor 32 that executes the process of step S301);
It is an information processing device.

According to the second aspect, results for knowing facial expression habits are presented to the user. This allows the user to know his own facial expressions.

The third aspect of this embodiment is
comprising means for acquiring user information regarding the user (for example, a processor 32 that executes the process of step S300),
comprising means (for example, a processor 32 that executes the process of step S311) for estimating the user's facial movement habits based on the user information,
comprising means (for example, processor 32 that executes the process of step S312) for presenting information regarding the estimated movement habit;
It is an information processing device.

According to the third aspect, the user can know facial expression habits that the user is likely to have in the future. Therefore, by taking measures based on advice information etc. before the facial expression habit actually appears on the user's face, the facial expression habit and at least one of the skin characteristics and facial features that may appear under the influence of the facial expression habit will appear. possibility can be reduced.

The fourth aspect of this embodiment is
The user information includes information regarding at least one of the user's personal characteristics and physical condition.
It is an information processing device.

According to the fourth aspect, the user can learn facial expressions that the user may have in the future that are correlated with at least one of personal characteristics and physical condition.

The fifth aspect of this embodiment is
The user information includes information regarding the user's skin or facial condition.
It is an information processing device.

According to the fifth aspect, the user can know facial expressions that the user may have in the future that are correlated with the condition of the skin or face.

The sixth aspect of this embodiment is
The information regarding the movement habit includes advice information for dealing with at least one of the movement habit and the skin characteristics and facial features that may appear due to the influence of the movement habit.
It is an information processing device.

According to the sixth aspect, advice information for dealing with at least one of facial expression habits and skin characteristics and facial features that may appear under the influence of facial expression habits is presented. Thereby, the user can take measures to more appropriately deal with facial expressions and at least one of the skin characteristics and facial features that may appear under the influence of facial expressions.

The seventh aspect of this embodiment is
comprising means for acquiring user information regarding the user (for example, the processor 32 that executes the process of step S320),
Means for selecting, based on user information, advice information for dealing with at least one of a movement habit and a skin feature and a facial feature that may appear due to the influence of the movement habit (for example, executing the process of step S321) processor 32);
Selected advice information is presented,
It is an information processing device.

According to the seventh aspect, advice information based on user information can be obtained. Therefore, the user can take measures according to the advice information suitable for the user.

The eighth aspect of this embodiment is
The information regarding movement habits includes estimated information about the condition of the user's skin or face.
It is an information processing device.

According to the eighth aspect, the user can know the estimated skin condition. Therefore, the user can know the skin condition that may be the cause of the facial expression habit or the future skin condition, and more appropriately understand the facial expression habit and the skin characteristics that may appear due to the influence of the facial expression habit. and facial features.

The ninth aspect of this embodiment is
The means for acquiring moving image information acquires moving image information of facial movements in response to a message prompting the user to make facial movements;
comprising means for associating and storing information regarding the message and video information;
It is an information processing device.

According to the ninth aspect, facial expression habits are identified in consideration of message content. Therefore, the user can know more accurate facial expressions. Additionally, advice information is presented that takes the message content into consideration. Therefore, the user can know more accurate advice information.

The tenth aspect of this embodiment is
comprising means for acquiring user information regarding the user (for example, a processor 32 that executes the process of step S300),
comprising means (for example, a processor 32 that executes the process of step S311) for estimating facial movement habits based on user information;
comprising means (for example, processor 32 that executes the process of step S312) for presenting information regarding the estimated movement habit;
It is an information processing device (for example, server 30).

According to the tenth aspect, the user can know facial expression habits that the user is likely to have in the future. Therefore, by taking measures based on advice information etc. before the facial expression habit actually appears on the user's face, the facial expression habit and at least one of the skin characteristics and facial features that may appear under the influence of the facial expression habit will appear. possibility can be reduced.

The eleventh aspect of this embodiment is
This is a program for causing a computer (for example, processor 32) to function as each of the above means.

(7) Other variations

The storage device 11 may be connected to the client device 10 via the network NW. The storage device 31 may be connected to the server 30 via the network NW.

Each step of the above information processing can be executed by either the client device 10 or the server 30.
One device may perform all information processing without going through a network.

In the present embodiment, an example has been shown in which the information regarding the message prompting the user to make facial movements is text, but the present embodiment is not limited to this. The information regarding the message may be a moving image. The moving image is, for example, an animated moving image of a face with various expressions, and the user operates the operation object B100 on the screen P100 while imitating the facial expressions, and the moving image data of the moving image of the user's face is generated. may be done.

In the present embodiment, an example is shown in which facial expressions are determined based on a moving image acquired from the camera 15, but the present embodiment is not limited to this. Facial expression habits may be determined based on moving images stored in the storage device 11 or the storage device 31.

In the present embodiment, an example has been shown in which the screen P100 includes the display object A100d that includes a moving image captured by the camera 15, but the present embodiment is not limited to this. The display object A100d may not be included in the screen P100. In this case, since the user cannot see his or her own face at all when moving each part of the face, it is more likely that the user's original facial expression will appear, and facial expression habits can be determined more accurately.

In this embodiment, an example is shown in which the name of facial expression and advice information are presented as information regarding facial expression, but this embodiment is not limited to this. The information regarding facial expressions that is presented may be, for example, either the name of facial expressions or advice information.

In Modification 1, an example was shown in which the name of facial expression and advice information are presented as information regarding the estimated facial expression, but the present embodiment is not limited to this. The information regarding facial expressions that is presented may be, for example, either the name of facial expressions or advice information.

In the present embodiment and Modification 1, an example was shown in which the presented facial expression name is a context related to the state (for example, "raised eyebrows"), but the present embodiment is not limited to this. The information regarding the name of the facial expression that is presented may be, for example, the context of a combination of a state, a degree, a facial part, and at least one of a situation. Specifically, the context of the combination of condition and degree (for example, ``severe raised eyebrows''), the context of the combination of condition and facial part (for example, ``right eye wrinkles''), and the condition and situation. (for example, "the corners of the mouth are hardly raised when smiling"). Further, the information regarding facial expressions that is presented may include information regarding message content.

In Modification 1, an example was shown in which the user information is information regarding the user's personal characteristics and information regarding the user's skin condition, but the present embodiment is not limited to this. The user information may be, for example, at least one of the user's personal characteristics, the user's physical condition, the user's skin condition, the user's facial condition, and the user's attributes. "Facial condition" refers to the condition of a person's face. The facial condition relates to, for example, facial blood flow, facial 3D shape, and the like. Examples of the "user attributes" include age, gender, and occupation.

Although the first modification shows an example in which the skin condition is specified in the field object F201b, the present embodiment is not limited to this. The skin condition may be identified by image analysis from moving image data generated by operating the operation objects B100a to B100c on the screen P100 (FIG. 6).

Although the second modification shows an example in which the user information is information regarding the user's personal characteristics, the present embodiment is not limited to this. The user information may be, for example, at least one of the user's personal characteristics, the user's physical condition, the user's skin condition, the user's facial condition, and the user's attributes.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Moreover, various improvements and changes can be made to the embodiments described above without departing from the spirit of the present invention. Furthermore, the above embodiments and modifications can be combined.

1 : Information processing system 10 : Client device 11 : Storage device 12 : Processor 13 : Input/output interface 14 : Communication interface 15 : Camera 30 : Server 31 : Storage device 32 : Processor 33 : Input/output interface 34 : Communication interface

Claims (14)

Equipped with means for acquiring video image information of the user's face,
comprising means for extracting movement information regarding the movement of the user's face based on the moving image information,
comprising means for identifying facial movement habits of the user based on the movement information;
comprising means for acquiring user information regarding the user,
comprising means for estimating facial movement habits of the user based on the user information;
comprising means for presenting information regarding the estimated movement habit;
The user information includes information regarding the condition of the user's skin or face ,
Information processing device. Equipped with means for acquiring video image information of the user's face,
comprising means for extracting movement information regarding the movement of the user's face based on the moving image information,
comprising means for identifying facial movement habits of the user based on the movement information;
comprising means for presenting information regarding the identified movement habit;
The information processing apparatus , wherein the information regarding the movement habit includes advice information for dealing with at least one of the movement habit and skin characteristics and facial features that may appear due to the movement habit. Equipped with means for acquiring video image information of the user's face,
comprising means for extracting movement information regarding the movement of the user's face based on the moving image information,
comprising means for identifying facial movement habits of the user based on the movement information;
comprising means for acquiring user information regarding the user,
comprising means for estimating facial movement habits of the user based on the user information;
comprising means for presenting information regarding the estimated movement habit;
The information processing apparatus, wherein the information regarding the movement habit includes advice information for dealing with at least one of the movement habit and skin characteristics and facial features that may appear due to the movement habit. comprising means for selecting, based on the user information, advice information for dealing with the movement habit and at least one of skin characteristics and facial features that may appear under the influence of the movement habit;
comprising means for presenting the selected advice information;
The information processing device according to claim 3 . Equipped with means for acquiring video image information of the user's face,
comprising means for extracting movement information regarding the movement of the user's face based on the moving image information,
comprising means for identifying facial movement habits of the user based on the movement information;
comprising means for presenting information regarding the identified movement habit;
The information regarding the movement habit includes estimated information about the condition of the user's skin or face .
Information processing device. Equipped with means for acquiring video image information of the user's face,
comprising means for extracting movement information regarding the movement of the user's face based on the moving image information,
comprising means for identifying facial movement habits of the user based on the movement information;
comprising means for acquiring user information regarding the user,
comprising means for estimating facial movement habits of the user based on the user information;
comprising means for presenting information regarding the estimated movement habit;
The information regarding the movement habit includes estimated information about the condition of the user's skin or face.
Information processing device. Equipped with means for acquiring video image information of the user's face,
comprising means for extracting movement information regarding the movement of the user's face based on the moving image information,
comprising means for identifying facial movement habits of the user based on the movement information;
The means for acquiring moving image information acquires moving image information of facial movements in response to a message urging the user to move his/her face,
comprising means for storing information related to the message and the video information in association with each other ;
Information processing device. A program for causing a computer to function as each means according to any one of claims 1 to 7. The computer is
performing the step of obtaining video information of the user's face;
Executing the step of extracting movement information regarding the movement of the user's face based on the moving image information,
identifying facial movement habits of the user based on the movement information;
performing the step of obtaining user information regarding the user;
performing a step of estimating facial movement habits of the user based on the user information;
presenting information regarding the inferred movement habit;
The user information includes information regarding the condition of the user's skin or face,
Information processing method. The computer is
performing the step of obtaining video information of the user's face;
Executing the step of extracting movement information regarding the movement of the user's face based on the moving image information,
identifying facial movement habits of the user based on the movement information;
presenting information regarding the identified movement habit;
The information processing method, wherein the information regarding the movement habit includes advice information for dealing with at least one of the movement habit and skin characteristics and facial features that may appear under the influence of the movement habit. The computer is
performing the step of obtaining video information of the user's face;
Executing the step of extracting movement information regarding the movement of the user's face based on the moving image information,
identifying facial movement habits of the user based on the movement information;
performing the step of obtaining user information regarding the user;
performing a step of estimating facial movement habits of the user based on the user information;
presenting information regarding the inferred movement habit;
The information processing method, wherein the information regarding the movement habit includes advice information for dealing with at least one of the movement habit and skin characteristics and facial features that may appear under the influence of the movement habit. The computer is
performing the step of obtaining video information of the user's face;
Executing the step of extracting movement information regarding the movement of the user's face based on the moving image information,
identifying facial movement habits of the user based on the movement information;
presenting information regarding the identified movement habit;
The information regarding the movement habit includes estimated information about the condition of the user's skin or face.
Information processing method. The computer is
performing the step of obtaining video information of the user's face;
Executing the step of extracting movement information regarding the movement of the user's face based on the moving image information,
identifying facial movement habits of the user based on the movement information;
performing the step of obtaining user information regarding the user;
performing a step of estimating facial movement habits of the user based on the user information;
presenting information regarding the inferred movement habit;
The information regarding the movement habit includes estimated information about the condition of the user's skin or face.
Information processing method. The computer is
performing the step of obtaining video information of the user's face;
Executing the step of extracting movement information regarding the movement of the user's face based on the moving image information,
identifying facial movement habits of the user based on the movement information;
The means for acquiring moving image information acquires moving image information of facial movements in response to a message urging the user to move his/her face,
executing a step of storing information related to the message and the video information in association with each other;
Information processing method.