JP7183005B2 - Skin analysis method and skin analysis system - Google Patents

Description

The present invention relates to a skin analysis method and a skin analysis system.

In recent years, research has been conducted on techniques for objectively estimating a person's visual impression. For example, Patent Literature 1 discloses a technique for extracting one or more images occurring simultaneously with the emission of a sound from a sequence of facial images of a person taken while emitting at least one sound, and evaluating clinical signs. is published.

In addition, in Patent Document 2, a plurality of tracking points are arranged in advance in an analysis area such as the outer corner of the eye of a subject, and a moving image is shot, and the amount of change in the tracking points due to changes in facial expression is tracked. A technique for analyzing the skin condition of a subject by obtaining the is disclosed.

Furthermore, Patent Literature 3 discloses a technique for a method of extracting factors that are highly correlated with facial visual impressions using dynamic information, and a method of discriminating facial visual impressions based on the factors. there is

JP 2017-502732 JP 2014-193197 JP 2016-194901

As mentioned above, research has been carried out on techniques for evaluating skin by analyzing videos that include human faces. In addition, the applicant has conducted research to analyze the structural state of the skin by analyzing the motion characteristics of the skin using moving images representing the process of facial expression changes. When performing such an analysis, accurate analysis results cannot be obtained unless the timing of changes in facial expression of the photographed person is accurately determined. Therefore, it has been required to accurately identify the start and end points of facial expressions and to improve the accuracy of extracting the part whose facial movement is to be analyzed from the input video.

Japanese Patent Application Laid-Open No. 2002-200000 discloses a technique of making a person to be photographed emit sound in accordance with a change in facial expression, recording the sound at the same time as a moving image, and extracting an image to be analyzed from sound analysis. However, since facial expression changes do not always occur at the same time as the sound is emitted, there was a problem with the accuracy of extraction of the analysis target. In addition, there is a problem in that the utterance of sound is a burden on the person to be photographed depending on the photographing location, and furthermore, a sound collector is required.

Therefore, the present invention provides skin analysis that can perform skin analysis with high accuracy by specifying an image where expression change starts and an image where expression change ends from a plurality of images representing the process of expression change and extracts an analysis target. The object is to provide a method and a skin analysis system.

In order to solve the above problems, the present invention provides a skin analysis method for analyzing skin characteristics based on dynamic information of facial expression changes,
an image acquiring step of acquiring a plurality of images representing the process of the facial expression change;
a facial part coordinate detection step of detecting a plurality of facial part coordinates indicating positions of the facial parts from the plurality of images;
a distance calculation step of calculating the distance between the coordinates of the facial parts for the plurality of images;
Based on the time-series change in the distance between the facial part coordinates, an image where the expression change starts and an image where the expression change ends are specified from the plurality of images, and the image where the expression change starts and the image where the expression change ends. an analysis target extraction step of extracting, as an analysis target, an image between images in which the facial expression change ends;
and an analysis step of analyzing skin features using the image extracted in the analysis target extraction step.

In this way, by specifying the start point and the end point of the facial expression change based on the change in the distance between the facial part coordinates from the image representing the facial expression change process, it is possible to specify the timing of the facial expression change directly from the image. can. This makes it possible to extract the analysis target more accurately than when specifying the timing of facial expression change based on voice or the like. In addition, even if there is no sound collector or if there is a reluctance to speak, the timing of facial expression changes can be specified without imposing a burden on the person being photographed, and skin analysis using images can be performed. be able to.

A preferred form of the present invention is characterized in that the analysis step includes a step of analyzing the skin characteristics of the cheek portion.
In this way, by analyzing the characteristics of the cheek skin, it is possible to obtain the analysis result of the part that occupies a large area of the entire face and tends to influence the impression.

In a preferred embodiment of the present invention, the plurality of images include a plurality of images representing expression changes of the face rotated in a range of 0 to 90 degrees with the front being 0 degrees around a vertical axis passing through the face. It is characterized by
In this way, by performing analysis using an image of the face photographed from an angle that is rotated from the front in the range of 0 to 90 degrees, the skin in areas that are likely to affect the impression of the face, such as the cheeks and the corners of the eyes, can be extracted more. Since a wide range of images can be acquired, skin analysis can be performed more accurately on the skin of the part that tends to affect the impression of the face.

In a preferred embodiment of the present invention, the plurality of images include a plurality of images representing expression changes of the face rotated in a range of 20 to 30 degrees with the front being 0 degrees around a vertical axis passing through the face. It is characterized by
In this way, by using an image of the face that has been rotated from the front by an angle of 20 to 30 degrees, the impression of the face, such as the cheeks and the corners of the eyes, can be left and right compared to the case of using the image taken from the front. It is possible to obtain an image that includes a wide range of skin that is likely to be affected, and perform more accurate skin analysis on the skin that is likely to influence the impression of the face. In addition, since the coordinates of facial parts can be detected with higher accuracy than when using an image of the face photographed at an angle that is rotated by more than 30 degrees, it is possible to accurately extract the image at which facial expression changes start and the image at which facial expression changes end. can be done.

In a preferred embodiment of the present invention, the facial part coordinate detection step includes:
It is characterized in that, as the face part coordinates, two points whose distance changes in conjunction with the change in facial expression are detected.
In this way, by detecting the two points whose distance changes in conjunction with the change in facial expression as facial part coordinates, the distance between the two points can be used to more accurately determine the image in which the facial expression starts to change and the image in which the facial expression ends. can be specified.

In a preferred embodiment of the present invention, the change in facial expression includes at least one of a vertical facial expression change in which the cheek skin is vertically stretched and a horizontal facial expression change in which the cheek skin is horizontally stretched and stretched. characterized by comprising
In this way, by using a plurality of images representing the process of facial expression changes in which the cheek skin expands and contracts in a specific direction, it is possible to analyze the motion characteristics of the skin when the skin expands and contracts in a specific direction. become.

In a preferred embodiment of the present invention, the facial expression change includes a plurality of facial expression changes that expand and contract the skin of the cheek portion in the same direction,
The analysis target extracting step is characterized by specifying an image where various facial expression changes start and an image where the various facial expression changes end, and extracting images between the images where the various facial expression changes start and end as analysis targets. do.
In this way, by acquiring an image containing a plurality of types of facial expression changes in which the cheek skin expands and contracts in the same direction, it is possible to analyze in more detail the motion characteristics of the skin associated with the expansion and contraction of the skin in a specific direction.

In a preferred embodiment of the present invention, the image acquiring step, the face part coordinate detecting step, the distance calculating step, and the analysis target extracting step are performed by combining the plurality of images including the vertical expression change and the horizontal expression change. performed for each of the plurality of images comprising;
The analysis step uses the analysis target in the plurality of images including the vertical facial expression change and the analysis target in the plurality of images including the horizontal facial expression change extracted in the analysis target extraction step. , characterized by analyzing the characteristics of the skin.
In this way, by extracting and analyzing the facial expression changes in the vertical direction and the facial expression changes in the horizontal direction, it is possible to obtain the motion characteristics of the skin in different directions and perform more detailed analysis. Become.

In a preferred embodiment of the present invention, a facial expression action evaluation step of evaluating the magnitude of the facial expression change is included between the distance calculation step and the analysis target extraction step,
The facial expression motion evaluation step evaluates the magnitude of the facial expression change based on the change in the distance between the facial part coordinates, and advances to the analysis target extraction step when the facial expression change magnitude exceeds a predetermined value. It is characterized by
In this way, by evaluating the magnitude of facial expression change, if the predetermined criteria are not met, the image is acquired again by instructing photography again, and the magnitude of facial expression change is used for skin analysis. Skin analysis can be performed on a series of images that are sufficient. Thereby, the accuracy of skin analysis can be improved.

In a preferred embodiment of the present invention, the distance calculation step includes a step of calculating a plurality of distances between the facial part coordinates,
The analysis target extracting step identifies an image in which the expression change starts and an image in which the expression change ends from the plurality of images based on the distance between the plurality of facial part coordinates, and the expression change starts. An image between an image where the facial expression change ends and an image between the image where the facial expression change ends is extracted as an analysis target.
In this way, by using a plurality of distances between facial part coordinates, it is possible to more accurately identify an image in which a change in facial expression starts and an image in which a change in facial expression ends. In addition, it becomes possible to extract analysis targets for a plurality of types of facial expression changes.

A preferred embodiment of the present invention is characterized in that the facial part coordinates include at least coordinates indicating the position of the lip contour.
In this way, by detecting the coordinates indicating the position of the lip contour as the facial part coordinates, the effect of reducing the influence of individual facial expressions on extraction of analysis targets can be expected. In addition, since the coordinates indicating the position of the lip contour move significantly in conjunction with the facial expression action, it is possible to more accurately specify the image where the facial expression change starts and the image where the facial expression change ends.

In a preferred embodiment of the present invention, the distance calculation step further includes a reference distance calculation step of calculating a reference distance for standardizing the distance between the facial part coordinates,
The analysis target extracting step specifies an image where the expression change starts and an image where the expression change ends, using a value obtained by standardizing the distance between the facial part coordinates based on the reference distance. do.
In this way, by standardizing the distances between facial part coordinates based on the reference distance, it is possible to reduce the effects of individual differences in face size and the like, magnification at the time of photographing, distance from the camera, and the like. As a result, it is possible to more accurately specify the timing of facial expression change and extract an analysis target.

The present invention is a skin analysis system for analyzing skin characteristics based on dynamic information of facial expression changes,
an image acquiring means for acquiring a plurality of images representing the process of facial expression change;
face part coordinate detection means for detecting a plurality of face part coordinates of the face for the plurality of images;
distance calculation means for calculating the distance between the facial part coordinates for the plurality of images;
Based on the time-series change in the distance between the facial part coordinates, an image where the expression change starts and an image where the expression change ends are specified from the plurality of images, and the image where the expression change starts and the image where the expression change ends. an analysis target extracting means for extracting as an analysis target an image between images in which the facial expression change ends;
and analysis means for analyzing the characteristics of the skin using the image extracted by the analysis target extraction means.

From multiple images showing the process of facial expression changes, it is possible to identify the image where facial expression changes start and the image where facial expression changes end, extract images to be analyzed, and perform highly accurate skin analysis. can.

FIG. 2 is a schematic diagram of a facial expression reproduced by a person to be photographed in Embodiment 1 of the present invention; 1 is a functional block diagram of a skin analysis system according to Embodiment 1 of the present invention; FIG. 4 is a flowchart showing a skin analysis method according to Embodiment 1 of the present invention; 4 is a flow chart showing processing from image acquisition to analysis target extraction in Embodiment 1 of the present invention. FIG. 2 is a diagram showing the photographing direction of a person to be photographed in Embodiment 1 of the present invention; FIG. 4 is a diagram showing distances between facial part coordinates according to Embodiment 1 of the present invention; 4 is a flow chart showing processing of an analysis target extraction step in Embodiment 1 of the present invention. FIG. 10 is a diagram showing an example of distances between facial part coordinates according to Embodiment 2 of the present invention; It is a figure which shows an example of the reference distance in Embodiment 2 of this invention.

<Embodiment 1>
An embodiment of the present invention will be described in detail below with reference to the drawings. In the following description, facial part coordinates are coordinates indicating the positions of parts on the face for tracking facial movements and analyzing changes in facial expressions. A part in the present invention refers to any feature such as eyes, nose, facial contours, and the like. For example, it is possible to detect and use the upper and lower edge points and the left and right edge points of the lip contour as facial part coordinates.

Also, the analysis target is a set of images to be subjected to skin analysis. In the present embodiment, skin analysis is performed by specifying an image at which facial expression change starts and an image at which facial expression change ends, and analyzing the movement using the image in the middle of facial expression change included between them.

In the present embodiment, a moving image showing the process of vertical facial expression change in which cheek skin is stretched and contracted in the vertical direction and a horizontal facial expression process in which cheek skin is stretched and contracted in the horizontal direction are shown. A moving image and are acquired respectively, and skin analysis is performed by extracting an analysis target for each of them.

FIG. 1 is a diagram showing facial expressions reproduced by a person to be photographed in this embodiment. In this embodiment, as shown here, the person to be photographed has an expressionless expression (Fig. 1(a)), an expression with the mouth open vertically (hereinafter referred to as an "a" expression, Fig. 1(b)), and an expression with the mouth open. There are four facial expressions: a facial expression of opening the mouth to the side (“I” facial expression, FIG. 1(c)) and a facial expression of pursed lips (“U” facial expression, FIG. 1(d)).

In the present embodiment, as the facial expression changes in the vertical direction, the person to be photographed changes facial expressions in the following order: expressionless, "ah" expression, "uh" expression, "ah" expression, "uh" expression, and expressionless expression. Reproduce. That is, the vertical facial expression change from expressionless to "a" facial expression, the vertical facial expression change from "a" facial expression to "u" facial expression, and the vertical facial expression change from "u" facial expression to "a" facial expression Skin analysis is performed by acquiring a moving image including a vertical facial expression change and a vertical facial expression change from an "up" expression to an expressionless expression.

In addition, as the change in facial expression in the horizontal direction, the photographed person reproduces facial expressions in the following order: expressionless, "I" expression, "U" expression, "I" expression, "U" expression, and expressionless expression. That is, the horizontal facial expression change from expressionless to "i" facial expression, the lateral facial expression change from "i" facial expression to "u" facial expression, and the horizontal facial expression change from "u" facial expression to "i" facial expression Skin analysis is performed by acquiring a moving image including a horizontal facial expression change and a horizontal facial expression change from a "U" facial expression to an expressionless facial expression.

FIG. 2 is a functional block diagram of the skin analysis system according to this embodiment. The skin analysis system according to this embodiment is configured such that a server device 1 and a terminal device 2 can communicate with each other. The arrows between each means in FIG. 2 indicate the direction of information flow and instructions.

The server device 1 includes a vertical facial expression change analysis target extraction unit, a horizontal facial expression change analysis target extraction unit, storage means 16 , and analysis means 17 . Further, the vertical facial expression change analysis target extraction unit and the horizontal facial expression change analysis target extraction unit are respectively an image acquisition unit 11, a facial part coordinate detection unit 12, a distance calculation unit 13, a facial expression motion evaluation unit 14, and an analysis unit. and a target extraction means 15 .

The storage means 16 stores the information of the image acquired by the image acquisition means 11, the facial part coordinates detected by the facial part coordinate detection means 12, the distance between the facial part coordinates calculated by the distance calculation means 13, the analysis object extraction means 15 specified It stores information such as the image at which the facial expression change starts and the image at which it ends.

The server device 1 includes an arithmetic unit such as a CPU (Central Processing Unit), a main storage device such as a RAM (Random Access Memory), an auxiliary storage device such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), and a flash memory. , a general computer device such as a server device equipped with various input/output devices including communication means with the terminal device 2 can be used. More specifically, programs for operating the server device 1 as each means described later are stored in the auxiliary storage device in advance or by duplication from a recording medium, etc., and those programs are developed on the main storage device. The computer device can be used as the server device 1 in the skin analysis system according to the present embodiment by performing calculations by the calculation device and controlling the input/output means.

The terminal device 2 includes an imaging unit 21 that captures a process of facial expression changes, and a display unit 22 that displays the captured image, skin analysis results, and the like. As the terminal device 2, a general computer device equipped with an arithmetic device, a main memory device, an auxiliary memory device, a communication means with the server device 1, an imaging means such as a digital camera, various input/output devices, etc. is used. can do. For example, a tablet terminal or the like having a camera function may be used. Specifically, for example, the terminal device can be used as the terminal device 2 in the skin analysis system according to the present embodiment by installing dedicated application software for operating the terminal device as each means described later. .

FIG. 3 is a flow chart showing a skin analysis method by the skin analysis system. First, in step S11, the vertical facial expression change analysis target extraction unit extracts an analysis target for a moving image representing the process of vertical facial expression change. Next, in step S12, the horizontal facial expression change analysis target extracting unit extracts an analysis target for the moving image representing the process of horizontal facial expression change. In this embodiment, the vertical expression change analysis target extraction step of step S11 is followed by the horizontal expression change analysis target extraction step of step S12. There are no restrictions on the order in which the directional expression change analysis target extraction step is performed, and these orders may be reversed.

When step S12 ends, the analysis means 17 performs analysis using the analysis targets extracted in steps S11 and S12, transmits the analysis results to the terminal device 2, and the display means 22 displays the analysis results on the terminal device 2. to end the process. Here, in the analysis step of step S13, the storage means 16 may store the analysis result.

FIG. 4 is a flow chart showing processing in the vertical facial expression change analysis target extraction step and the horizontal facial expression change analysis target extraction step in steps S11 and S12 of FIG. These steps are executed by the vertical facial expression change analysis target extraction unit and the horizontal facial expression change analysis target extraction unit, respectively. In other words, the vertical facial expression change analysis target extraction unit acquires videos including vertical facial expression changes, and the horizontal facial expression change analysis target extraction unit acquires videos including horizontal facial expression changes. extracted.

First, in the image acquisition step of step S21, the image acquisition unit 11 causes the display unit 22 to display information prompting the user to shoot a moving image, and the imaging unit 21 shoots the moving image. The image acquisition means 11 acquires information on the moving image captured by the imaging means 21 and records it in the storage means 16 . FIG. 5 is a diagram showing directions in which moving images are captured in this embodiment. In the present embodiment, the image acquiring means 11 captures the moving image captured by the imaging means 21 from a direction rotated in a range of 20 to 30 degrees with the front being 0 degrees around the vertical axis passing through the face of the subject. Get information.

When the photographing by the imaging means 21 and the acquisition of the moving image by the image acquiring means 11 are completed in the image acquiring step of step S21, the face part coordinate detecting means 12 detects the moving image acquired in step S21 in the face part coordinate detecting step of step S22. Multiple face part coordinates of the included face are detected for each frame. Here, at least one set of two points whose distance changes in association with changes in facial expression included in the moving image is detected. In particular, it is preferable to detect at least one set of two points whose distance changes in conjunction with expansion and contraction of the cheek skin when the facial expression changes. As a result, it is possible to more accurately specify the image in which the facial expression change that expands and contracts the skin of the cheek portion starts and the image in which the facial expression change ends.

In this embodiment, the vertical facial expression change analysis target extraction unit extracts two points at the upper and lower ends of the lips, the temple and the chin, and the horizontal facial expression change analysis target extraction unit extracts the upper and lower ends of the lip contour. Two points and two points at the left and right ends of the lip contour are detected as a set.

Here, the distance between the two points at the upper and lower ends of the lips and the two points at the temple and chin increased when the cheek skin was stretched in the vertical direction, and the cheek skin was shortened in the vertical direction. sometimes the distance becomes smaller. This makes it possible to specify the timing of various vertical facial expression changes. Also, the distance between the two points on the left and right ends of the lip contour becomes smaller when the skin of the cheek portion is stretched in the horizontal direction, and the distance becomes larger when the skin of the cheek portion is contracted in the horizontal direction. Furthermore, by using the distance between the two points at the upper and lower ends of the lip contour, for example, the change in the distance between the two points at the left and right ends of the lip contour is small. The starting image and the ending image of facial expression change can be specified, and the timing of various lateral facial expression changes can be specified.

In this manner, in the face part coordinate detection step, it is preferable to detect a set of two points whose distance changes in conjunction with expansion and contraction of the cheek skin. In addition, by detecting a plurality of pairs of such two points and using the distances thereof, an image at which various facial expression changes start and an image at which facial expression changes end are determined for an image including a plurality of facial expression changes. can be specified.

Any generally known method can be used to detect facial part coordinates. For example, application software developed using a commercially available development kit may be used. As a development kit, for example, Face Sensing Engine (registered trademark) (https://www.oki.com/jp/fse/function/#fp, last viewed: November 16, 2017), etc. are used. be able to.

When detection of face part coordinates for the entire moving image is completed in the face part coordinate detection step of step S22, the distance calculation means 13 calculates the distance between the face part coordinates in step S23. Distance calculation is performed for each frame of the entire moving image, and the calculation result is stored in the storage means 16 .

FIG. 6 is a diagram showing distances between facial part coordinates used in this embodiment. FIG. 6A shows facial part coordinates used in the vertical facial expression change analysis target extraction unit, and FIG. 6B shows facial part coordinates used in the horizontal facial expression change analysis target extraction unit. As shown here, in the present embodiment, the vertical facial expression change analysis target extracting unit extracts the horizontal facial expression change analysis target by extracting the distance between two points at the upper and lower ends of the lips and the distance between the two points on the temple and the chin. In part, the distance between two points on the upper and lower ends of the lip contour and the distance between two points on the left and right ends of the lip contour are calculated. In this way, by setting the facial part coordinates to be used according to a series of facial expression changes to be analyzed, it is possible to accurately identify the start and end of facial expression changes with a small amount of information.

In the facial expression motion evaluation process of step S24, the magnitude of facial expression change is evaluated based on the change in the distance between the facial part coordinates calculated in step S23. Specifically, for example, the distance between the coordinates of the facial parts when the person to be photographed reproduces an expressionless expression, and the expressions other than expressionlessness ("A" expression, "I" expression, "U" expression) can be calculated as an evaluation value of facial expression change. By using the ratio in this way, the influence of individual differences such as face size can be reduced.

Next, in step S25, the evaluation value calculated in step S24 is compared with a reference value, and if the evaluation value exceeds the reference value, the process proceeds to step S26 for extracting an analysis target. If it does not meet the reference value, the process returns to the image acquisition step of step S21, and the moving image is acquired again by a method such as prompting the user to shoot the moving image again. Steps S22 to S25 are performed for the reacquired moving image, and each of the facial part coordinate detection process, the distance calculation process, and the facial expression action evaluation process is performed.

In this embodiment, since a moving image to be acquired includes a plurality of facial expression changes, in the facial expression action evaluation process, all facial expression changes included in the moving image are compared with a reference value. When the evaluation value of exceeds the reference value, proceed to the analysis target extraction step.

In step S26, the analysis target extracting means 15 extracts the analysis target. Here, each process shown in FIG. 4 moves to the next process after the processing of the entire moving image is completed. That is, for example, the facial expression action evaluation process of step S24 is started after the calculation of the distances of the facial part coordinates is completed for all the frames constituting the moving image in the distance calculation process of step S23.

FIG. 7 is a flowchart showing an example of processing in the analysis target extraction step. Note that the processing shown in FIG. 7 is processing performed for each distance between two points of facial part coordinates detected as a set. In this embodiment, since a plurality of distances between facial part coordinates are used to extract an analysis target, this process is performed for each distance between two points of facial part coordinates detected as a set, and the combination of the results Extract the analysis target. Specifically, the analysis target is extracted using the distance between the coordinates of the two facial parts detected as a pair, and the facial expression changes that could not be extracted at that time are calculated using the distance between the coordinates of the other two facial parts. Analysis targets can be extracted by performing the processing shown in FIG. 7 using the distance.

First, in step S31, the analysis target extracting means 15 calculates the absolute value of the difference (change amount) of the distance between the facial part coordinates between the frames. Next, in step S32, a vertex (maximum point) having a distance difference between facial part coordinates larger than a reference value is detected. That is, in each process of a plurality of facial expression changes included in the moving image, the point at which the distance between the facial part coordinates changes the most is detected.

In step S33, a frame is specified in which the difference in distance between facial part coordinates before and after the vertex detected in step S32 is less than a reference value. As a result, the frame in which the difference in the distance between the facial part coordinates is less than the reference value immediately before the vertex detected in step S32 is specified as the frame in which the facial expression change starts. A frame in which the difference in distance is less than the reference value is specified as a frame in which the change in facial expression has ended, and is recorded in the storage means 16.例文帳に追加As a result, the analysis target extracting means 15 extracts the frames included between the frame at which the facial expression change started and the frame at which the facial expression change ended as the analysis target.

It should be noted that, when performing the processing shown in FIG. 7, filtering or the like may be performed as appropriate. For example, in step S31, noise is removed by applying a low-pass filter (LPF) to temporal changes in distance between facial part coordinates (change in distance between facial part coordinates in each frame). After that, the difference between frames may be calculated. In addition, in order to remove noise, the absolute value of the difference in the distance between the facial part coordinates between the frames calculated in step S31 may be set to 0 when it is less than a predetermined value. can.

As described above, facial part coordinates of a face are detected from a plurality of images, and based on the distance between the facial part coordinates, an image where expression change starts and an image where expression change ends are specified, and images included in between are specified. is extracted as an analysis target, it is possible to accurately identify a frame including a predetermined facial expression change. As a result, it is possible to more accurately analyze the motion characteristics of the skin when a predetermined facial expression changes, and to improve the accuracy of the skin analysis.

Furthermore, by detecting two or more sets of two points whose distance changes in conjunction with contraction of the cheek skin as facial part coordinates, and using the distances of each set in combination, multiple types of facial expression changes are continuously performed. It is possible to specify an image in which various facial expression changes start and an image in which various facial expression changes end from among a plurality of images of the situation. Thus, for a series of facial expression changes including a plurality of facial expression changes, the timing of various facial expression changes included therein can be specified by uniform processing.

In this embodiment, for example, a vertical facial expression change from an expressionless expression to an "ah" facial expression, a vertical facial expression change from an "ah" facial expression to an "uh" facial expression, and a vertical facial expression change from an "ah" facial expression to an "uh" facial expression The processing shown in FIG. 4 is performed on a moving image of facial expression changes in which a vertical facial expression change to a facial expression of "Ah" and a vertical facial expression change from a facial expression of "U" to expressionless are performed continuously. It can be carried out. As a result, it is possible to specify the timings of various facial expression changes through uniform processing.

Further, according to the present invention, since information such as sound other than the image is not required, the burden on the person to be photographed is small, and the analysis target can be extracted without using a sound collector or the like.

<Embodiment 2>
Other embodiments of the present invention will be described below. In addition, the same code|symbol is attached|subjected about the structure similar to Embodiment 1, and description is abbreviate|omitted. In this embodiment, a configuration using other points as facial part coordinates is exemplified. Further, in the present embodiment, in order to reduce the effects of individual differences in face size and the like, magnification at the time of photographing, distance from the camera, etc., the distance between face part coordinates is standardized according to a predetermined standard. Extract the analysis target.

FIG. 8 is a diagram showing an example of facial part coordinates used for extracting an analysis target for facial expression changes in the vertical direction in this embodiment. As described above, in the present embodiment, in the face part coordinate detection step, the face part coordinate detection means 12 detects the coordinates of a plurality of points on each of the left and right cheeks and the points on the chin in addition to the points on the upper and lower ends of the lip contour. Detect as facial part coordinates. Then, in the distance calculation step (S23 in FIG. 4), the distance calculation means 13 calculates the distance between each point on the left and right cheeks and the point on the chin, and based on the calculation result, the analysis target is extracted.

In this way, by detecting a plurality of points from the left and right sides of the face and using a plurality of sets of two points whose distance changes according to the change in facial expression, it is possible to extract the analysis target with higher accuracy. Similarly, for lateral facial expression changes, a plurality of pairs of facial part coordinates may be used to extract analysis targets.

Further, in the present embodiment, in the distance calculation process, a reference distance calculation process for calculating a reference distance for standardizing the distance between facial part coordinates is executed. FIG. 9 is a diagram showing examples of reference distances that can be used in this embodiment. In this embodiment, the distance between the left and right eyes (the distance of "A" in the illustrated example) is used as the reference distance to standardize the distance between the facial part coordinates. Specifically, for example, the distance between the facial part coordinates can be standardized by calculating the distance of A and dividing all the distances between the facial part coordinates by A.

In addition, an arbitrary distance that is considered to be able to correct the influence of individual differences, such as the length of the vertical line drawn from the line connecting the left and right eyes to the chin (distance "B" in the example shown), is used as the reference distance. It is preferable to use Further, a configuration may be adopted in which a plurality of reference distances are combined to standardize the distance between facial part coordinates. As the reference distance, it is sufficient to use a distance relating to a point indicating an arbitrary feature on the face, and there is no particular limitation.

Then, in the facial expression action evaluation step (S24 in FIG. 4) to the analysis target extraction step (S26 in FIG. 4), the analysis target is extracted using the value obtained by standardizing the distance between the facial part coordinates by the reference distance. . The processing here is the same as in the first embodiment, and in this embodiment, instead of the distance between facial part coordinates, a value standardized by a reference distance is used.

As described above, in the present embodiment, an analysis target is extracted using a value obtained by standardizing the distance between facial part coordinates. By doing so, it is possible to reduce the influence of individual differences and extract the analysis target with higher accuracy.

Note that the configurations described in each of the above embodiments are examples of the present invention, and the present invention is not limited to these embodiments. For example, the content and number of facial part coordinates to be used may be arbitrarily determined, and it is preferable to configure the system so that the change in distance can be analyzed more effectively. Also, the execution order of each process may be changed arbitrarily, and for example, some processes may be executed in parallel. In addition, the configuration can be arbitrarily changed without changing the gist of the present invention.

1 Server Device 11 Image Acquisition Means 12 Face Part Coordinate Detection Means 13 Distance Calculation Means 14 Facial Expression Action Evaluation Means 15 Analysis Object Extraction Means 16 Storage Means 17 Analysis Means 2 Terminal Device 21 Imaging Means 22 Display Means

Claims (12)

A skin analysis method for analyzing skin characteristics based on dynamic information of changes in facial expression,
an image acquiring step of acquiring a plurality of images representing the process of the facial expression change;
a facial part coordinate detection step of detecting a plurality of facial part coordinates indicating positions of the facial parts from the plurality of images;
a distance calculation step of calculating the distance between the coordinates of the facial parts for the plurality of images;
Based on the time-series change in the distance between the facial part coordinates, an image where the expression change starts and an image where the expression change ends are specified from the plurality of images, and the image where the expression change starts and the image where the expression change ends. an analysis target extraction step of extracting, as an analysis target, an image between images in which the facial expression change ends;
an analysis step of analyzing skin characteristics using the image extracted in the analysis target extraction step ;
The skin analysis method, wherein, in the facial part coordinate detection step, two points whose distance changes in association with the change in facial expression are detected as the facial part coordinates. 2. The skin analysis method according to claim 1, wherein said analysis step includes a step of analyzing skin characteristics of cheek portions. wherein said plurality of images include a plurality of images representing expression changes of said face rotated in a range of 0 to 90 degrees with the front being 0 degrees around a vertical axis passing through said face. The skin analysis method according to claim 1 or 2. wherein said plurality of images include a plurality of images representing expression changes of said face rotated in a range of 20 to 30 degrees with the front being 0 degrees around a vertical axis passing through said face. The skin analysis method according to claim 1 or 2. The expression change includes at least one of a vertical expression change that stretches the skin of the cheek part in the vertical direction and a horizontal expression change that stretches the skin of the cheek part in the horizontal direction, The skin analysis method according to any one of claims 2-4 . The facial expression change includes a plurality of facial expression changes that expand and contract the skin of the cheek part in the same direction,
The analysis target extracting step is characterized by specifying an image where various facial expression changes start and an image where the various facial expression changes end, and extracting images between the images where the various facial expression changes start and end as analysis targets. The skin analysis method according to claim 5 , wherein The image acquiring step, the facial part coordinate detecting step, the distance calculating step, and the analysis target extracting step include the plurality of images including the vertical expression change, the plurality of images including the horizontal expression change, is executed for each of the
The analysis step uses the analysis target in the plurality of images including the vertical facial expression change and the analysis target in the plurality of images including the horizontal facial expression change extracted in the analysis target extraction step. 7. The skin analysis method according to claim 5 or 6 , characterized in that the characteristics of the skin are analyzed. including a facial expression action evaluation step of evaluating the magnitude of the facial expression change between the distance calculation step and the analysis target extraction step;
The facial expression motion evaluation step evaluates the magnitude of the facial expression change based on the change in the distance between the facial part coordinates, and advances to the analysis target extraction step when the facial expression change magnitude exceeds a predetermined value. The skin analysis method according to any one of claims 1 to 7 , characterized by: The distance calculation step includes a step of calculating a plurality of distances between the facial part coordinates,
The analysis target extracting step identifies an image in which the expression change starts and an image in which the expression change ends from the plurality of images based on the distance between the plurality of facial part coordinates, and the expression change starts. The skin analysis method according to any one of claims 1 to 8 , wherein an image between an image where the expression change ends and an image between the image where the facial expression change ends is extracted as an analysis target. The skin analysis method according to any one of claims 1 to 9 , wherein the facial part coordinates include at least coordinates indicating the position of a lip contour. The distance calculation step further includes a reference distance calculation step of calculating a reference distance for standardizing the distance between the facial part coordinates,
The analysis target extracting step specifies an image where the expression change starts and an image where the expression change ends, using a value obtained by standardizing the distance between the facial part coordinates based on the reference distance. The skin analysis method according to any one of claims 1 to 10 . A skin analysis system for analyzing skin characteristics based on dynamic information of changes in facial expression,
an image acquiring means for acquiring a plurality of images representing the process of facial expression change;
face part coordinate detection means for detecting a plurality of face part coordinates indicating the coordinates of the face part for the plurality of images;
distance calculation means for calculating the distance between the facial part coordinates for the plurality of images;
Based on the time-series change in the distance between the facial part coordinates, an image where the expression change starts and an image where the expression change ends are specified from the plurality of images, and the image where the expression change starts and the image where the expression change ends. an analysis target extracting means for extracting as an analysis target an image between images in which the facial expression change ends;
analysis means for analyzing skin characteristics using the image extracted by the analysis target extraction means ;
The skin analysis system, wherein the facial part coordinate detection means is means for detecting, as the facial part coordinates, two points whose distance changes in association with the change in facial expression.

Images