JP2021182399A - Facial appearance impression determination method, facial impression analyzer, facial impression analysis program, facial appearance impression analysis method, and makeup effect determination method - Google Patents

Abstract

To provide a method of extracting facial parts that significantly affect the facial appearance impression and factors having strong correlation with facial appearance impression, and a method of determining the facial appearance impression on the basis of the factors.SOLUTION: A facial appearance impression determination method is provided, comprising an impression determinant parts extraction step S13 for extracting impression determinant parts that determine the facial appearance impression, and a physical quantity measurement step S14 for measuring physical quantities of skin variation caused by change in facial expression. A facial appearance impression of a subject is determined on the basis of the physical quantities measured in the physical quantity measurement step S14.SELECTED DRAWING: Figure 1

Description

The present invention is a method for extracting a determination part of a facial appearance impression, a method for extracting a determinant of a facial appearance impression, a method for distinguishing the appearance impression of a face, a face impression analysis device, a face impression analysis program, and an analysis of the appearance impression of a face. It's about the method.

People give a wide variety of impressions to others due to various factors such as their appearance and behavior. Among them, visual information, especially the appearance of the face, can be said to be one of the very big factors that determine the impression of the person.

Therefore, research is being conducted on techniques for objectively estimating the visual impression of a human face. For example, in Patent Document 1, a plurality of tracking points are arranged in advance in an analysis area such as the outer corner of the eyes of a subject, a moving image is taken, and the amount of change in the tracking points due to a change in facial expression is tracked to obtain a compression rate of the skin. The technology to analyze the skin condition of the subject by acquiring the above is open to the public.

Japanese Unexamined Patent Publication No. 2014-193197

As described above, parameters for objectively estimating the visual impression of the subject's face have been studied, but this is merely to acquire a specific parameter from a specific part of the subject's face.

However, the impression of human appearance is formed by various factors, and a more accurate estimation means has been desired.

Therefore, the present invention provides a method for extracting a portion having a large influence on the appearance impression of the face, a method for extracting a factor having a high correlation with the appearance impression of the face, and a method for distinguishing the appearance impression of the face based on the factor. Is the subject.

In order to solve the above problems, the method for extracting the determination site of the appearance impression of the face according to the present invention is
A visual evaluation process that presents dynamic information on changes in facial expressions to the evaluator and causes the evaluator to evaluate the appearance impression of the face.
It is characterized by including a line-of-sight analysis step of analyzing the line-of-sight data of the evaluator who visually observes the dynamic information and analyzing the region of the dynamic information that the evaluator gazes at.

Thereby, it is possible to extract a part that a person pays attention to when evaluating the appearance impression of the face based on dynamic information, that is, a part that determines the appearance impression of the face.

A preferred embodiment of the present invention is characterized in that the appearance impression of the face is an age impression.

The method for extracting a determinant of the appearance impression of a face according to the present invention is as follows.
A physical quantity measurement process for measuring the physical quantity of skin changes caused by changes in facial expressions at the impression determination site that determines the appearance impression of the face, and
It is characterized by including a factor extraction step of extracting a factor having a high correlation with the appearance impression of the face from the physical quantity.
This makes it possible to specify the physical quantity used in the discrimination of the appearance impression of the face, which will be described later.

In a preferred embodiment of the present invention, the physical quantity measuring step includes a step of measuring an amount related to the marker as the physical quantity from a motion capture moving image taken by setting a plurality of markers on the face. ..

A preferred embodiment of the present invention is characterized in that the amount related to the marker includes at least one of the position, the amount of movement, the magnitude of velocity, the direction of velocity, the magnitude of acceleration, and the direction of acceleration of the marker. do.

In a preferred embodiment of the invention, the amount with respect to the marker comprises an amount with respect to the time variation of the distance between the plurality of markers.

In a preferred embodiment of the invention, the amount with respect to the marker comprises a time variation of a value indicating the area and / or shape of the surface formed by the three or more markers.

In a preferred embodiment of the present invention, the physical quantity measuring step includes a grid point setting step for setting a plurality of grid points on the face.
The physical quantity is characterized by including a quantity related to the grid points.

In a preferred embodiment of the present invention, the quantity with respect to the grid points includes a quantity with respect to the magnitude and / or orientation of the velocity vector for each grid point.

In a preferred embodiment of the present invention, the method for extracting a determinant of the facial expression impression presents dynamic information including the facial expression change to the evaluator to obtain a visual evaluation value of the facial expression impression. Has an acquisition process,
The factor extraction step is characterized by including analyzing the correlation between the physical quantity and the visual evaluation value and extracting the factor.
By using the visual evaluation value, it is possible to extract a factor that has a high correlation with the impression that a person actually receives when visually observing the face.

In a preferred embodiment of the invention, the method of extracting a determinant of the appearance impression of a face comprises a physiological quantity measuring step of measuring a physiological quantity relating to the skin.
The factor extraction step is characterized by analyzing the correlation between the physical quantity and the physiological quantity to extract the factor.
By using a physiological amount, it is possible to extract a determinant of the appearance impression of the face, which is highly correlated with the physiological phenomenon in the living tissue such as the skin.

In a preferred embodiment of the invention, the physiological amount is characterized by skin surface and / or internal properties at the impression-determining site.

A preferred embodiment of the present invention is characterized in that the appearance impression of the face is an age impression.

In a preferred embodiment of the present invention, the physical quantity is a quantity relating to the movement of the skin at the impression-determining site.

The method for distinguishing the appearance impression of a face according to the present invention is as follows.
It is characterized in that the appearance impression of a subject's face is discriminated based on the physical quantity of skin changes caused by changes in facial expressions at the impression determination site that determines the appearance impression of the face.
By using the physical quantity generated at the impression determination site, it is possible to more accurately discriminate the appearance impression of the face given to others.

A preferred embodiment of the present invention is characterized in that the physical quantity includes at least one of the magnitude, direction of velocity, magnitude of acceleration, and direction of acceleration of feature points contained in the face. do.

In a preferred embodiment of the invention, the physical quantity is characterized by comprising a time variation of a distance between one reference point contained in the face and each of at least two feature points contained in the face. do.

In a preferred embodiment of the present invention, the physical quantity is characterized by including a time-varying quantity of an area surrounded by at least three feature points contained in the face.

In a preferred embodiment of the present invention, the appearance of the face of the subject is discriminated from the measured value of the physical quantity of the subject based on the correlation between the physical quantity and the visual evaluation value of the appearance impression of the face. It is characterized by including calculating a visual evaluation value of an impression.

In a preferred embodiment of the present invention, the discrimination of the appearance impression of the face is based on the correlation between the physical quantity and the physiological quantity related to the skin, and the physiological quantity related to the skin of the subject is obtained from the measured value of the physical quantity for the subject. It is characterized by including calculation.

In a preferred embodiment of the invention, the physiological amount is characterized by skin surface and / or internal properties at the impression-determining site.

A preferred embodiment of the present invention is characterized in that the appearance impression of the face is an age impression.

In a preferred embodiment of the present invention, the physical quantity is a quantity relating to the movement of the skin at the impression-determining site.

The face impression analysis device according to the present invention is
It is a facial impression analysis device that analyzes the appearance impression of the subject's face based on the movement of the skin when the facial expression changes.
A moving image acquisition means for acquiring a moving image including the process of changing facial expressions,
It is characterized by comprising a physical quantity measuring means for measuring a physical quantity generated by the change in facial expression from the moving image.
By using the device having such a configuration, it is possible to carry out the method for discriminating the appearance impression of the face according to the present invention.

In a preferred embodiment of the present invention, the physical quantity measuring means comprises a feature point extracting means for extracting a plurality of feature points from the moving image, and a feature point extracting means.
A motion vector specifying means for specifying a motion vector of the feature point from the moving image,
It is characterized by having a motion vector analysis means for analyzing the motion vector and calculating the physical quantity.

In a preferred embodiment of the present invention, the physical quantity is characterized by including a quantity related to a change in magnitude and / or orientation of the motion vector over time.

In a preferred embodiment of the present invention, the physical quantity measuring means comprises a feature point extracting means for extracting a plurality of feature points from the moving image, and a feature point extracting means.
A feature point coordinate calculation means for calculating the coordinates of the feature points in each frame of the moving image, and
It is characterized by having a feature point tracking means for specifying the corresponding feature points between the frames and calculating the physical quantity.

A preferred embodiment of the present invention is characterized in that the physical quantity includes at least one of the coordinate displacement amount, the moving speed, the acceleration, the moving direction, and the direction of the acceleration of at least one of the feature points.

In a preferred embodiment of the invention, the physical quantity is characterized by comprising a time variation of a distance between one reference point contained in the face and each of at least two feature points contained in the face. do.

A preferred embodiment of the invention is characterized in that the physical quantity comprises a time-varying quantity of an area surrounded by at least three feature points.

A preferred embodiment of the present invention is characterized in that the measurement of the physical quantity is performed on an impression determining portion that determines the appearance impression of the face.

A preferred embodiment of the present invention is characterized in that the appearance impression of the face is an age impression.

In a preferred embodiment of the present invention, the face impression analysis device has a correlation storage means for storing the correlation between the physical quantity and the visual evaluation value of the appearance impression of the face obtained by visually evaluating the face. ,
It is characterized by having a visual evaluation value calculating means for calculating a visual evaluation value of a visual impression of a subject from the measured value of the physical quantity of the subject based on the correlation stored in the correlation storage means.

In a preferred embodiment of the present invention, the facial impression analyzer has a correlation storage means for storing the correlation between the physical quantity and the physiological quantity related to the skin.
It is characterized by having a physiological quantity calculating means for calculating the physiological quantity of the skin of the subject from the measured value of the physical quantity of the subject based on the correlation stored in the correlation storage means.

In a preferred embodiment of the invention, the physiological amount is characterized by skin surface and / or internal properties at the impression-determining site.

The face impression analysis program according to the present invention is
It is a facial impression analysis program that analyzes the appearance impression of the subject's face based on the movement of the skin when the facial expression changes.
Computer,
A means to measure the physical quantity of skin changes caused by changes in facial expressions,
A means for discriminating the appearance impression of the subject's face based on the physical quantity,
It is characterized by functioning as.

The method for analyzing the appearance impression of a face according to the present invention is as follows.
It is a method for analyzing the appearance of a face using a face impression analysis device equipped with a moving image acquisition means and a physical quantity measuring means for analyzing the appearance impression of the subject's face based on the movement of the skin when the facial expression changes. hand,
The step in which the moving image acquisition means acquires a moving image including the process of changing the facial expression,
The physical quantity measuring means includes a step of calculating a physical quantity caused by the change in facial expression.

The method for distinguishing the cosmetic effect according to the present invention is
It is a method of distinguishing the cosmetic effect by distinguishing the appearance impression of the subject's face based on the physical quantity of the skin change caused by the change of facial expression.
A pre-makeup physical quantity measurement step for extracting the physical quantity before make-up, which is caused by a change in facial expression before make-up.
A pre-makeup visual evaluation value calculation step for calculating a pre-makeup visual evaluation value based on the correlation between the physical quantity and the visual evaluation value of the appearance impression of the face.
A post-makeup physical quantity measurement step for extracting the post-makeup physical quantity caused by a change in facial expression after makeup,
A post-makeup visual evaluation value calculation step for calculating a post-makeup visual evaluation value based on the correlation, and a post-makeup visual evaluation value calculation step.
It is characterized by comprising a cosmetic effect evaluation step of comparing the visual evaluation value before makeup and the visual evaluation value after makeup to evaluate the makeup effect.

Of the physical quantities generated by changes in facial expressions, the physical quantities (determining factors) that have a high correlation with the facial appearance impression in the part that determines the facial appearance impression are actually used as an index for distinguishing the facial appearance impression. It is possible to easily and accurately distinguish the visual impression of a face that a person receives visually.

It is a flowchart which shows the extraction method of the determination part of the appearance impression of a face which concerns on this invention, and the extraction method of the determinant of the appearance impression of a face. It is a flowchart which shows the measurement process of the physical quantity from a moving image. It is a functional block diagram of the face impression analysis apparatus which concerns on this invention. It is a flowchart which shows the discrimination method of the appearance impression of a face by the face impression analysis apparatus which concerns on this invention. It is a flowchart which shows the calculation process of the visual evaluation value / physiological amount by the face impression analysis apparatus which concerns on this invention. It is a figure which shows the example of the moving image / continuous still image used for the acquisition of the visual evaluation value. (Photo substitute for drawing) It is a figure which shows the presentation method of the moving image / continuous still image to an evaluator at the time of acquisition of a visual evaluation value. (Photo substitute for drawing) It is a figure which shows the aggregation result of the visual evaluation value with respect to the moving image / continuous still image. It is a figure which shows the example of the result of the line-of-sight evaluation for a moving image / continuous still image. (Photo substitute for drawing) It is a figure which shows the facial expression change of the evaluation target in a moving image. (Photo substitute for drawing) It is a figure which shows the time change of the moving speed of a marker in a moving image. It is a figure which shows the integrated value of the moving speed of a marker in a moving image. It is a figure which shows the arrangement of the marker for motion capture. It is a figure which shows the facial expression change of the evaluation target in a moving image. (Photo substitute for drawing) It is a figure which shows the time change of the amount of change in the distance between markers in a moving image per unit time. It is a figure which shows the area setting to analyze the time change of the area. It is a graph which shows the measurement result of the stretch ratio of the skin in the analysis target area. It is a figure which shows the setting of a grid point and an example of a motion vector. (Photo substitute for drawing) It is a figure which shows the facial expression change of the evaluation target in a moving image. (Photo substitute for drawing) It is a figure which shows the measurement result of the time change of the magnitude of a motion vector. It is a figure which shows the average value of the width of the waveform of the motion vector in a moving image.

[1] A method for extracting a determination portion of a facial expression impression of the present invention and a method for extracting a determinant of a facial expression impression in the present invention The present invention presents a moving image including a change in facial expression to an evaluator and presents the appearance of the face. A method of evaluating the impression and extracting the determination part of the facial expression impression by analyzing the line of sight of the evaluator at that time, and measuring the physical quantity from the moving image and correlating it with the facial expression impression. It is a method of extracting the determinants of the appearance impression of the face, which extracts the factors with high facial expressions.

The "appearance impression of a face" as used in the present invention is an impression that is visually received when another person sees the face to be evaluated. For example, if we focus on the impression of age, we can see the impression of youthfulness such as looking younger than the actual age, or the impression of aging such as looking old. In addition, there are various impressions such as the attractiveness felt when looking at the evaluation target.

FIG. 1 is a flowchart showing a method of extracting a determination portion of a facial appearance impression and a method of extracting a determinant of a facial appearance impression of the present invention. With reference to this, the determination site of the appearance impression of the face and the extraction method of the determinant of the present invention will be described in detail.

<Acquisition of moving image>
First, in step S11, a moving image of the face to be evaluated is acquired. Here, as described above, a moving image including a change in facial expression is acquired.

Here, an image obtained by capturing a moving image of the face to be evaluated with a general camera device may be used, but it is preferable that the image has a resolution sufficient to withstand image analysis in a later physical quantity measurement step.

In general, a moving image is composed of a series of a large number of still images (frames), and the smoothness of the movement is represented by a frame rate representing the number of frames per unit time. Here, when presenting a moving image to the evaluator in the later visual evaluation value acquisition process, the frame rate is set to a level higher than the evaluator can perceive as a moving image rather than just a series of still images. It is preferable to acquire a moving image to have.

Further, a stereo moving image may be taken using a stereo camera. With such a configuration, three-dimensional analysis can be performed in the physical quantity measuring step described later.

More specifically, a motion that tracks the three-dimensional position of each marker by attaching a plurality of markers to the face to be evaluated and then taking a motion image of the face to be evaluated with a stereo camera. Capturing can be performed.

<Acquisition of visual evaluation value>
In step S12, the visual evaluation value for the moving image is acquired. This is done by presenting the moving image acquired in step S11 to the evaluator and presenting the impression received by the evaluator from the moving image.

For example, if the purpose is to extract the determinant and the determinant of the age impression among the facial facial expressions, a moving image including the facial expression change of the face to be evaluated is presented to the evaluator for evaluation. There are methods such as requesting an estimate of the target age (requesting an answer for the estimated age). As another example, if the purpose is to extract the determinants of the facial expression impression related to attractiveness, a moving image including the facial expression change of the face to be evaluated is presented to the evaluator as in the case of the age impression. After that, there is a method of asking for an answer as to whether or not the evaluation target is attractive. It can be said that the visual evaluation value obtained in this way is a value that directly indicates the appearance impression of the face to be evaluated.

In addition, one frame before the facial expression change and one frame after the facial expression change are extracted as still images from the moving image, and only those two frames are presented to the evaluator as continuous still images in the same manner. A step may be provided to promote the evaluation of the impression. As a result, the impression of the face of the evaluation target given by the continuous still image as the static stimulus is acquired as a contrast with the impression of the face of the evaluation target given to the evaluator by the moving image as the dynamic stimulus. More detailed analysis can be performed for the extraction of decision sites and impression determinants.

When presenting a moving image to the evaluator, the evaluator's line-of-sight information is acquired, and information such as how many times and how many seconds the evaluator has retained the line of sight in which area of the face to be evaluated is acquired. ..

<Extraction of impression determination site>
In step S13, the line-of-sight information of the evaluator when the moving image is presented to the evaluator is analyzed, and the impression determination portion is extracted.

When an evaluator who imposes an evaluation of a certain impression observes a moving image, the evaluator judges the impression many times and in the area where the line of sight is retained for a long time, regardless of whether it is conscious or unconscious. It can be said that it includes the area to be watched when doing so, that is, the part that determines the impression.

For example, when the evaluator is informed that an answer to the estimated age of the target displayed on the moving image is requested and the moving image is presented, the area where the evaluator's line of sight stays for a long time many times is an age impression. Contains the decision site.

In step S13, as described above, the region where the evaluator's line of sight stays a lot during the visual evaluation and the surrounding area are extracted as the determination site of the appearance impression.

<Measurement of physical quantity>
In the following step S14, the moving image is analyzed and the physical quantity is measured. This may be performed only on the impression determination portion specified in step S13, or may be performed on the entire region included in the moving image. FIG. 2A is a flowchart showing a physical quantity calculation process using a skin motion vector obtained by analyzing a two-dimensional moving image by an optical flow method.

First, in step S21, feature points are extracted from each frame in the moving image. Here, a method of discriminating the unevenness of the skin from the brightness value of each pixel in the frame and setting the feature points, or a method of setting the feature points in a grid pattern in a moving image, that is, the feature points in a grid pattern. Any method such as a method of arranging may be used.

Then, in step S22, the motion vector of each feature point is specified. The motion vector can be obtained by the change in brightness between frames and the like.

After that, in step S23, the motion vector is analyzed, and the physical quantity generated by the change in the facial expression is calculated. Examples of physical quantities here are related to the displacement of the coordinates of the feature points mentioned above, the moving speed, the acceleration, the moving direction, and the wave motion parameters of the plane movement, such as the period and the circumference, and the movement of the skin of the face. The amount can be mentioned. Further, it may be configured to calculate optical quantities such as brightness and color tone, the amount of change thereof, frequency characteristics, etc., and to perform analysis by a particle image velocimetry method (PIV, Personal Image Velocity) or other methods. ..

Each such physical quantity may be obtained as a value in each frame, or may be obtained as an average value over the entire moving image. Alternatively, a region such as a cheek or a chin may be set, and an integrated value or an average value of values in the region may be obtained.

As another method of measuring the physical quantity, the calculation process of the physical quantity by tracking the coordinates of the feature points is shown in the flowchart of FIG. 2 (b).

First, in step S31, the feature points are calculated by an arbitrary method as in step S21. Alternatively, when shooting a moving image in step S11, markers or the like may be arranged in advance on the face to be shot and used as feature points.

Then, in step S32, the coordinates of each feature point in each frame in the moving image are calculated.

Then, in step S33, the change in the coordinates of the feature points between each frame is tracked, and the physical quantity as described in the explanation of step S23 is calculated. For example, it is the time change of the velocity and acceleration of each feature point. Further, a feature point as a reference point may be set, and the time change of the distance between the feature point and any other feature point may be obtained. That is, it is a value corresponding to the relative velocity of any feature point on the face. Yet another physical quantity is the time variation of the area surrounded by three or more feature points. Thereby, the mode of expansion and contraction of the skin at any part on the face can be quantified.

Further, as described above, when the stereo moving image is acquired in step S11, it is possible to perform a three-dimensional analysis in which not only the plane but also the depth is taken into consideration. That is, in step S32, the coordinate of the feature point in the three-dimensional space can be calculated by an arbitrary method such as the stereo method, and in step S33, the coordinate change of the feature point in the three-dimensional space can be tracked. This makes it possible to more accurately analyze the three-dimensional movement of the skin due to changes in facial expressions.

The physical quantity measurement using the motion vector described above and the physical quantity measurement by tracking the coordinates of the feature points may be configured to perform either one or both of them. Further, a moving image analysis by a method different from these may be performed to measure the physical quantity.

<Measurement of physiological amount>
In step S15, the physiological amount of the skin of the face to be photographed for the moving image is measured.
Examples of the physiological amount related to the skin include skin surface physical characteristics and / or internal physical characteristics, skin tissue composition, and the like, and those known to have a correlation with the appearance impression of the face can be used. For example, if the visual impression of the face is an age impression, physiological amounts for the skin for which the mechanism involved in aging has been clarified can be used. Such physiological quantities include skin surface characteristics such as skin surface roughness, stratum corneum skin properties such as stratum corneum water content, dermal properties such as collagen fiber bundle amount and structure, and fat layer physical properties such as fat. Physical characteristics of the papillary layer or papillary layer such as the position and size of the body, for example, the number and height of the papillary structure per unit area and the like can be mentioned. The physiological amount is preferably the physiological amount of the impression-determining site.
The measurement of the physiological amount can be performed by a conventionally known method. The present invention is limited to non-invasive methods (methods that do not involve surgery).

Each step of steps S11 to S15 is performed on a plurality of different faces, which is sufficient to perform the following correlation analysis.

<Extraction of appearance determinants>
In step S16, a determinant of the appearance impression of the face is extracted from the physical quantity acquired from the moving image in step S14.

Here, the correlation between the physical quantity acquired in step S14 and the visual evaluation value acquired in step S12 or the physiological quantity measured in step S15 is analyzed, and the visual evaluation value or the physiological quantity is analyzed from the physical quantities. Extract the factors that determine.

For example, a plurality of moving images of evaluation targets having different visual evaluation values are compared, and among the physical quantities measured from those moving images, those having a difference between the two have a correlation with the visual evaluation value. Expected to have. Then, by making the same comparison for another moving image, the physical quantity that has a correlation with the visual evaluation value and is a determinant of the appearance impression, and the correlation between the physical quantity and the visual evaluation value are clarified. Can be.
Correlation analysis can be performed using conventionally known methods.

Further, as described above, when the visual evaluation value is acquired in step S12, when the visual evaluation value when two still images before and after the facial expression change are presented, the corresponding moving image is obtained. May be compared with the visual evaluation value. For example, when the visual evaluation value for a still image of the same evaluation target and the visual evaluation value for a moving image are significantly different, the physical quantity contained in the moving image is a determinant of the visual impression of the face. Can be estimated.

Similarly, by analyzing the correlation between the physical quantity obtained from the moving image and the physiological quantity related to the skin, it is possible to determine the physical quantity having a high correlation with the physiological quantity related to the skin, and the physical quantity is the appearance of the face. It can be presumed to be a determinant of the impression of.

As described above, according to the present invention, a moving image including a change in facial expression is presented to the evaluator, and by analyzing the line of sight of the evaluator, a portion for determining the appearance impression of the face is determined, and the moving image. By extracting more physical quantities and analyzing them, it is possible to extract the determinants of the appearance impression of the face.

It should be noted that the acquisition of the visual evaluation value and the analysis of the correlation with the physical quantity, the measurement of the physiological quantity and the analysis of the correlation with the physical quantity may be performed both, or only one of them may be performed as necessary. You may go. Further, different evaluation values and quantities may be used to analyze the correlation between the evaluation values and the physical quantities measured from the moving images.

[2] Method for Discriminating Appearance Impression of Face of the Present Invention and Face Impression Analytical Device The present invention acquires a moving image of an impression-determining part of a subject's face, including a change in facial expression, and measures the physical quantity by analyzing the moving image. Then, it is a method of discriminating the appearance impression of the face of the subject, and an appearance analysis device of the face.

As shown in the functional block diagram in FIG. 3, the face impression analysis device according to the present invention includes a moving image acquisition means 1, a physical quantity measuring means 2, a correlation storage means 3, a visual evaluation value calculating means 4, and physiology. A physical quantity calculation means 5 is provided.

Further, the moving image acquiring means 1 has a stereo moving image acquiring means 11, and the physical quantity measuring means 2 includes a feature point extracting means 21, a motion vector specifying means 22, a motion vector analysis means 23, and feature point coordinate calculation. It has means 24 and feature point tracking means 25.

In the process of extracting the determinant of the appearance impression of the face, the correlation storage means 3 analyzes the formula showing the correlation obtained by analyzing the physical quantity and the visual evaluation value, and the physical quantity and the physiological quantity. Information such as an expression showing the obtained correlation is retained.

The face impression analysis device according to the present invention uses a general-purpose computer device including an arithmetic unit, a main storage device, an auxiliary storage device, and various input / output devices, and the auxiliary storage device thereof is used. This is realized by storing a face impression analysis program for making the computer device function as each of the above-mentioned means. Then, when the computer device is made to function as the face impression analysis device, the face impression analysis program is developed on the main storage device as needed, and the processing by the arithmetic unit is performed.

Further, any configuration such as a configuration in which a camera device is used as the moving image acquisition means 1 and the moving image obtained thereby is passed to the physical quantity measuring means 2, or a configuration in which the correlation storage means 3 is realized by a general-purpose server device, etc. A plurality of devices may be configured to enable communication via a USB (Universal Real Bus) connection, data transfer using a recording medium, and a network such as the Internet or a LAN (Local Area Network).

<Acquisition of moving image>
FIG. 4 is a flowchart showing a method of discriminating the appearance impression of the subject's face by the face impression analysis device. First, in step S41, the moving image acquisition means 1 is used to acquire a moving image including a change in the facial expression of the subject's face. The stereo moving image acquisition means 11 may acquire a stereo moving image including a change in the facial expression of the subject's face. Further, at this time, a moving image in which a marker used as a feature point in the subsequent measurement of the physical quantity is previously placed on the face of the subject may be acquired.

<Measurement of physical quantity>
Then, in step S42, the physical quantity is measured by using the physical quantity measuring means 2.
Here, the physical quantity extracted as the determinant of the facial appearance impression is calculated by the above-mentioned method for extracting the facial appearance impression determinant by the same processing as that described with reference to FIG. conduct.

When the physical quantity is calculated using the motion vector shown in FIG. 2A, the feature point extraction process in step S21 is performed by the feature point extraction means 21, and the motion vector identification process in step S22 is performed by the motion vector identification means. After performing the motion vector analysis process in step S23, the motion vector analysis means 23 performs the motion vector analysis process in step S23.

When the physical quantity is calculated using the feature point coordinates shown in FIG. 2B, the feature point extraction process in step S31 is performed by the feature point extraction means 21, and the feature point coordinates calculation process in step S32 is performed. After the coordinate calculation means 24, the feature point tracking means 25 performs the tracking process of the feature point coordinates in step S33.

Further, when the stereo moving image was acquired in step S41, the feature point coordinates in the three-dimensional space are calculated by an arbitrary method such as the stereo method in step S32, and the feature points in the three-dimensional space are calculated in step S33. Coordinate changes may be tracked.

When the physical quantity is calculated using both the physical quantity measurement using the motion vector and the physical quantity measurement using the feature point coordinates, which of the feature point coordinate calculation processes is performed in step S21 and step S31? A configuration may be performed in which only one is performed to share the feature points, or a configuration in which steps S21 and S31 are executed together to extract different feature points in order to effectively measure the physical quantity in each method. good.

<Analysis of facial appearance impression>
In step S43, the physical quantity measured in step S42 is analyzed to distinguish the appearance impression of the face. Thereby, at least one of the visual evaluation value of the visual impression of the subject's face and the physiological amount of the subject's facial skin is calculated.

FIG. 5A is a flowchart showing the calculation process of the visual evaluation value by the visual evaluation value calculation means 4. First, in step S51, the correlation data between the physical quantity and the visual evaluation value is acquired from the correlation storage means 3.

Then, in step S52, the visual evaluation value is calculated based on the physical quantity acquired in step S42 and the correlation data acquired in step S51. From the calculated visual evaluation value, it is possible to discriminate the appearance impression of the subject's face.

FIG. 5B is a flowchart showing a physiological quantity calculation process by the physiological quantity calculation means 5. This is also the same process as the above-mentioned visual evaluation value calculation process. First, in step S61, the correlation data between the physical quantity and the physiological quantity is acquired from the correlation storage means 3.

Then, in step S62, the physiological quantity is calculated based on the physical quantity acquired in step S42 and the correlation data acquired in step S61. From the calculated physiological amount, it is possible to discriminate the appearance impression of the subject's face.

As described above, the physical quantity is measured by analyzing the moving image including the facial expression change of the subject, and the subject is based on the correlation between the physical quantity and the visual evaluation value or the physiological quantity from the physical quantity of the subject. The visual evaluation value and / or the physiological quantity of the subject can be calculated, and the appearance impression of the subject's face can be discriminated from the calculated value.

If the stereo moving image is acquired by using the stereo moving image acquiring means 11, the physical quantity measuring means 2 can perform a three-dimensional analysis. However, the stereo moving image acquiring means 11 is a general monocular camera. It is expensive compared to such. Therefore, the stereo moving image acquisition means 11 may be omitted, a general monocular camera or the like may be provided as the moving image acquisition means 1, and only the two-dimensional analysis may be performed by the physical quantity measuring means 2. With such a configuration, the face impression analysis device can be realized at low cost.

Further, as a method for discriminating the appearance impression of the face and a method for using the face impression analysis device according to the present invention, there is a method for discriminating the cosmetic effect. That is, by comparing the analysis result of the appearance impression performed using the moving image of the face before makeup of the subject and the analysis of the appearance impression performed using the moving image of the face after makeup of the subject. It distinguishes the effect of makeup on the appearance impression.

Hereinafter, the present invention will be described in more detail by giving examples of acquisition of a visual evaluation value of an age impression of the appearance of a face, extraction of an impression determining site, and measurement of a physical quantity. It goes without saying that it is not limited to just an example.

[1] Acquisition of visual evaluation values using moving images and continuous still images <Acquisition of moving images>
Nine seconds of moving images including changes in facial expressions were taken for 20 evaluation subjects (ages in their 20s and 60s). Then, one frame before the facial expression change and one frame after the facial expression change were both extracted as still images from the moving image, and a continuous still image was created in which only those two frames were displayed in 9 seconds. FIG. 6A shows a schematic diagram of a moving image, and FIG. 6B shows a schematic diagram of a continuous still image.

<Acquisition of visual evaluation value>
The above moving image or continuous still image was randomly displayed on 15 evaluators using a monitor, and the visual evaluation value of the age impression of the appearance of the face was acquired. When presenting a moving image and a continuous still image to an evaluator, a line-of-sight analysis device is used to analyze how many times and how many seconds the evaluator's line of sight stays in which area of the moving image or still image. rice field. FIG. 7 shows a schematic diagram of a method of presenting a moving image and a still image to an evaluator.

First, as shown in FIG. 7A, the numbers are now displayed, then the face to be evaluated is displayed, and instructions are presented asking for an answer as to whether the numbers are in the first half or the second half of the generation. As a result, when viewing the moving image or the continuous still image displayed after that, the evaluator becomes aware of the determination of the age impression of the evaluation target.

Then, as shown in FIG. 7 (b), the numbers were displayed. This number is subsequently rounded down to the first place of the actual age of the evaluation target in the moving image or the continuous still image. Since the target age in this embodiment is in the 20s to 60s as described above, any number of 20, 30, 40, 50, and 60 is displayed.

Then, as shown in FIG. 7 (c), a moving image or a continuous still image for 9 seconds is displayed, and then, as shown in FIG. 7 (d), the evaluation target displayed in FIG. 7 (c) is shown in FIG. We asked for answers on whether the numbers displayed in 7 (b) were the first half or the second half of the generation.

That is, in the example shown in FIG. 7, the age of the evaluation target was either between 30 and 39 years old, and the evaluator replied whether he was in his early 30s or late 30s.

In the response of the evaluator obtained by the above procedure, if the evaluator evaluates that it is in the latter half of the generation, that is, it can be judged that the evaluator feels an aging impression.

FIG. 8 is a compilation of the visual evaluation values obtained from the evaluators for all evaluation targets. As shown here, it can be read that the moving image gave the evaluator a higher age impression than the continuous still image. From this, it can be seen that the movement during the facial expression change included in the moving image contains a factor that gives a high age impression to the evaluator.

<Extraction of impression determination site>
FIG. 9 shows the results of line-of-sight analysis of the evaluator when displaying a moving image and a continuous still image of a certain evaluation target. FIG. 9 (a) shows a point where the evaluator's line of sight was often retained when the moving image of the evaluation target was displayed, and FIG. 9 (b) was shown when the continuous still image of the evaluation target was displayed. Is.

Looking at these line-of-sight analysis results, it can be seen that the evaluator focused on the cheeks and the areas around the eyes to be evaluated regardless of whether the moving image or the continuous still image was displayed. From this, it was found that the areas where the appearance of the face was determined by the age impression were the cheeks and the periphery of the eyes.

In addition, although the determination site of the age impression is the same for both the moving image and the continuous still image, the age impression given by the moving image is higher, so that the determinant of the age impression is the determination of the age impression. It was found that it is in the physical quantity generated by the movement when the facial expression changes in the part.

[2] Measurement of physical quantities by motion capture <Acquisition of moving images>
For 30 women in their 20s to 60s, a reflective marker was attached to their faces as markers to indicate measurement points, and multiple cameras were used to shoot moving images including facial expression changes for each evaluation target. went. Then, by analyzing this moving image, motion capture was performed to track the change in the three-dimensional coordinates of the marker. In addition, a moving image was presented to the evaluator, and the visual evaluation value of the age of each evaluation target was acquired.

Here, the facial expression is changed from a straight face to a wide open mouth as shown in FIG. 10 (a), and after taking a closed facial expression as shown in FIG. 10 (b), the face is returned to the true face again. A moving image containing a series of facial expression changes was used for motion capture and acquisition of visual evaluation values.

Then, as shown below, the physical quantity is extracted by analyzing the movement position, movement amount, movement speed of the markers, the relationship between the movements of the two markers, and the time change of the area surrounded by three or more markers. bottom.

<Measurement of marker movement speed>
The movement of the marker on the face of each evaluation target was tracked, and the movement speed was measured. As a result, the results shown in FIG. 11 were obtained.

FIG. 11A shows an evaluation target whose age visual evaluation value is higher than the actual age, that is, an evaluation target having a high visual age impression, and FIG. 11B shows an evaluation target whose age visual evaluation value is higher than the actual age. It is a graph showing the time change of the movement speed of the marker in each of the evaluation targets having a low age impression, that is, the evaluation targets having a low visual age impression. In this case, the vertical axis is the moving speed of the marker, and the horizontal axis is the number of frames in the moving image as the time element. Further, the speed direction of the marker when the facial expression changes from the state of a straight face to the facial expression with the mouth wide open as shown in FIG. 10 (a) is the speed in the extension direction, and the mouth as shown in FIG. 10 (b). The moving speed of the marker when the facial expression changes to the closed facial expression is shown as the speed in the contraction direction.

As shown here, the graph of the velocity change of the feature points of the evaluation target that looks old has a sharp peak and the velocity converges faster than that of the evaluation target that looks young. That is, it can be said that the change in the moving speed of the feature points of the evaluation target that looks old is more severe than that of the evaluation target that looks young.

12 shows the speed of movement of the feature points of the evaluation target that looks old and the speed of movement of the feature points of the evaluation target that looks young as shown in FIG. 11 (b), respectively. It is a graph which calculated the area of each graph by integrating.

In FIG. 12, since the graph area of the evaluation target that looks younger is larger, the change in the moving speed of the feature points of the evaluation target that looks younger is more gradual than that of the evaluation target that looks old. Can be read.

From this result, it was found that the characteristic points, that is, the supple movements in which the movements of the skin gently follow the changes in facial expressions, may give the evaluator the impression of being young.

<Relationship between the movements of the two markers>
FIG. 13 is a diagram showing a marker set on the left side surface of the face to be evaluated. As shown here, a marker M1 at two points on the cheek, a marker M2, and a marker M3 located on the chin were set with the marker at the upper part of the forehead as the reference point M0.

Then, the distance L1 between the reference point M0 and the marker M1 when changing from the state of a straight face as shown in FIG. 14 (a) to the expression with a wide open mouth as shown in FIG. 14 (b). Change amount V1 per unit time and the change amount V2 per unit time of the distance L2 between the reference point M0 and the marker M2, and the change L3 of the distance L3 between the reference point M0 and the marker M3 per unit time. The time change of the three changes of the quantity V3 was measured. That is, it can be said that this is a time change of the relative velocities of the markers M1, M2, and M3 with respect to the reference point M0.

As a result of the above measurement, the result as shown in the graph of FIG. 15 was obtained. FIG. 15A is a graph showing the measurement results of the evaluation target evaluated to have a young appearance impression by visual evaluation. Looking at this, it can be read that the absolute values of the maximum values of V1 to V3 are different, but the times when the maximum values are taken by each are very close. On the other hand, looking at the measurement results of the evaluation target, which was evaluated to have an aged appearance impression by visual evaluation, as shown in FIG. The result was that

That is, it can be said that the movement of the cheeks represented by the markers M1 and the marker M2 is delayed from the movement of the jaw indicated by the markers M3 in the facial expression change of the evaluation target having an aged appearance impression.

From this result, it was found that there is a possibility of giving the evaluator an impression of being young when the followability of the cheek movement when the facial expression changes is good.

<Time change of the area surrounded by 3 or more markers>
As shown in FIG. 16, two regions were set: a jaw A1 surrounded by three markers and a cheek A2 surrounded by four markers. Then, when the time change of the area of each region at the time of facial expression change is measured and the result of time integration of those values is calculated as the stretch ratio of the skin in each region, the result as shown in the graph of FIG. 17 is obtained. Obtained.

FIG. 17A is a graph showing the measurement results of the evaluation target evaluated to have a young appearance impression by visual evaluation. Further, FIG. 17B is a graph showing the measurement results of the evaluation target evaluated to have an aged appearance impression by visual evaluation. Comparing these, the evaluation target having a young appearance impression has a higher expansion / contraction rate of the cheek A2 than the jaw A1, while the evaluation target having an old appearance impression has a cheek A2 rather than the jaw A1. The result was that the elasticity of the cheeks was low.

From this result, it was found that when the skin of the cheek part A2 expands and contracts more flexibly than the chin part A1, it may give the evaluator the impression of looking younger.

[3] Measurement of physical quantities by optical flow method <Acquisition of moving images>
Two-dimensional moving images including changes in facial expressions were taken for each of the 80 females in their 20s to 60s to be evaluated. Then, grid-shaped grid points were set in the moving image, and the movement vector of the skin at each grid point of each frame was obtained from the shading change between each frame in the moving image, as shown in FIG. In addition, the moving images were presented to the evaluators, and the visual evaluation values of the ages of each evaluation target were acquired.

For the analysis of the moving image and the acquisition of the visual evaluation value, the facial expression with the mouth opened in FIG. 19 (a) is taken from the straight face, and the straight face in FIG. 19 (b) is passed through, as shown in FIG. 19 (c). We used a moving image that changed to a nice facial expression and returned to a true face.

<Changes in motion vector>
As a result of analyzing the magnitude of the motion vector during the facial expression change, that is, the speed of the skin movement for the cheek region, a graph as shown in FIG. 20 was obtained. This shows the time change of the speed of skin movement, with the vertical axis representing the average value of the magnitudes of the motion vectors in each frame and the horizontal axis representing the number of frames, that is, the time axis. Looking at this, it is evaluated that it has an old appearance impression by the visual evaluation shown by the broken line in the graph, compared with the measurement result of the evaluation target evaluated as having a young appearance impression by the visual evaluation shown by the solid line in the graph. The measurement result of the evaluation target was formed into a shape having a sharp peak, particularly at the time of change to the facial expression as shown in FIG. 19 (c).

FIG. 21 is a graph showing the results of calculating the convergence time width between the speed of movement of the skin of the evaluation target that looks old and the speed of movement of the skin of the evaluation target that looks young, as shown in FIG. Looking at this, the evaluation target that looks younger is similar to the skin physical properties shown by the measurement result of the movement speed of the marker when the facial expression changes by the motion capture video explained earlier with reference to FIG. It can be seen that the graph value is large and the change in movement is gradual.

In other words, instead of the analysis by 3D video motion capture, which requires physically installing a marker on the face to be evaluated and shooting with multiple cameras, 2D that can be easily shot. It was suggested that the same analysis result may be obtained by the analysis of the moving image by the optical flow method.

INDUSTRIAL APPLICABILITY The present invention can be used to easily and accurately discriminate an age impression of a customer's appearance in cosmetics sales, counseling, aesthetics, and the like.

1 Motion image acquisition means 11 Stereo video acquisition means 2 Physical quantity measurement means 21 Feature point extraction means 22 Motion vector identification means 23 Motion vector analysis means 24 Feature point coordinate calculation means 25 Feature point tracking means 3 Correlation storage means 4 Visual evaluation value Calculation means 5 Physiological quantity calculation means

Claims (21)

A method for distinguishing the appearance impression of a face, which comprises distinguishing the appearance impression of a subject's face based on the physical quantity of the skin change caused by the change of the facial expression at the impression determination site for determining the appearance impression of the face. The first aspect of the invention, wherein the physical quantity includes at least one of the magnitude of the moving speed, the direction of the velocity, the magnitude of the acceleration, and the direction of the acceleration included in the face. How to distinguish the appearance impression of the face. 1. Item 2. The method for distinguishing the appearance impression of a face according to item 2. The discrimination of the appearance impression of the face according to any one of claims 1 to 3, wherein the physical quantity includes an amount related to a time change of an area surrounded by at least three feature points contained in the face. Method. The discrimination of the appearance impression of the face calculates the visual evaluation value of the appearance impression of the subject's face from the measured value of the physical quantity for the subject based on the correlation between the physical quantity and the visual evaluation value of the appearance impression of the face. The method for discriminating the appearance impression of a face according to any one of claims 1 to 4, wherein the method comprises the same. The method for discriminating the appearance impression of a face according to any one of claims 1 to 5, wherein the appearance impression of the face is an age impression. It is a facial impression analysis device that analyzes the appearance impression of the subject's face based on the movement of the skin when the facial expression changes.
A moving image acquisition means for acquiring a moving image including the process of changing facial expressions,
A face impression analysis device comprising: a physical quantity measuring means for measuring a physical quantity of a skin change caused by the facial expression change from the moving image. The physical quantity measuring means includes a feature point extracting means for extracting a plurality of feature points from the moving image, and a feature point extracting means.
A motion vector specifying means for specifying a motion vector of the feature point from the moving image,
The face impression analysis apparatus according to claim 7, further comprising a motion vector analysis means for analyzing the motion vector and calculating the physical quantity. The face impression analysis apparatus according to claim 8, wherein the physical quantity includes a quantity related to a change in the magnitude and / or direction of the motion vector. The physical quantity measuring means includes a feature point extracting means for extracting a plurality of feature points from the moving image, and a feature point extracting means.
A feature point coordinate calculation means for calculating the coordinates of the feature points in each frame of the moving image, and
The face impression analysis apparatus according to claim 7, further comprising a feature point tracking means for identifying the corresponding feature points between the frames and calculating the physical quantity. The facial impression according to claim 10, wherein the physical quantity includes at least one of the coordinate displacement amount, the moving speed, the acceleration, the moving direction, and the direction of the acceleration of the at least one feature point. Analytical device. 10. Claim 10 or claim, wherein the physical quantity includes a time variation of a distance between one reference point contained in the face and each of at least two feature points contained in the face. Item 11. The face impression analysis device according to item 11. The face impression analysis apparatus according to any one of claims 10 to 12, wherein the physical quantity includes a quantity related to a time change of an area surrounded by at least three feature points. The face impression analysis apparatus according to any one of claims 7 to 13, wherein the measurement of the physical quantity is performed on an impression determining portion that determines the appearance impression of the face. The face impression analysis device according to any one of claims 7 to 13, wherein the appearance impression of the face is an age impression. It has a correlation storage means for storing the correlation between the physical quantity and the visual evaluation value of the appearance impression of the face obtained by visually evaluating the face.
A claim comprising a visual evaluation value calculating means for calculating a visual evaluation value of a visual impression of a subject from the measured value of the physical quantity of the subject based on the correlation stored in the correlation storage means. The facial impression analysis apparatus according to any one of claims 7 to 15. It has a correlation storage means for storing the correlation between the physical quantity and the physiological quantity related to the skin.
The claim is characterized by having a physiological quantity calculating means for calculating the physiological quantity of the skin of the subject from the measured value of the physical quantity of the subject based on the correlation stored in the correlation storage means. 7. The facial impression analyzer according to any one of claims 16. The facial impression analyzer according to claim 17, wherein the physiological amount is the skin surface physical characteristics and / or the internal physical characteristics of the face. It is a facial impression analysis program that analyzes the appearance impression of the subject's face based on the movement of the skin when the facial expression changes.
Computer,
A means to measure the physical quantity of skin changes caused by changes in facial expressions,
A means for discriminating the appearance impression of the subject's face based on the physical quantity,
A facial impression analysis program characterized by functioning as. It is a method for analyzing the appearance of a face using a face impression analysis device equipped with a moving image acquisition means and a physical quantity measuring means for analyzing the appearance impression of the subject's face based on the movement of the skin when the facial expression changes. hand,
The step in which the moving image acquisition means acquires a moving image including the process of changing the facial expression,
A method for analyzing the appearance impression of a face, wherein the physical quantity measuring means includes a step of calculating a physical quantity of a skin change caused by the facial expression change. It is a method of distinguishing the cosmetic effect by distinguishing the appearance impression of the subject's face based on the physical quantity of the skin change caused by the change of facial expression.
A pre-makeup physical quantity measurement step for extracting the physical quantity before make-up, which is caused by a change in facial expression before make-up.
A pre-makeup visual evaluation value calculation step for calculating a pre-makeup visual evaluation value based on the correlation between the physical quantity and the visual evaluation value of the appearance impression of the face.
A post-makeup physical quantity measurement step for extracting the post-makeup physical quantity caused by a change in facial expression after makeup,
A post-makeup visual evaluation value calculation step for calculating a post-makeup visual evaluation value based on the correlation, and a post-makeup visual evaluation value calculation step.
A method for discriminating cosmetic effects, which comprises a cosmetic effect evaluation step of comparing the visual evaluation value before makeup and the visual evaluation value after makeup to evaluate the makeup effect.

Images