JP7177224B2 - Facial visual impression discrimination method, facial impression analysis device, facial impression analysis program, facial visual impression analysis method, and makeup effect discrimination method - Google Patents

Description

The present invention provides a method for extracting a determining part of a facial visual impression, a method for extracting a determinant of a facial visual impression, a method for discriminating a facial visual impression, a facial impression analysis device, a facial impression analysis program, and a facial visual impression analysis. It is about the method.

People give a wide variety of impressions to others depending on various factors such as their appearance and behavior. Among them, it can be said that visual information, particularly the appearance of a face, is one of the very important factors that determine the impression of a person.

Therefore, research is being conducted on techniques for objectively estimating the visual impression of a person's face. For example, in Patent Document 1, a plurality of tracking points are arranged in advance in an analysis area such as the outer corner of the eye of a subject, and a moving image is captured. A technique for analyzing the skin condition of a subject by obtaining the is disclosed.

JP 2014-193197 A

As mentioned above, parameters for objectively estimating the visual impression of a subject's face have been studied, but this is nothing more than acquiring specific parameters from specific parts of the subject's face.

However, a person's visual impression is formed by various factors, and a more accurate estimation means has been desired.

Therefore, the present invention provides a method for extracting a site that greatly affects the visual impression of the face, a factor that is highly correlated with the visual impression of the face, and a method for discriminating the visual impression of the face based on the factor. is the subject.

In order to solve the above problems, a method for extracting a determining part of a facial visual impression according to the present invention includes:
a visual evaluation step of presenting dynamic information of changes in facial expression to an evaluator and having the evaluator evaluate the visual impression of the face;
and a line-of-sight analysis step of analyzing line-of-sight data of the evaluator viewing the dynamic information, and analyzing a region of the dynamic information gazed at by the evaluator.

This makes it possible to extract the part that a person pays attention to when evaluating the visual impression of the face based on the dynamic information, that is, the part that determines the visual impression of the face.

A preferred embodiment of the present invention is characterized in that the visual impression of the face is an impression of age.

The method for extracting determinants of visual impression of a face according to the present invention includes:
a physical quantity measuring step of measuring a physical quantity of a change in skin caused by a change in facial expression at an impression-determining site that determines the visual impression of the face;
and a factor extraction step of extracting a factor highly correlated with the visual impression of the face from the physical quantity.
As a result, it is possible to specify the physical quantity used in discriminating the visual impression of the face, which will be described later.

In a preferred embodiment of the present invention, the physical quantity measuring step includes a step of measuring quantities related to the markers as the physical quantities from a motion-capture moving image shot with a plurality of markers set on the face. .

In a preferred embodiment of the present invention, the quantity relating to the marker includes at least one of the position of the marker, the amount of movement, the magnitude of velocity, the direction of velocity, the magnitude of acceleration, and the direction of acceleration. do.

A preferred embodiment of the present invention is characterized in that the amount relating to the markers includes an amount relating to temporal changes in the distances between the plurality of markers.

A preferred embodiment of the present invention is characterized in that the amount relating to the markers includes an amount relating to time change of values indicating the area and/or shape of the surface formed by the three or more markers.

In a preferred embodiment of the present invention, the physical quantity measurement step includes a grid point setting step of setting a plurality of grid points on the face,
The physical quantity includes a quantity relating to the lattice points.

A preferred embodiment of the present invention is characterized in that the quantities relating to the grid points include quantities relating to temporal changes in the magnitude and/or direction of the velocity vector for each of the grid points.

In a preferred embodiment of the present invention, the method for extracting determinants of visual impression of a face comprises visual evaluation values obtained by presenting dynamic information including changes in facial expression to an evaluator to obtain visual evaluation values of visual impression of the face. having an acquisition step;
The factor extracting step includes extracting the factor by analyzing the correlation between the physical quantity and the visual evaluation value.
By using the visual evaluation value, it is possible to extract factors that are highly correlated with the impression that a person receives when actually looking at the face.

In a preferred form of the present invention, a method for extracting determinants of a facial visual impression comprises a physiological quantity measurement step of measuring a physiological quantity relating to the skin,
The factor extraction step includes extracting the factor by analyzing the correlation between the physical quantity and the physiological quantity.
By using the physiological quantity, it is possible to extract determinants of facial visual impressions that are highly correlated with physiological phenomena in body tissues such as skin.

A preferred embodiment of the present invention is characterized in that the physiological quantity is skin surface physical properties and/or internal physical properties at the impression-determining site.

A preferred embodiment of the present invention is characterized in that the visual impression of the face is an impression of age.

A preferred embodiment of the present invention is characterized in that the physical quantity is a quantity relating to skin movement at the impression-determining site.

The method for distinguishing the visual impression of a face according to the present invention includes:
The visual impression of the subject's face is discriminated based on the physical quantity of the skin change caused by the change in facial expression at the impression determining site that determines the visual impression of the face.
By using the physical quantity generated in the impression-determining part, it is possible to more accurately discriminate the visual impression of the face given to others.

In a preferred embodiment of the present invention, the physical quantity includes at least one of a magnitude of movement speed, a direction of speed, a magnitude of acceleration, and a direction of acceleration of feature points included in the face. do.

In a preferred embodiment of the present invention, the physical quantity includes a quantity relating to temporal change in distance between one reference point included in the face and each of at least two feature points included in the face. do.

A preferred embodiment of the present invention is characterized in that the physical quantity includes a quantity related to time change of an area surrounded by at least three feature points included in the face.

In a preferred embodiment of the present invention, the visual impression of the face is discriminated based on the correlation between the physical quantity and the visual evaluation value of the visual impression of the face, and the visual impression of the face of the subject is determined from the measured value of the physical quantity of the subject. It is characterized by including calculating a visual evaluation value of the impression.

In a preferred embodiment of the present invention, the discrimination of the visual impression of the face is based on the correlation between the physical quantity and the physiological quantity relating to the skin, and the physiological quantity relating to the skin of the subject is calculated from the measured value of the physical quantity relating to the subject. It is characterized by including calculating.

A preferred embodiment of the present invention is characterized in that the physiological quantity is skin surface physical properties and/or internal physical properties at the impression-determining site.

A preferred embodiment of the present invention is characterized in that the visual impression of the face is an impression of age.

A preferred embodiment of the present invention is characterized in that the physical quantity is a quantity relating to skin movement at the impression-determining site.

The facial impression analysis device according to the present invention includes:
A facial impression analysis device that analyzes the visual impression of a subject's face based on the movement of the skin when the facial expression changes,
a moving image acquiring means for acquiring a moving image including the facial expression change process;
and physical quantity measuring means for measuring a physical quantity caused by the facial expression change from the moving image.
With the device having such a configuration, it is possible to carry out the method for discriminating a visual impression of a face according to the present invention.

In a preferred embodiment of the present invention, the physical quantity measuring means includes feature point extraction means for extracting a plurality of feature points from the moving image;
motion vector identification means for identifying a motion vector of the feature point from the moving image;
and motion vector analysis means for analyzing the motion vector and calculating the physical quantity.

A preferred embodiment of the present invention is characterized in that the physical quantity includes a quantity relating to temporal change in magnitude and/or direction of the motion vector.

In a preferred embodiment of the present invention, the physical quantity measuring means includes feature point extraction means for extracting a plurality of feature points from the moving image;
feature point coordinate calculation means for calculating the coordinates of the feature points in each frame of the moving image;
and feature point tracking means for identifying the corresponding feature points between the frames and calculating the physical quantity.

In a preferred embodiment of the present invention, the physical quantity includes at least one of coordinate displacement amount, moving speed, acceleration, moving direction, and acceleration direction of at least one feature point.

In a preferred embodiment of the present invention, the physical quantity includes a quantity relating to temporal change in distance between one reference point included in the face and each of at least two feature points included in the face. do.

A preferred embodiment of the present invention is characterized in that the physical quantity includes a quantity relating to time change of the area surrounded by the at least three feature points.

A preferred embodiment of the present invention is characterized in that the physical quantity is measured with respect to an impression-determining part that determines the visual impression of the face.

A preferred embodiment of the present invention is characterized in that the visual impression of the face is an impression of age.

In a preferred embodiment of the present invention, the facial impression analysis device has correlation storage means for storing a correlation between the physical quantity and a visual evaluation value of the visual impression of the face obtained by visual evaluation of the face. ,
The present invention further comprises visual evaluation value calculation means for calculating a visual evaluation value of the subject's visual impression from the measured values of the physical quantities of the subject based on the correlation stored in the correlation storage means.

In a preferred embodiment of the present invention, the facial impression analysis device has correlation storage means for storing a correlation between the physical quantity and a physiological quantity relating to skin,
The apparatus is characterized by comprising physiological quantity calculation means for calculating a physiological quantity of the subject's skin from the measured values of the physical quantity of the subject based on the correlation stored in the correlation storage means.

A preferred embodiment of the present invention is characterized in that the physiological quantity is skin surface physical properties and/or internal physical properties at the impression-determining site.

A facial impression analysis program according to the present invention includes:
A facial impression analysis program that analyzes the visual impression of a subject's face based on the movement of the skin when the facial expression changes,
the computer,
means for measuring physical quantities of skin changes caused by facial expression changes;
means for discriminating the visual impression of the subject's face based on the physical quantity;
It is characterized by functioning as

The method for analyzing the visual impression of a face according to the present invention includes:
A facial visual impression analysis method using a facial impression analysis device comprising moving image acquiring means and physical quantity measuring means for analyzing the visual impression of a subject's face based on the movement of the skin when the expression changes. hand,
a step in which the moving image acquiring means acquires a moving image including the process of the facial expression change;
and calculating a physical quantity caused by the facial expression change by the physical quantity measuring means.

The method for distinguishing cosmetic effects according to the present invention includes:
A method for discriminating cosmetic effects by discriminating the visual impression of a subject's face based on physical quantities of skin changes caused by changes in facial expression,
a pre-makeup physical quantity measurement step of extracting the physical quantity before makeup caused by a change in facial expression before makeup;
a pre-makeup visual evaluation value calculation step of calculating a pre-makeup visual evaluation value based on the correlation between the physical quantity and the visual evaluation value of the visual impression of the face;
A post-makeup physical quantity measurement step of extracting the physical quantity after makeup caused by a change in facial expression after makeup;
A post-makeup visual evaluation value calculation step of calculating a post-makeup visual evaluation value based on the correlation;
and a cosmetic effect evaluation step of comparing the visual evaluation value before makeup and the visual evaluation value after makeup to evaluate the cosmetic effect.

Of the physical quantities generated by changes in facial expression, physical quantities (determinants) that are highly correlated with the visual impression of the face in the parts that determine the visual impression of the face are used as indices for discriminating the visual impression of the face. It is possible to simply and accurately discriminate the visual impression of a face that a person receives visually.

4 is a flow chart showing a method of extracting a determining part of a facial visual impression and a method of extracting a determinant of a facial visual impression according to the present invention. 4 is a flow chart showing a process of measuring a physical quantity from a moving image; 1 is a functional block diagram of a facial impression analysis device according to the present invention; FIG. 4 is a flow chart showing a method for discriminating a visual impression of a face by the facial impression analysis device according to the present invention; 4 is a flowchart showing visual evaluation value/physiological quantity calculation processing by the facial impression analysis apparatus according to the present invention. FIG. 10 is a diagram showing an example of moving images/continuous still images used for obtaining a visual evaluation value; (Photo substitute for drawing) FIG. 10 is a diagram showing a method of presenting moving images/continuous still images to an evaluator when obtaining a visual evaluation value; (Photo substitute for drawing) FIG. 10 is a diagram showing the total result of visual evaluation values for moving images/continuous still images; FIG. 10 is a diagram showing an example of a result of line-of-sight evaluation for moving images/continuous still images; (Photo substitute for drawing) FIG. 10 is a diagram showing changes in facial expression of an evaluation target in a moving image; (Photo substitute for drawing) FIG. 10 is a diagram showing temporal changes in moving speed of a marker in a moving image; FIG. 10 is a diagram showing integrated values of moving speeds of markers in a moving image; FIG. 10 is a diagram showing placement of markers for motion capture; FIG. 10 is a diagram showing changes in facial expression of an evaluation target in a moving image; (Photo substitute for drawing) It is a figure which shows the time change of the variation|change_quantity per unit time of the distance between the markers in a moving image. FIG. 10 is a diagram showing setting of a region to be analyzed for changes in area over time; 4 is a graph showing measurement results of skin stretch ratio in an analysis target region. FIG. 4 is a diagram showing an example of grid point setting and motion vectors; (Photo substitute for drawing) FIG. 10 is a diagram showing changes in facial expression of an evaluation target in a moving image; (Photo substitute for drawing) FIG. 10 is a diagram showing a measurement result of temporal change in motion vector magnitude; FIG. 10 is a diagram showing average widths of waveforms of motion vectors in a moving image;

[1] A method of extracting a determining part of a facial visual impression and a method of extracting a determinant of a facial visual impression according to the present invention. A method of evaluating the impression and extracting the determining part of the visual impression of the face by analyzing the gaze of the evaluator at that time, and measuring the physical quantity from the moving image and correlating it with the visual impression of the face from among them. This is a method for extracting determinants of visual impression of a face, which extracts factors with high .

The “visual impression of the face” referred to in the present invention is the visual impression received when another person sees the face to be evaluated. For example, focusing on the age impression, there are impressions such as youthful impression such as looking younger than actual age and aging impression such as looking older. In addition, there are various other impressions, such as attractiveness felt when viewing an evaluation target.

FIG. 1 is a flow chart showing a method of extracting a determining part of a facial visual impression and a method of extracting a determinant of a facial visual impression according to the present invention. With reference to this, a detailed description will be given of the method of extracting the determinant site and determinant factor of the visual impression of the face according to the present invention.

<Acquisition of moving images>
First, in step S11, a moving image of a face to be evaluated is acquired. Here, as described above, a moving image including changes in facial expression is acquired.

A moving image of the face to be evaluated may be used here, which is obtained by shooting a moving image of the face to be evaluated with a general camera device.

In general, moving images are composed of a series of many still images (frames), and the smoothness of motion is expressed by the frame rate, which indicates the number of frames per unit time. Here, when presenting a moving image to the evaluator in the subsequent process of obtaining a visual evaluation value, the frame rate should be set at a rate higher than the level at which the evaluator perceives that it is a moving image rather than a series of still images. It is preferable to acquire a moving image with

Alternatively, a stereo camera may be used to capture stereo moving images. With such a configuration, a three-dimensional analysis can be performed in the physical quantity measurement process described later.

More specifically, a motion that tracks the three-dimensional position of each marker by attaching multiple markers to the face to be evaluated, and then shooting a moving image of the face to be evaluated with a stereo camera. Capturing can be performed.

<Acquisition of visual evaluation value>
In step S12, a visual evaluation value for the moving image is acquired. This is done by presenting the moving image acquired in step S11 to the evaluator and presenting the impression that the evaluator received from the moving image.

For example, if the purpose is to extract the determinant parts and determinants of the age impression of the visual impression of the face, a moving image containing changes in the facial expression to be evaluated is presented to the evaluator, and the evaluation is performed. There are methods such as obtaining an estimate of the age of the subject (requiring an estimated age response). As another example, if the purpose is to extract determinants of facial visual impressions related to attractiveness, a moving image containing facial expression changes to be evaluated is presented to the evaluator as in the case of age impression. There is a method of requesting an answer as to whether or not the subject was found attractive after the evaluation. It can be said that the visual evaluation value thus obtained is a value that directly indicates the visual impression of the face to be evaluated.

In addition, one frame before the expression change and one frame after the expression change were extracted as still images from the moving image, and only those two frames were presented to the evaluator as continuous still images. A step of prompting the evaluation of the impression may be provided. As a result, the impression of the face of the evaluation object given to the evaluator by the continuous still images as the static stimulus is obtained in comparison with the impression of the face of the evaluation object given to the evaluator by the moving image as the dynamic stimulus. More detailed analysis can be performed to extract critical sites and impression determinants.

When presenting the moving image to the evaluator, the evaluator's line of sight information is acquired, and information such as how many times and for how many seconds the evaluator holds the line of sight on which area of the evaluation target face is obtained. .

<Extraction of impression-determining site>
In step S13, the visual line information of the evaluator when the moving image is presented to the evaluator is analyzed to extract impression-determining parts.

When an evaluator assigning an evaluation of a certain impression observes the moving image, the evaluator judges the impression regardless of consciously or unconsciously in the area where the gaze is held many times or for a long time. It can be said that it includes the area to be gazed at when doing so, that is, the site that determines the impression.

For example, when the evaluator is asked to answer the estimated age of the subject displayed in the moving image, and when the moving image is presented, the area where the evaluator's gaze stays many times and stays for a long time may indicate the age impression. Contains decision sites.

In step S13, as described above, the area where the evaluator's line of sight stays a lot during the visual evaluation and its surroundings are extracted as the determined parts of the visual impression.

<Measurement of physical quantity>
In the subsequent step S14, the moving image is analyzed and physical quantities are measured. This may be performed for only the impression-determining site specified in step S13, or may be performed for all regions included in the moving image. FIG. 2(a) is a flow chart showing a process of calculating a physical quantity using a skin motion vector obtained by analyzing a two-dimensional moving image by the optical flow method.

First, in step S21, feature points are extracted from each frame in the moving image. Here, a method of determining the unevenness of the skin from the brightness value of each pixel in the frame and setting the feature points, and a method of setting the feature points in a grid pattern in the moving image. Any method, such as an arrangement method, may be used.

Then, in step S22, the motion vector of each feature point is specified. A motion vector can be obtained from a change in luminance between frames or the like.

After that, in step S23, the motion vector is analyzed, and the physical quantity caused by the change in facial expression is calculated. Examples of physical quantities here include displacement amount, movement speed, acceleration, movement direction, etc. of coordinates of feature points mentioned above, and wave parameters of plane movement, such as period, circumference, etc., related to facial skin movement. quantity. In addition, it may be configured to calculate optical quantities such as luminance and color tone, their variation and frequency characteristics, and to perform analysis by PIV (Particle Image Velocity) or other methods. .

Each of these physical quantities may be obtained as a value in each frame, or may be obtained as an average value over the entire moving image. Alternatively, areas such as cheeks and chin may be set, and an integrated value or average value of values within the area may be obtained.

As another method of measuring the physical quantity, the process of calculating the physical quantity by tracking the coordinates of the feature points is shown in the flowchart of FIG. 2(b).

First, in step S31, similar to step S21, feature points are calculated by an arbitrary method. Alternatively, when the moving image is captured in step S11, markers or the like may be placed in advance on the face to be captured and used as feature points.

Then, in step S32, the coordinates of each feature point in each frame in the moving image are calculated.

Then, in step S33, changes in the coordinates of feature points between frames are tracked, and physical quantities are calculated as described in step S23. For example, it is the time change of the velocity and acceleration of each feature point. Alternatively, a feature point may be set as a reference point, and the change over time in the distance between it and any other feature point may be obtained. This is a value corresponding to the relative velocity of arbitrary feature points on the face. Still another physical quantity is the change over time of the area surrounded by three or more feature points. As a result, it is possible to quantify the mode of expansion and contraction of the skin at any site on the face.

Further, as described above, when stereo moving images are acquired in step S11, three-dimensional analysis can be performed taking into account not only the plane but also the depth. That is, in step S32, feature point coordinates in the three-dimensional space can be calculated by an arbitrary method such as the stereo method, and in step S33, coordinate changes of the feature points in the three-dimensional space can be tracked. As a result, it is possible to more accurately analyze the three-dimensional movement of the skin that accompanies changes in facial expression.

Either one or both of the measurement of the physical quantity using the motion vector and the measurement of the physical quantity by tracking the coordinates of the feature point may be used. Furthermore, moving image analysis may be performed by a method different from these methods to measure the physical quantity.

<Measurement of physiological quantity>
In step S15, a physiological quantity is measured for the skin of the face that is the object of shooting the moving image.
Physiological quantities related to the skin include surface physical properties and/or internal physical properties of the skin, composition of skin tissue, and the like, and those known to be correlated with the visual impression of the face can be used. For example, when the visual impression of the face is the impression of age, a physiological quantity relating to the skin, for which the mechanism involved in aging has been clarified, can be used. Such physiological quantities include skin surface physical properties such as skin surface roughness, stratum corneum epidermis physical properties such as stratum corneum water content, dermal physical properties such as collagen fiber bundle amount and structure, and fat layer physical properties such as fat. Physical properties of the papillary layer or papillary layer, such as the position and size of the body, such as the number of papillary structures per unit area, height, and the like. Said physiological quantity is preferably the physiological quantity of said impression-determining site.
Physiological quantities can be measured by conventionally known methods. The present invention is limited to non-invasive methods (methods without surgery).

Each of steps S11-S15 is performed on a plurality of different faces, sufficient for the following correlation analysis.

<Extraction of visual impression determinant>
In step S16, determinants of the visual impression of the face are extracted from the physical quantity obtained from the moving image in step S14.

Here, the correlation between the physical quantity obtained in step S14 and the visual evaluation value obtained in step S12 or the physiological quantity measured in step S15 is analyzed, and from among the physical quantities, the visual evaluation value or the physiological quantity Extract factors that determine

For example, comparing multiple moving images of evaluation targets with different visual evaluation values, and among the physical quantities measured from those moving images, if there is a difference between the two, the correlation with the visual evaluation value is calculated. expected to have Then, by performing similar comparisons on other video images, we clarified the physical quantities that have correlations with the visual evaluation values and are determinants of the visual impression, as well as the correlations between these physical quantities and the visual evaluation values. can be
Correlation analysis can be performed using a conventionally known method.

In addition, as described above, when obtaining the visual evaluation value in step S12, if the visual evaluation value for the case where two still images before and after the expression change are presented is obtained, the corresponding moving image may be compared with a visual evaluation value for For example, when the visual evaluation value for a still image and the visual evaluation value for a moving image of the same evaluation target are significantly different, the physical quantity contained in the moving image is considered to be the determinant of the visual impression of the face. can be estimated.

Similarly, by analyzing the correlation between the physical quantity obtained from the moving image and the physiological quantity related to the skin, it is possible to determine the physical quantity highly correlated with the physiological quantity related to the skin, and the physical quantity is the appearance of the face. can be presumed to be a determinant of the impression of

As described above, according to the present invention, a moving image including changes in facial expression is presented to an evaluator, and by analyzing the line of sight of the evaluator, a portion that determines the visual impression of the face is determined, and the moving image is used. By extracting more physical quantities and analyzing them, it is possible to extract determinants of the visual impression of the face.

Acquisition of visual evaluation values and analysis of correlations with physical quantities, measurement of physiological quantities and analysis of correlations with physical quantities may be performed in both, or only one of them may be performed as necessary. you can go Further, different evaluation values and quantities may be used, and the correlation between them and physical quantities measured from moving images may be analyzed.

[2] A method for discriminating a visual impression of a face and a facial impression analysis device according to the present invention The present invention obtains a moving image of a part of the subject's face that determines the impression, including expression changes, and analyzes the moving image to measure the physical quantity. and a method for discriminating the visual impression of a subject's face, and a facial visual impression analysis apparatus.

As shown in the functional block diagram of FIG. and a quantitative calculation means 5 .

Furthermore, the moving image acquiring means 1 has a stereo moving image acquiring means 11, and the physical quantity measuring means 2 includes a feature point extracting means 21, a motion vector identifying means 22, a motion vector analyzing means 23, and a feature point coordinate calculating means. It has means 24 and feature point tracking means 25 .

The correlation storage means 3 analyzes the formula showing the correlation obtained by analyzing the physical quantity and the visual evaluation value, and the physical quantity and the physiological quantity in the process of extracting the determinant of the visual impression of the face. Holds information such as the formula that indicates the obtained correlation.

The facial impression analysis apparatus according to the present invention uses a general-purpose computer device that includes an arithmetic device, a main memory device, an auxiliary memory device, and various input/output devices. It is realized by storing a facial impression analysis program for causing the computer device to function as each means described above. When the computer device functions as a facial impression analysis device, the facial impression analysis program is developed on the main storage device as necessary and processed by the arithmetic device.

In addition, a camera device is used as the moving image acquisition means 1, and the moving image obtained by the camera device is transferred to the physical quantity measuring means 2, or a configuration in which the correlation storage means 3 is realized by a general-purpose server device. A plurality of devices may be configured to enable data transfer using a USB (Universal Serial Bus) connection or recording media, and communication via a network such as the Internet or a LAN (Local Area Network).

<Acquisition of moving images>
FIG. 4 is a flow chart showing a method for discriminating visual impressions of a subject's face by a facial impression analyzer. First, in step S41, the moving image acquisition means 1 is used to acquire a moving image including facial expression changes of the subject. Note that the stereo moving image acquisition means 11 may acquire a stereo moving image including facial expression changes of the subject. Also, at this time, a moving image may be obtained in which markers used as feature points in subsequent measurement of physical quantities are placed in advance on the subject's face.

<Measurement of physical quantity>
Then, in step S42, the physical quantity is measured using the physical quantity measuring means 2. FIG.
Here, the physical quantity extracted as the determinant of the visual impression of the face is calculated by the above-described method of extracting the determinant of the visual impression of the face in the same manner as the processing described above with reference to FIG. conduct.

When the physical quantity is calculated using the motion vector shown in FIG. 2(a), the feature point extracting process of step S21 is performed by the feature point extracting means 21, and the motion vector specifying process of step S22 is performed by the motion vector specifying means. 22, the motion vector analysis means 23 performs the motion vector analysis process of step S23.

When calculating physical quantities using the feature point coordinates shown in FIG. After the coordinate calculation means 24 performs the tracking process of the feature point coordinates in step S33, the feature point tracking means 25 performs the tracking process.

If the stereo moving image has been acquired in step S41, the feature point coordinates in the three-dimensional space are calculated by any method such as the stereo method in step S32, and the feature point coordinates in the three-dimensional space are calculated in step S33. Tracking of coordinate changes may be performed.

Note that when the physical quantity is calculated using both the measurement of the physical quantity using the motion vector and the measurement of the physical quantity using the feature point coordinates, whichever of the feature point coordinate calculation processes in steps S21 and S31 is performed Only one method may be performed and the feature points may be shared. Alternatively, in order to effectively measure the physical quantity in each method, both steps S21 and S31 may be performed to extract different feature points. good.

<Analysis of visual impression of face>
In step S43, the physical quantity measured in step S42 is analyzed, and the visual impression of the face is discriminated. Thereby, at least one of a visual evaluation value of the visual impression of the subject's face and a physiological quantity relating to the skin of the subject's face is calculated.

FIG. 5(a) is a flow chart showing a visual evaluation value calculation process by the visual evaluation value calculation means 4. FIG. First, in step S51, the correlation data between the physical quantity and the visual evaluation value is acquired from the correlation storage means 3. FIG.

Then, in step S52, a visual evaluation value is calculated based on the physical quantity obtained in step S42 and the correlation data obtained in step S51. The visual impression of the subject's face can be discriminated from the calculated visual evaluation value.

FIG. 5(b) is a flow chart showing the physiological quantity calculation process by the physiological quantity calculation means 5. As shown in FIG. This is also the same processing as the above-described visual evaluation value calculation processing.

Then, in step S62, a physiological quantity is calculated based on the physical quantity obtained in step S42 and the correlation data obtained in step S61. The visual impression of the subject's face can be discriminated from the calculated physiological quantity.

As described above, the physical quantity is measured by analyzing the moving image containing the change in facial expression of the subject, and from the physical quantity of the subject, based on the correlation between the physical quantity and the visual evaluation value or the physiological quantity, the subject A visual evaluation value and/or a physiological quantity for is calculated, and the visual impression of the subject's face can be discriminated from the calculated value.

If a stereo moving image is obtained using the stereo moving image obtaining means 11, three-dimensional analysis can be performed by the physical quantity measuring means 2, but the stereo moving image obtaining means 11 is a general monocular camera. It is expensive compared to others. Therefore, the stereo moving image acquiring means 11 may be omitted, a general monocular camera or the like may be provided as the moving image acquiring means 1, and only two-dimensional analysis may be performed by the physical quantity measuring means 2. FIG. With such a configuration, the facial impression analysis device can be realized at low cost.

Further, as a method for discriminating a visual impression of a face and a method for using the facial impression analysis apparatus according to the present invention, there is a method for discriminating a makeup effect. That is, by comparing the results of visual impression analysis performed using a moving image of the subject's face before makeup and the visual impression analysis performed using a moving image of the subject's face after applying makeup, It distinguishes the effect that makeup has on the visual impression.

Hereinafter, the present invention will be described in more detail with examples of obtaining a visual evaluation value of the age impression of the appearance of the face, extracting impression-determining parts, and measuring physical quantities. It goes without saying that it is not limited only to the examples.

[1] Acquisition of visual evaluation values using moving images and continuous still images <Acquisition of moving images>
For 20 evaluation subjects (aged 20s to 60s), 9-second moving images including changes in facial expressions were taken. Then, one frame before the expression change and one frame after the expression change were both extracted as still images from the moving image, and continuous still images were created by displaying only these two frames for 9 seconds. FIG. 6A shows a schematic diagram of a moving image, and FIG. 6B shows a schematic diagram of continuous still images.

<Acquisition of visual evaluation value>
Using a monitor, 15 evaluators randomly displayed the above moving images or continuous still images, and acquired a visual evaluation value of the impression of the appearance of the face. When the moving image and continuous still images are presented to the evaluator, a line-of-sight analysis device is used to analyze how many times and how many seconds the evaluator's line of sight stays in which area of the moving image and still image. rice field. FIG. 7 shows a schematic diagram of a method of presenting moving images and still images to the evaluator.

First, as shown in FIG. 7(a), a number was displayed, then a face to be evaluated was displayed, and an instruction was presented asking for an answer as to whether the generation of the number is the first half or the second half. As a result, the evaluator will be aware of the age impression of the evaluation target when viewing the moving image or continuous still images that will be displayed thereafter.

Then, numbers were displayed as shown in FIG. 7(b). This number is obtained by truncating the real age of the subject to be evaluated in the moving image or continuous still image after this. Since the ages of the subjects in this example are in their 20s to 60s as described above, any number of 20, 30, 40, 50, and 60 is displayed.

Then, as shown in FIG. 7(c), a moving image or continuous still images for 9 seconds are displayed, and then, as shown in FIG. 7(d), the evaluation object displayed in FIG. Answers were asked whether it was the first half or the second half of the generation indicated in 7(b).

In other words, in the example shown in FIG. 7, the age of the subject of evaluation is somewhere between 30 and 39 years old, and the evaluator answered that he was in his early 30s or late 30s.

If the evaluator evaluates that the evaluator is in the second half of the generation based on the evaluator's responses obtained by the above-described procedure, it can be determined that the evaluator has an aging impression on the subject.

FIG. 8 shows visual evaluation values acquired from evaluators for all evaluation objects. As shown here, it can be read that the moving images gave the evaluators a higher impression of age than the continuous still images. From this, it can be seen that the movement during facial expression change contained in the moving image contains a factor that gives the evaluator a high impression of age.

<Extraction of impression-determining site>
FIG. 9 shows the result of analysis of the line of sight of an evaluator when a moving image and continuous still images of a certain evaluation object are displayed. Fig. 9(a) shows that when the moving image to be evaluated is displayed, and Fig. 9(b) shows that when the continuous still images to be evaluated are displayed, the evaluator's gaze stays frequently. is.

Looking at these line-of-sight analysis results, it can be seen that the evaluator focused on the cheeks and around the eyes of the evaluation target regardless of whether the moving image or the continuous still image was displayed. From this, it was found that the areas that determine the impression of age on the face are the cheeks and the periphery of the eyes.

In addition, although the site that determines the age impression is the same for both the moving image and the continuous still images, the age impression given by the moving image is higher than that of the moving image. It was found that it is in the physical quantity caused by the movement at the time of facial expression change in the part.

[2] Measurement of physical quantities by motion capture <Acquisition of video images>
Thirty women in their 20s to 60s were evaluated, and reflective markers were attached to their faces to indicate measurement points. Multiple cameras were used to capture moving images of each subject's facial expressions. gone. By analyzing this moving image, motion capture was performed to track changes in the three-dimensional coordinates of the marker. In addition, the moving images were presented to the evaluators, and visual evaluation values for the age of each evaluation subject were obtained.

Here, from the state of a straight face, to the state where the mouth is wide open as shown in FIG. 10(a), and after taking the expression with the mouth closed as shown in FIG. A video containing a series of facial expression changes was used for motion capture and acquisition of visual evaluation values.

Then, as shown below, the movement position, movement amount, movement speed of the markers, the relationship between the movements of the two markers, and the time change of the area surrounded by three or more markers are analyzed, and physical quantities are extracted. did.

<Measurement of moving speed of marker>
When the movement of the marker on the face of each evaluation target was tracked and the movement speed was measured, the results shown in FIG. 11 were obtained.

FIG. 11(a) shows an evaluation target whose visual evaluation value for age is higher than the actual age, that is, an evaluation target with a high visual impression of age, and FIG. 10 is a graph showing temporal changes in the moving speed of the marker in each of the evaluation subjects with a low age impression, that is, the evaluation subjects with a low visual impression of age. It should be noted that the vertical axis represents the moving speed of the marker, and the horizontal axis represents the number of frames in the moving image as time elements. Also, the velocity direction of the marker when the facial expression changes from a straight face to a facial expression with the mouth wide open as shown in FIG. The movement speed of the marker when the expression changes to the closed expression is shown as the speed in the contraction direction.

As shown here, the graph of the speed change of the feature point of the evaluation object that looks old has a sharper peak than that of the evaluation object that looks young, and the speed converges faster. In other words, it can be said that the change in the moving speed of the feature point of the evaluation target that looks old is more drastic than that of the evaluation target that looks young.

FIG. 12 shows the moving speed of the evaluation target feature points that look old shown in FIG. 11A and the moving speed of the evaluation target feature points that look young shown in FIG. 11B. are integrated, and the area of each graph is calculated.

In FIG. 12, the young-looking evaluation target has a larger graph area, which also indicates that the change in the moving speed of the feature points of the young-looking evaluation target is slower than that of the old-looking evaluation target. can be read.

From these results, it was found that feature points, i.e., supple movements in which the skin gently follows changes in facial expression, may give the evaluator an impression of looking younger.

<Relationship between movement of two markers>
FIG. 13 is a diagram showing markers set on the left side of the face to be evaluated. As shown here, a marker on the upper part of the forehead is set as a reference point M0, and two cheek markers M1, M2, and a chin marker M3 are set.

Then, the distance L1 between the reference point M0 and the marker M1 when the expression changes from the straight face state as shown in FIG. 14(a) to the expression with the mouth wide open as shown in FIG. 14(b) change per unit time V1, change per unit time V2 in distance L2 between reference point M0 and marker M2, and change per unit time in distance L3 between reference point M0 and marker M3 Time changes of three amounts of change of the amount V3 and V3 were measured. In other words, it can be said that this is the change over time of the relative velocities of the markers M1, M2, and M3 with reference to the reference point M0.

As a result of the above measurements, the results shown in the graph of FIG. 15 were obtained. FIG. 15(a) is a graph showing measurement results of an evaluation object evaluated as having a young visual impression in visual evaluation. Looking at this, it can be read that although the absolute values of the maximum values of V1 to V3 are different, the times at which the respective maximum values are obtained are very close. On the other hand, looking at the measurement results of the evaluation target that was evaluated to have an aged visual impression in the visual evaluation shown in FIG. It resulted in taking

In other words, it can be said that the movement of the cheek indicated by the markers M1 and M2 lags behind the movement of the jaw indicated by the marker M3 in the facial expression change of the evaluation target with an aged visual impression.

From this result, it was found that when the cheek movements follow well when facial expressions change, the evaluator may be given the impression of looking younger.

<Time change of area surrounded by three or more markers>
As shown in FIG. 16, two regions were set, a chin A1 surrounded by three markers and a cheek A2 surrounded by four markers. Then, when the change in the area of each region over time was measured when the facial expression changed, and the results of integrating these values over time were calculated as the stretch ratio of the skin in each region, the results shown in the graph of FIG. 17 were obtained. Got.

FIG. 17(a) is a graph showing measurement results of an evaluation object evaluated as having a young visual impression in visual evaluation. FIG. 17(b) is a graph showing measurement results of an evaluation object that was visually evaluated as having an aged visual impression. Comparing these, the evaluation subjects with a young appearance impression have a higher expansion ratio in the cheek area A2 than the chin area A1, while the evaluation subjects with an old appearance impression have a higher elasticity ratio in the cheek area A2 than in the chin area A1. The result was that the elasticity of the material was low.

From this result, it was found that when the skin of the cheek A2 expands and contracts more flexibly than the skin of the chin A1, there is a possibility of giving the evaluator an impression of looking younger.

[3] Measurement of physical quantity by optical flow method <Acquisition of moving images>
For each of 80 women in their twenties to sixties to be evaluated, two-dimensional moving images including changes in facial expressions were taken. Grid-like lattice points were set in the moving image, and the motion vector of the skin at each lattice point in each frame was obtained from the change in gradation between each frame in the moving image, as shown in FIG. In addition, the moving images were presented to the evaluators, and visual evaluation values for the age of each evaluation subject were obtained.

For the analysis of the moving image and the acquisition of the visual evaluation value, the facial expression with the mouth open shown in FIG. We used a moving image that changed to a serious expression and then returned to a straight face.

<Change in motion vector>
A graph as shown in FIG. 20 was obtained as a result of analyzing the magnitude of the motion vector during facial expression change, ie, the speed of skin movement, for the cheek region. This graph shows the temporal change in the speed of skin movement, with the vertical axis representing the average magnitude of the motion vector in each frame and the horizontal axis representing the number of frames, that is, the time axis. Looking at this, compared to the measurement results of the evaluation target, which was evaluated as having a young visual impression in the visual evaluation indicated by the solid line in the graph, it was evaluated as having an aged visual impression in the visual evaluation indicated by the broken line in the graph. The measurement result of the evaluated subject has a shape with a sharp peak, especially when the facial expression changes as shown in FIG. 19(c).

FIG. 21 is a graph showing the result of calculating the time width for convergence of the speed of motion of the skin of the evaluation target that looks old and the speed of motion of the skin of the evaluation target that looks young shown in FIG. 20 . Looking at this, similar to the skin physical properties indicated by the measurement results of the speed of movement of the marker when the facial expression changes with the motion capture video, which was described earlier with reference to FIG. The graph value is large, and it can be read that the change in movement is gradual.

In other words, instead of 3D video motion capture analysis, which requires physically placing a marker on the face to be evaluated and shooting with multiple cameras, a 2D image can be easily captured. It was suggested that similar analysis results could be obtained by analyzing moving images using the optical flow method.

INDUSTRIAL APPLICABILITY The present invention can be used to easily and accurately discriminate a customer's visual age impression in situations such as cosmetics sales, counseling, and esthetics.

1 Moving image acquiring means 11 Stereo moving image acquiring means 2 Physical quantity measuring means 21 Feature point extracting means 22 Motion vector identifying means 23 Motion vector analyzing means 24 Feature point coordinate calculating means 25 Feature point tracking means 3 Correlation storing means 4 Visual evaluation value Calculation means 5 Physiological quantity calculation means

Claims (16)

The visual impression of the subject's face is discriminated based on the physical quantity of skin changes caused by changes in facial expression at characteristic points on the cheeks, which are impression-determining parts that determine the visual impression of the face, and characteristic points on the chin . , a method for discriminating the visual impression of a face,
wherein the physical quantity is the difference between the time when the characteristic points of the cheek and the characteristic points of the chin take the maximum value of the speed of movement of each characteristic point associated with the change in facial expression. A method of distinguishing the visual impression of The physical quantity is calculated based on a quantity related to time change of a distance between one reference point included in the face and each of the characteristic points of the cheek and the characteristic points of the chin. The method for discriminating the visual impression of a face according to claim 1, wherein: Discrimination of the visual impression of the face calculates a visual evaluation value of the visual impression of the subject's face from the measured value of the physical quantity of the subject based on the correlation between the physical quantity and the visual evaluation value of the visual impression of the face. 3. The method for discriminating a visual impression of a face according to claim 1 or 2 , characterized by comprising: 4. The method for discriminating visual impression of a face according to claim 1 , wherein the visual impression of the face is an impression of age. A facial impression analysis device that analyzes the visual impression of a subject's face based on the movement of the skin when the facial expression changes,
a moving image acquiring means for acquiring a moving image including the facial expression change process;
a physical quantity measuring means for measuring a physical quantity of a skin change caused by the facial expression change from the moving image;
The physical quantity measuring means has feature point extraction means for extracting a plurality of feature points from the moving image,
wherein the plurality of feature points includes facial cheek feature points and facial chin feature points;
wherein the physical quantity is a difference between the time when the characteristic points of the cheek and the characteristic points of the chin take the maximum speed of movement of the respective characteristic points due to the change in facial expression. Impression analysis device. The physical quantity measuring means is
motion vector identification means for identifying a motion vector of the feature point from the moving image;
6. The facial impression analysis apparatus according to claim 5, further comprising motion vector analysis means for analyzing said motion vector and calculating said physical quantity. 7. The facial impression analysis apparatus according to claim 6 , wherein said physical quantity includes a quantity relating to temporal change in magnitude and/or direction of said motion vector. The physical quantity measuring means is
feature point coordinate calculation means for calculating the coordinates of the feature points in each frame of the moving image;
6. The facial impression analysis apparatus according to claim 5, further comprising feature point tracking means for identifying the corresponding feature points between the frames and calculating the physical quantity. The physical quantity is calculated based on a quantity related to time change of a distance between one reference point included in the face and each of the characteristic points of the cheek and the characteristic points of the chin. 9. The facial impression analysis device according to claim 8 , wherein: 10. The facial impression analysis apparatus according to any one of claims 5 to 9 , wherein the visual impression of said face is an impression of age. Correlation storage means for storing a correlation between the physical quantity and a visual evaluation value of the visual impression of the face obtained by visual evaluation of the face,
further comprising visual evaluation value calculation means for calculating a visual evaluation value of visual impression of the subject from the measured values of the physical quantities of the subject based on the correlation stored in the correlation storage means. The facial impression analysis device according to any one of claims 5 to 10 . having correlation storage means for storing a correlation between the physical quantity and a physiological quantity related to the skin;
Physiological quantity calculation means for calculating a physiological quantity of the subject's skin from the measured values of the physical quantity of the subject based on the correlation stored in the correlation storage means. The facial impression analysis device according to any one of claims 5 to 11 . 13. The facial impression analysis apparatus according to claim 12 , wherein said physiological quantity is skin surface physical properties and/or internal physical properties of said face. A facial impression analysis program that analyzes the visual impression of a subject's face based on the movement of the skin when the facial expression changes,
the computer,
means for measuring physical quantities of skin changes caused by facial expression changes;
means for discriminating the visual impression of the subject's face based on the physical quantity;
A facial impression analysis program that functions as
The physical quantity is the difference between the time when the speed of movement of each characteristic point of the cheek included in the face and the characteristic point of the chin included in the face reaches the maximum value due to the change in facial expression. A facial impression analysis program characterized by: A facial visual impression analysis method using a facial impression analysis device comprising moving image acquiring means and physical quantity measuring means for analyzing the visual impression of a subject's face based on the movement of the skin when the expression changes. hand,
a step in which the moving image acquiring means acquires a moving image including the process of the facial expression change;
the physical quantity measuring means calculating a physical quantity of the skin change caused by the facial expression change;
The physical quantity is the difference between the time when the speed of movement of each characteristic point of the cheek included in the face and the characteristic point of the chin included in the face reaches the maximum value due to the change in facial expression. A method for analyzing the visual impression of a face, characterized by: A method for discriminating cosmetic effects by discriminating the visual impression of a subject's face based on physical quantities of skin changes caused by changes in facial expression,
a pre-makeup physical quantity measurement step of extracting the physical quantity before makeup caused by a change in facial expression before makeup;
a pre-makeup visual evaluation value calculation step of calculating a pre-makeup visual evaluation value based on the correlation between the physical quantity and the visual evaluation value of the visual impression of the face;
A post-makeup physical quantity measurement step of extracting the physical quantity after makeup caused by a change in facial expression after makeup;
A post-makeup visual evaluation value calculation step of calculating a post-makeup visual evaluation value based on the correlation;
a cosmetic effect evaluation step of comparing the pre-makeup visual evaluation value and the post-makeup visual evaluation value to evaluate the cosmetic effect;
The physical quantity is the difference between the time when the speed of movement of each characteristic point of the cheek part included in the face and the characteristic point of the chin part included in the face reaches the maximum value due to the change in expression. A method for distinguishing a cosmetic effect, characterized by:

Images