JP6651366B2 - Extraction method of determinants of facial appearance impression - Google Patents

Description

  The present invention relates to a method of extracting a determined part of a facial appearance impression, a method of extracting a determinant of a facial appearance impression, a method of distinguishing a facial appearance impression, a facial impression analysis device, a facial impression analysis program, and an analysis of a facial appearance impression. It is about the method.

  People give a variety of impressions to others due to various factors such as their appearance and behavior. Among them, visual information, especially the appearance of the face, can be said to be one of the very important factors that determine the impression of the person.

  Therefore, research is being conducted on a technique for objectively estimating the impression of the appearance of a human face. For example, in Patent Document 1, a plurality of tracking points are arranged in advance in an analysis area such as the corner of the eye of a subject, a moving image is photographed, and the amount of change of the tracking points accompanying a change in facial expression is tracked to reduce the skin compression ratio. A technique for analyzing the skin condition of a subject by acquiring the subject has been disclosed.

JP 2014-193197 A

  As described above, the parameters for objectively estimating the visual impression of the subject's face have been studied. However, this is only to obtain specific parameters from a specific part of the subject's face.

  However, a human appearance impression is formed by various factors, and a more accurate estimation means has been desired.

  In view of the above, the present invention provides a method of extracting a portion having a large influence on a face appearance impression, a factor having a high correlation with the face appearance impression, and a method of discriminating the face appearance impression based on the factor. As an issue.

In order to solve the above problems, a method for extracting a determined part of a facial appearance impression according to the present invention,
A visual evaluation step of presenting dynamic information of the facial expression change to the evaluator, and causing the evaluator to evaluate the visual impression of the face;
A line-of-sight analysis step of analyzing line-of-sight data of the evaluator who views the dynamic information and analyzing an area in the dynamic information that the evaluator gazes at.

  Thereby, a part to which the person pays attention when evaluating the facial appearance impression based on the dynamic information, that is, a part that determines the facial appearance impression can be extracted.

  In a preferred aspect of the present invention, the appearance impression of the face is an age impression.

The method for extracting the determinant of the visual impression of the face according to the present invention,
A physical quantity measuring step of measuring a physical quantity of a skin change caused by a change in facial expression, in an impression determining portion for determining a visual impression of the face,
Extracting a factor having a high correlation with the visual impression of the face from the physical quantities.
Thereby, it is possible to specify a physical quantity used in discriminating a face impression described later.

  In a preferred aspect of the present invention, the physical quantity measuring step includes a step of measuring a quantity related to the marker as the physical quantity from a motion capture moving image captured by setting a plurality of markers on the face. .

  In a preferred embodiment of the present invention, the amount relating to the marker includes at least one of a position, an amount of movement, a magnitude of a velocity, a direction of a velocity, a magnitude of an acceleration, and a direction of an acceleration of the marker. I do.

  In a preferred aspect of the present invention, the amount related to the marker includes an amount related to a temporal change in a distance between the plurality of markers.

  In a preferred aspect of the present invention, the amount related to the marker includes an amount related to a time change of a value indicating an area and / or a shape of a surface formed by three or more of the markers.

In a preferred embodiment of the present invention, the physical quantity measuring step includes a grid point setting step of setting a plurality of grid points on the face,
The physical quantity includes a quantity related to the grid point.

  In a preferred aspect of the present invention, the quantity relating to the grid point includes a quantity relating to a temporal change in the magnitude and / or direction of the velocity vector for each grid point.

In a preferred embodiment of the present invention, a method for extracting a determinant of a facial appearance impression includes: providing dynamic information including a change in the facial expression to an evaluator to obtain a visual evaluation value of the facial appearance impression; Having an acquisition step,
The factor extracting step includes extracting a factor by analyzing a correlation between the physical quantity and the visual evaluation value.
By using the visual evaluation value, it is possible to extract a factor having a high correlation with the impression received when a person actually looks at the face.

In a preferred embodiment of the present invention, the method for extracting a determinant of a facial appearance impression includes measuring a physiological amount related to the skin, comprising a physiological amount measuring step.
The factor extracting step includes analyzing a correlation between the physical quantity and the physiological quantity to extract the factor.
By using the physiological amount, it is possible to extract a determinant of a face impression that is highly correlated with a physiological phenomenon in a living tissue such as the skin.

  In a preferred aspect of the present invention, the physiological quantity is a skin surface property and / or an internal property at the impression determining site.

  In a preferred aspect of the present invention, the appearance impression of the face is an age impression.

  In a preferred aspect of the present invention, the physical quantity is a quantity relating to skin movement at the impression determining site.

The method for discriminating the visual impression of the face according to the present invention,
A facial impression of a subject is distinguished based on a physical quantity of a skin change caused by a facial expression change in an impression determining portion for determining a facial impression.
By using the physical quantity generated in the impression determining portion, the appearance impression of the face given to another person can be more accurately distinguished.

  In a preferred aspect of the present invention, the physical quantity includes at least one of a magnitude of a movement speed of a feature point included in the face, a direction of the velocity, a magnitude of the acceleration, and a direction of the acceleration. I do.

  In a preferred aspect of the present invention, the physical quantity includes a quantity relating to a temporal change in a distance between one reference point included in the face and each of at least two feature points included in the face. I do.

  In a preferred aspect of the present invention, the physical quantity includes a quantity related to a temporal change of an area surrounded by at least three feature points included in the face.

  In a preferred embodiment of the present invention, the discrimination of the facial impression is based on a correlation between the physical quantity and a visual evaluation value of the facial impression. The method includes calculating a visual evaluation value of the impression.

  In a preferred embodiment of the present invention, the discrimination of the visual impression of the face is based on a correlation between the physical quantity and a physiological quantity related to the skin, the physiological quantity related to the skin of the subject from the measured value of the physical quantity related to the subject. It is characterized by including calculating.

  In a preferred aspect of the present invention, the physiological quantity is a skin surface property and / or an internal property at the impression determining site.

  In a preferred aspect of the present invention, the appearance impression of the face is an age impression.

  In a preferred aspect of the present invention, the physical quantity is a quantity relating to skin movement at the impression determining site.

The face impression analysis device according to the present invention,
A facial impression analysis device that analyzes a visual impression of a subject's face based on a movement of the skin when a facial expression changes,
Moving image acquisition means for acquiring a moving image including the process of the facial expression change,
Physical quantity measuring means for measuring a physical quantity caused by the change of the facial expression from the moving image.
With the device having such a configuration, the method for distinguishing the visual impression of a face according to the present invention can be implemented.

In a preferred embodiment of the present invention, the physical quantity measuring unit extracts a plurality of feature points from the moving image,
A motion vector specifying unit that specifies a motion vector of the feature point from the moving image;
Motion vector analyzing means for analyzing the motion vector and calculating the physical quantity.

  In a preferred aspect of the present invention, the physical quantity includes a quantity related to a temporal change in the magnitude and / or direction of the motion vector.

In a preferred embodiment of the present invention, the physical quantity measuring unit extracts a plurality of feature points from the moving image,
Feature point coordinate calculation means for calculating coordinates of the feature points in each frame of the moving image,
And a feature point tracking unit for identifying the corresponding feature point between the frames and calculating the physical quantity.

  In a preferred aspect of the present invention, the physical quantity includes at least one of a coordinate displacement amount, a moving speed, an acceleration, a moving direction, and an acceleration direction of at least one of the feature points.

  In a preferred aspect of the present invention, the physical quantity includes a quantity relating to a temporal change in a distance between one reference point included in the face and each of at least two feature points included in the face. I do.

  In a preferred aspect of the present invention, the physical quantity includes a quantity relating to a temporal change of an area surrounded by at least three of the feature points.

  In a preferred aspect of the present invention, the measurement of the physical quantity is performed on an impression determining portion for determining a visual impression of a face.

  In a preferred aspect of the present invention, the appearance impression of the face is an age impression.

In a preferred embodiment of the present invention, the face impression analysis device includes a correlation storage unit that stores a correlation between the physical quantity and a visual evaluation value of a visual impression of the face obtained by visually evaluating the face. ,
A visual evaluation value calculation unit configured to calculate a visual evaluation value of a visual impression of the subject from the measured value of the physical quantity of the subject based on the correlation stored in the correlation storage unit.

In a preferred embodiment of the present invention, the face impression analysis device has a correlation storage unit that stores a correlation between the physical quantity and a physiological quantity related to skin,
A physiological quantity calculating means for calculating a physiological quantity of the skin of the subject from the measured value of the physical quantity of the subject based on the correlation stored in the correlation storing means.

  In a preferred aspect of the present invention, the physiological quantity is a skin surface property and / or an internal property at the impression determining site.

The face impression analysis program according to the present invention comprises:
A facial impression analysis program that analyzes a visual impression of a subject's face based on movement of the skin at the time of a facial expression change,
Computer
Means for measuring the physical quantity of skin changes caused by changes in facial expressions;
Means for distinguishing the visual impression of the subject's face based on the physical quantity,
It is characterized by functioning as

The method for analyzing the visual impression of a face according to the present invention includes:
A method for analyzing a visual impression of a face using a facial impression analysis device including a moving image acquisition unit and a physical quantity measurement unit, which analyzes a visual impression of a subject's face based on a movement of the skin when a facial expression changes. hand,
The moving image obtaining means obtains a moving image including the process of changing the facial expression,
Calculating the physical quantity generated by the change in the facial expression.

The method for discriminating the cosmetic effect according to the present invention includes:
A method for discriminating a cosmetic effect by discriminating a visual impression of a subject's face based on a physical quantity of a skin change caused by a facial expression change,
A pre-makeup physical quantity measurement step of extracting the physical quantity before make-up, which is caused by a facial expression change before make-up,
Based on the correlation between the physical quantity and the visual evaluation value of the visual impression of the face, a visual evaluation value before makeup calculating a visual evaluation value before makeup,
A post-makeup physical quantity measuring step of extracting the post-makeup physical quantity, which is caused by a change in facial expression after makeup,
Based on the correlation, a post-makeup visual evaluation value calculation step of calculating a post-makeup visual evaluation value,
A cosmetic effect evaluation step of comparing the visual evaluation value before the makeup and the visual evaluation value after the makeup to evaluate a cosmetic effect.

  By using a physical quantity (determining factor), which has a high correlation with the facial appearance impression, in the part that determines the facial appearance impression among the physical quantities generated by the facial expression change as an index for discriminating the facial appearance impression, In addition, it is possible to easily and accurately discriminate the visual impression of the face received by a person visually.

4 is a flowchart showing a method for extracting a determined part of a facial appearance impression and a method for extracting a determinant of a facial appearance impression according to the present invention. 9 is a flowchart illustrating a process of measuring a physical quantity from a moving image. It is a functional block diagram of a face impression analysis device concerning the present invention. 4 is a flowchart illustrating a method for distinguishing a face appearance impression by the face impression analysis device according to the present invention. It is a flowchart which shows the calculation process of the visual evaluation value / physiological quantity by the face impression analysis device which concerns on this invention. FIG. 9 is a diagram illustrating an example of a moving image / continuous still image used for acquiring a visual evaluation value. (Drawing substitute photo) FIG. 9 is a diagram illustrating a method of presenting a moving image / continuous still image to an evaluator when acquiring a visual evaluation value. (Drawing substitute photo) It is a figure showing the total result of the visual evaluation value about a moving picture / continuous still picture. FIG. 14 is a diagram illustrating an example of a result of eye gaze evaluation on a moving image / continuous still image. (Drawing substitute photo) FIG. 9 is a diagram illustrating a change in the expression of an evaluation target in a moving image. (Drawing substitute photo) FIG. 9 is a diagram illustrating a temporal change of a moving speed of a marker in a moving image. FIG. 10 is a diagram illustrating an integrated value of a moving speed of a marker in a moving image. FIG. 3 is a diagram showing an arrangement of markers for motion capture. FIG. 9 is a diagram illustrating a change in the expression of an evaluation target in a moving image. (Drawing substitute photo) It is a figure which shows the time change of the amount of change per unit time of the distance between the markers in a moving image. FIG. 7 is a diagram showing a region setting to be analyzed for a change in area over time. It is a graph which shows the measurement result of the expansion-contraction rate of the skin in the analysis target area. It is a figure which shows the setting of a grid point, and the example of a motion vector. (Drawing substitute photo) FIG. 9 is a diagram illustrating a change in the expression of an evaluation target in a moving image. (Drawing substitute photo) It is a figure showing the measurement result of a time change of the size of a motion vector. FIG. 7 is a diagram illustrating an average value of a width of a waveform of a motion vector in a moving image.

[1] Method of Extracting Determining Part of Determined Face Appearance Impression and Method of Extracting Determinant of Face Appearance Impression According to the present invention, a moving image including a change in facial expression is presented to an evaluator, and the appearance of the face is evaluated. A method of evaluating the impression and analyzing the line of sight of the evaluator to extract the determined part of the facial appearance impression, and measuring the physical quantity from the moving image and correlating it with the facial appearance impression This is a method for extracting a determinant of a facial appearance impression, which extracts a factor having a high degree of appearance.

  The “face impression” of the present invention is an impression that is visually received when another person views the face to be evaluated. For example, focusing on the age impression, there are impressions such as youthfulness that looks younger than the actual age and aging impressions that look old. In addition, there are various impressions such as attractiveness felt when viewing the evaluation target.

  FIG. 1 is a flowchart showing a method for extracting a determined portion of a facial appearance impression and a method for extracting a determinant of a facial appearance impression according to the present invention. With reference to this, the method for extracting the determinant and determinant of the facial appearance impression of the present invention will be described in detail.

<Acquisition of moving image>
First, in step S11, a moving image of a face to be evaluated is acquired. Here, as described above, a moving image including a change in facial expression is acquired.

  Here, a video obtained by capturing a moving image of a face to be evaluated with a general camera device may be used, but it is preferable that the video camera has a resolution enough to withstand image analysis in a later physical quantity measurement step.

  In general, a moving image is composed of a continuation of a large number of still images (frames), and the smoothness of the movement is represented by a frame rate representing the number of frames per unit time. Here, when presenting a moving image to the evaluator in a later visual evaluation value acquisition step, the evaluator sets a frame rate that is higher than the frame rate at which the evaluator can perceive it as a moving image rather than a mere continuation of still images. It is preferable to acquire a moving image having the same.

  Alternatively, a stereo moving image may be captured using a stereo camera. With such a configuration, a three-dimensional analysis can be performed in a physical quantity measurement step described later.

  More specifically, a motion that tracks the three-dimensional position of each marker by attaching a plurality of markers to the face to be evaluated and then capturing a moving image of the face to be evaluated by a stereo camera. Capture can be performed.

<Acquisition of visual evaluation value>
In step S12, a visual evaluation value for the moving image is obtained. This is performed by presenting the moving image acquired in step S11 to the evaluator, and presenting the impression that the evaluator has received from the moving image.

  For example, if the purpose is to extract a determined part and a determinant of an age impression from a facial appearance impression, a moving image including a facial expression change of the evaluation target face is presented to the evaluator, and the evaluation is performed. There is a method of obtaining an estimate of the age of the target (finding the answer of the estimated age). As another example, if the purpose is to extract the determinant of the facial impression of a face related to attractiveness, a moving image including a change in the facial expression of the face to be evaluated is presented to the evaluator as in the case of the age impression. After that, there is a method of obtaining an answer as to whether or not the evaluation target is attractive. The visual evaluation value obtained in this manner can be said to be a value directly indicating the visual impression of the face to be evaluated.

  In addition, one frame before the facial expression change and one frame after the facial expression change are extracted as still images from the moving image, and only those two frames are presented to the evaluator as a continuous still image. A step for prompting the evaluation of the impression may be provided. As a result, the impression of the face to be evaluated given by the evaluator given by the continuous still images as the static stimulus is obtained in comparison with the impression of the face to be evaluated given to the evaluator by the moving image as the dynamic stimulus. The analysis for extracting the determined part and the impression determinant can be performed in more detail.

  When presenting the moving image to the evaluator, the evaluator obtains gaze information of the evaluator, and obtains information such as how many times and how many seconds the gaze has stayed in which area of the evaluation target face. .

<Extraction of impression decision site>
In step S13, the gaze information of the evaluator when the moving image is presented to the evaluator is analyzed, and an impression determination site is extracted.

  When an evaluator who has imposed a certain impression evaluates a moving image, the evaluator judges the impression regardless of consciously or unconsciously in the area where the gaze stays many times and for a long time. It can be said that the image includes a region to be watched when performing, that is, a portion where the impression is determined.

  For example, when the evaluator is informed that the estimated age of the target displayed in the moving image is required to be answered, and when the moving image is presented, the area in which the evaluator's line of sight stays for a number of times for a long time is an age impression. Contains the decision site.

  In step S13, as described above, the area where the evaluator's line of sight stays during the visual evaluation and the surrounding area are extracted as the determined part of the visual impression.

<Measurement of physical quantity>
In a succeeding step S14, a moving image is analyzed and a physical quantity is measured. This may be performed only for the impression determination site specified in step S13, or may be performed for the entire region included in the moving image. FIG. 2A is a flowchart illustrating a physical quantity calculation process using a skin motion vector obtained by analyzing a two-dimensional moving image by an optical flow method.

  First, in step S21, feature points are extracted from each frame in the moving image. Here, a method of setting the feature points by determining the unevenness of the skin from the luminance value of each pixel in the frame or a method of setting the feature points in a grid in a moving image, that is, the feature points in a grid shape Any method, such as a method of arranging, may be used.

  Then, in step S22, the motion vector of each feature point is specified. The motion vector can be obtained from a change in luminance between frames.

  Then, in step S23, the motion vector is analyzed, and the physical quantity generated due to the change in the facial expression is calculated. Examples of the physical quantity here include the displacement amount of the coordinate of the feature point described above, the moving speed, the acceleration, the moving direction, and the like, and the wave property parameter of the plane motion, for example, the period, the perimeter, etc. Quantity. Further, a configuration for calculating an optical amount such as luminance or color tone, a change amount thereof, a frequency characteristic, or the like, or a configuration for performing analysis by a particle image flow velocity measurement method (PIV, Particle Image Velocity) or another method may be employed. .

  Each of these physical quantities may be obtained as a value in each frame, or may be obtained as an average value over the entire moving image. Alternatively, an area such as a cheek or a jaw may be set, and an integrated value or an average value of the values in the area may be obtained.

  As another method of measuring the physical quantity, a flowchart of FIG. 2B shows a calculation process of the physical quantity by tracking the coordinates of the feature points.

  First, in step S31, similar to step S21, a feature point is calculated by an arbitrary method. Alternatively, at the time of capturing a moving image in step S11, a marker or the like may be arranged in advance on the face to be captured, and these may be used as feature points.

  Then, in step S32, the coordinates of each feature point in each frame in the moving image are calculated.

  Then, in step S33, the change of the coordinates of the feature point between the frames is tracked, and the physical quantity as described in step S23 is calculated. For example, a time change of the speed or acceleration of each feature point. Further, a feature point serving as a reference point may be set, and a time change of a distance between the feature point and another arbitrary feature point may be obtained. This is a value corresponding to the relative speed of an arbitrary feature point on the face. Still another physical quantity is a temporal change of an area surrounded by three or more feature points. This makes it possible to quantify the manner of skin expansion and contraction at an arbitrary part on the face.

  Further, as described above, when a stereo moving image is obtained in step S11, it is possible to perform a three-dimensional analysis taking into account not only the plane but also the depth. That is, in step S32, the coordinates of the feature points in the three-dimensional space can be calculated by an arbitrary method such as the stereo method, and in step S33, the change in the coordinates of the feature points in the three-dimensional space can be tracked. This makes it possible to more accurately analyze the three-dimensional movement of the skin caused by the change in facial expression.

  The measurement of the physical quantity using the above-described motion vector and the measurement of the physical quantity by tracking the coordinates of the feature points may be configured to perform one or both of them. Furthermore, a moving image analysis may be performed by a method different from these methods to measure a physical quantity.

<Measurement of physiological quantity>
In step S15, a physiological quantity related to the skin of the face to be a moving image is measured.
Examples of the physiological amount relating to the skin include physical properties and / or internal physical properties of the skin, the composition of the skin tissue, and the like, and those known to be correlated with the facial impression can be used. For example, when the appearance impression of the face is an age impression, a physiological quantity related to skin whose mechanism of participation in aging is clarified can be used. Such physiological quantities include physical properties of the skin surface, for example, skin surface roughness, physical properties of the stratum corneum, such as the water content of the stratum corneum, physical properties of the dermis, such as the amount and structure of collagen fiber bundles, and physical properties of the fat layer, such as fat The physical properties of the nipple layer or the nipple layer, such as the position and size of the body, such as the number and height of the nipple structure per unit area, may be mentioned. The physiological quantity is preferably a physiological quantity of the impression determining site.
The measurement of the physiological amount can be performed by a conventionally known method. In the present invention, the method is limited to the case where the method is performed by a non-invasive method (a method without surgery).

  Steps S11 to S15 are performed on a plurality of different faces, which are sufficient for performing the next correlation analysis.

<Extraction of appearance impression determinant>
In step S16, a determinant of the facial appearance impression is extracted from the physical quantity acquired from the moving image in step S14.

  Here, the correlation between the physical quantity acquired in step S14 and the visual evaluation value acquired in step S12 or the physiological quantity measured in step S15 is analyzed, and the visual evaluation value or the physiological quantity is calculated from the physical quantities. Are extracted as factors that determine.

For example, a plurality of moving images of evaluation targets having different visual evaluation values are compared, and among physical quantities measured from the moving images, those having a difference between the two are represented by a correlation with the visual evaluation value. It is expected to have. By performing a similar comparison on still another moving image, the correlation between the visual evaluation value and the physical quantity that is a determinant of the visual impression, and the correlation between the physical quantity and the visual evaluation value are clarified. Can be
The analysis of the correlation can be performed using a conventionally known method.

  Also, as described above, when acquiring the visual evaluation value in step S12, if the visual evaluation value obtained when the two still images before and after the facial expression change are presented is acquired, the corresponding moving image is acquired. May be compared with a visual evaluation value. For example, if the visual evaluation value for a still image of the same evaluation target is significantly different from the visual evaluation value for a moving image, the physical quantity included in the moving image is a determinant of the visual impression of the face. Can be estimated.

  Similarly, by analyzing the correlation between the physical quantity obtained from the moving image and the physiological quantity relating to the skin, it is possible to determine a physical quantity having a high correlation with the physiological quantity relating to the skin, and the physiological quantity is determined by the face value. Can be presumed to be a determinant of the visual impression of.

  As described above, according to the present invention, by presenting a moving image including a facial expression change to an evaluator, by analyzing the line of sight of the evaluator, a part that determines the appearance impression of the face, By extracting more physical quantities and analyzing them, it is possible to extract determinants of the appearance impression of the face.

  The acquisition of the visual evaluation value and the analysis of the correlation with the physical quantity, and the measurement of the physiological quantity and the analysis of the correlation with the physical quantity may be performed on both of them, or only one of them may be performed as necessary. May go. Further, the correlation between the evaluation value and the physical quantity measured from the moving image may be analyzed using different evaluation values and quantities.

[2] Method of Differentiating Facial Appearance Impression of the Present Invention and Face Impression Analysis Apparatus The present invention measures a physical quantity by acquiring a moving image of an impression determining part of a subject's face including a facial expression change and analyzing the moving image. Then, a method for discriminating the visual impression of the face of the subject and a facial visual impression analyzing apparatus are provided.

  As shown in a functional block diagram in FIG. 3, the facial impression analysis device according to the present invention includes a moving image acquisition unit 1, a physical quantity measurement unit 2, a correlation storage unit 3, a visual evaluation value calculation unit 4, a physiology, Target calculation means 5.

  Further, the moving image acquiring means 1 has a stereo moving image acquiring means 11, and the physical quantity measuring means 2 has a feature point extracting means 21, a motion vector specifying means 22, a motion vector analyzing means 23, and a feature point coordinate calculation. Means 24 and feature point tracking means 25.

  The correlation storage unit 3 analyzes the expression indicating the correlation obtained by analyzing the physical quantity and the visual evaluation value, and analyzes the physical quantity and the physiological quantity in the process of extracting the determinant of the facial appearance impression. Information such as an expression indicating the obtained correlation is held.

  The face impression analysis device according to the present invention uses a general-purpose computer device including an arithmetic device, a main storage device, an auxiliary storage device, and various input / output devices. This is realized by storing a facial impression analysis program for causing the computer device to function as each of the above-described units. When causing the computer device to function as a facial impression analysis device, the facial impression analysis program is loaded on the main storage device as needed, and the processing by the arithmetic device is performed.

  In addition, an arbitrary configuration such as a configuration in which a camera device is used as the moving image acquisition unit 1 and a moving image obtained thereby is passed to the physical quantity measurement unit 2 or a configuration in which the correlation storage unit 3 is realized by a general-purpose server device. A plurality of devices may be configured to be able to communicate via a USB (Universal Serial Bus) connection, data transfer using a recording medium, or a network such as the Internet or a LAN (Local Area Network).

<Acquisition of moving image>
FIG. 4 is a flowchart illustrating a method of discriminating the appearance impression of the face of the subject by the face impression analysis device. First, in step S41, using the moving image acquisition means 1, a moving image including a change in the facial expression of the subject is acquired. Note that the stereo moving image acquisition unit 11 may acquire a stereo moving image including a change in the facial expression of the subject. At this time, a moving image in which a marker to be used as a feature point in the measurement of a physical quantity later is set on the face of the subject may be acquired.

<Measurement of physical quantity>
Then, in step S42, the physical quantity is measured using the physical quantity measuring means 2. Here, in the same processing as the contents described above with reference to FIG. 2, the calculation of the physical quantity extracted as the determinant of the face appearance impression is performed by the above-described method of extracting the determinant of the face appearance impression. Do.

  When calculating a physical quantity using the motion vector shown in FIG. 2A, the feature point extraction processing in step S21 is performed by the feature point extraction means 21, and the motion vector specification processing in step S22 is performed. After performing the processing in step S22, the motion vector analyzing unit 23 performs the motion vector analysis processing in step S23.

  When calculating the physical quantity using the feature point coordinates shown in FIG. 2B, the feature point extraction processing of step S31 is performed by the feature point extraction means 21, and the calculation processing of the feature point coordinates of step S32 is performed. After performing by the coordinate calculating means 24, the feature point tracking means 25 performs the tracking processing of the feature point coordinates in step S33.

  If a stereo moving image has been obtained in step S41, the feature point coordinates in the three-dimensional space are calculated in an arbitrary method such as the stereo method in step S32, and the feature point coordinates in the three-dimensional space are calculated in step S33. Tracking of coordinate changes may be performed.

  When calculating the physical quantity using both the measurement of the physical quantity using the motion vector and the measurement of the physical quantity using the feature point coordinates, either of the feature point coordinate calculation processing in step S21 and step S31 is performed. A configuration in which feature points are shared by performing only one of them may be adopted, or a configuration in which steps S21 and S31 are executed together to extract different feature points in order to effectively measure a physical quantity in each method. Good.

<Analysis of facial impression>
In step S43, the physical quantity measured in step S42 is analyzed, and the appearance impression of the face is discriminated. Thereby, at least one of the visual evaluation value of the visual impression of the subject's face and the physiological amount related to the skin of the subject's face is calculated.

  FIG. 5A is a flowchart showing a process of calculating the visual evaluation value by the visual evaluation value calculation means 4. First, in step S51, correlation data between a physical quantity and a visual evaluation value is acquired from the correlation storage unit 3.

  Then, in step S52, a visual evaluation value is calculated based on the physical quantity acquired in step S42 and the correlation data acquired in step S51. From the calculated visual evaluation value, the appearance impression of the subject's face can be distinguished.

  FIG. 5B is a flowchart showing a process of calculating a physiological amount by the physiological amount calculating means 5. This is the same process as the above-described visual evaluation value calculation process. First, in step S61, correlation data between a physical quantity and a physiological quantity is acquired from the correlation storage unit 3.

  Then, in step S62, a physiological quantity is calculated based on the physical quantity acquired in step S42 and the correlation data acquired in step S61. From the calculated physiological quantity, it is possible to determine the appearance impression of the subject's face.

  As described above, the physical quantity is measured by analyzing the moving image including the facial expression change of the subject, and from the physical quantity of the subject, based on the correlation between the physical quantity and the visual evaluation value or the physiological quantity, , A visual evaluation value and / or a physiological amount of the subject can be calculated, and the visual impression of the subject's face can be discriminated from the calculated value.

  Note that if a stereo moving image is acquired by using the stereo moving image acquiring unit 11, three-dimensional analysis can be performed by the physical quantity measuring unit 2. However, the stereo moving image acquiring unit 11 is a general monocular camera. It is more expensive than such. Therefore, the stereo moving image acquiring unit 11 may be omitted, a general monocular camera or the like may be provided as the moving image acquiring unit 1, and the physical quantity measuring unit 2 may perform only two-dimensional analysis. With such a configuration, the face impression analysis device can be realized at low cost.

  In addition, as a method for distinguishing a visual impression of a face and a method for using the facial impression analysis device according to the present invention, a method for distinguishing a cosmetic effect is exemplified. In other words, by comparing the analysis result of the visual impression performed using the moving image of the face before makeup of the subject and the analysis of the visual impression performed using the moving image of the face after makeup of the subject, It is to discriminate the effect of makeup on the visual impression.

  Hereinafter, the present invention will be described in more detail with reference to examples of acquiring a visual evaluation value of a face appearance age impression, extracting an impression determination site, and measuring a physical quantity. It goes without saying that the invention is not limited to the examples.

[1] Obtaining Visual Evaluation Value from Moving Image and Continuous Still Image <Acquiring Moving Image>
For 20 evaluation subjects (ages 20 to 60), a 9-second moving image including a change in facial expression was photographed. Then, from the moving image, one frame before the facial expression change and one frame after the facial expression change were both extracted as still images, and a continuous still image in which only those two frames were displayed for 9 seconds was created. FIG. 6A is a schematic diagram of a moving image, and FIG. 6B is a schematic diagram of a continuous still image.

<Acquisition of visual evaluation value>
The above moving image or continuous still image was randomly displayed to 15 evaluators using a monitor, and a visual evaluation value of the age impression of the face was obtained. When presenting a moving image and a continuous still image to the evaluator, use a line-of-sight analyzer to analyze how many times and how many seconds the evaluator's line of sight stays in which region of the moving image and still image. Was. FIG. 7 shows a schematic diagram of a method of presenting a moving image and a still image to an evaluator.

  First, as shown in FIG. 7A, a number is to be displayed, then a face to be evaluated is displayed, and an instruction for asking for an answer of the first half or the second half of the generation of numbers is presented. As a result, when viewing a moving image or a continuous still image displayed thereafter, the evaluator is conscious of the discrimination regarding the age impression to be evaluated.

  Next, as shown in FIG. 7 (b), numerals were displayed. This number is obtained by cutting off the ones place of the actual age of the evaluation target in the moving image or the continuous still images thereafter. Since the age of the subject in the present example is in the 20s to 60s as described above, any one of 20, 30, 40, 50, and 60 is displayed.

  Then, as shown in FIG. 7C, a moving image or a continuous still image for 9 seconds is displayed, and then, as shown in FIG. 7D, the evaluation target displayed in FIG. An answer was asked for the first half or second half of the generation of the number displayed in 7 (b).

  That is, in the example shown in FIG. 7, the age of the subject to be evaluated is any one between the ages of 30 and 39, and the evaluator replies to it in the early thirties or late thirties.

  In the evaluator's response obtained by the above-described procedure, when the evaluator evaluates that it is the latter half of the generation, that is, it can be determined that the evaluation target has an aging impression.

  FIG. 8 summarizes the visual evaluation values obtained from the evaluator for all evaluation targets. As shown here, it can be read that the moving image gave a higher age impression to the evaluator than the continuous still image. From this, it can be understood that the movement during the facial expression change included in the moving image includes a factor that gives a high age impression to the evaluator.

<Extraction of impression decision site>
FIG. 9 shows the gaze analysis result of the evaluator when displaying a moving image and a continuous still image for a certain evaluation target. FIG. 9 (a) shows a point where the evaluator's line of sight is often retained when displaying a moving image to be evaluated and FIG. 9 (b) when displaying a continuous still image which is to be evaluated. It is.

  Looking at these eye-gaze analysis results, it can be seen that the evaluator paid attention to the area around the cheek and eyes to be evaluated, regardless of whether a moving image or a continuous still image was displayed. From this, it was found that the determined parts of the facial age impression were the cheeks and the periphery of the eyes.

  In addition, despite the fact that the age impression decision site is the same for both the moving image and the continuous still image, the age impression given by the moving image is higher. It was found that the physical quantity caused by the movement when the facial expression changed in the part.

[2] Measurement of physical quantity by motion capture <Acquisition of moving image>
With 30 women in their 20s to 60s as evaluation targets, a reflective marker is pasted on the face as a marker to indicate measurement points, and multiple cameras are used to shoot moving images including facial expression changes of each evaluation target. went. Then, by analyzing the moving image, a motion capture for tracking a change in the three-dimensional coordinates of the marker was performed. In addition, a moving image was presented to the evaluator, and a visual evaluation value of the age of each evaluation target was obtained.

  Here, the face is changed from a straight face to a state in which the mouth is widely opened as shown in FIG. 10A, and after taking the expression with the mouth closed as shown in FIG. A moving image including a series of facial expression changes was used for motion capture and acquisition of a visual evaluation value.

  Then, as shown below, each of the marker movement position, the movement amount, the movement speed, the relationship between the movements of the two markers, and the time change of the area surrounded by the three or more markers is analyzed to extract the physical quantity. did.

<Measurement of marker moving speed>
The movement of the marker on the face to be evaluated was tracked, and the movement speed was measured. As a result, a result as shown in FIG. 11 was obtained.

  FIG. 11A shows an evaluation target whose visual evaluation value of age was higher than the actual age, that is, an evaluation target having a high apparent age impression. FIG. 11B shows that the visual evaluation value of the age was higher than the actual age. 6 is a graph showing the time change of the moving speed of the marker in each of the evaluation targets having a low evaluation, that is, the evaluation targets having a low apparent age impression. In this case, the vertical axis represents the moving speed of the marker, and the horizontal axis represents the number of frames in the moving image as a time element. Also, when the facial expression changes from a true face state to a facial expression in which the mouth is greatly opened as shown in FIG. 10A, the speed direction of the marker is the speed of the extension direction, and the mouth as shown in FIG. The moving speed of the marker when the expression changes to a closed expression is shown as the speed in the contraction direction.

  As shown here, the graph of the speed change of the feature point of the evaluation target that looks old has a sharper peak than that of the evaluation target that looks young, resulting in faster convergence of the speed. That is, it can be said that the change in the moving speed of the feature point of the evaluation target that looks old is more intense than that of the evaluation target that looks young.

  FIG. 12 shows the moving speed of the characteristic point of the evaluation target that looks old and the moving speed of the characteristic point of the evaluation target that looks young shown in FIG. Is a graph in which the area of each graph is calculated.

  In FIG. 12, the change in the moving speed of the feature point of the evaluation object that looks young is slower than that of the evaluation object that looks old because the evaluation area that looks young has a larger graph area. Can be read.

  From this result, it was found that a feature point, that is, a supple movement in which the movement of the skin slowly follows the change of the facial expression, may give the evaluator an impression that the evaluator looks young.

<Relationship between the movements of the two markers>
FIG. 13 is a diagram illustrating markers set on the left side of the face to be evaluated. As shown here, two markers M1, a marker M2 on the cheek, and a marker M3 located on the chin were set with the marker on the forehead as the reference point M0.

  Then, the distance L1 between the reference point M0 and the marker M1 at the time of the change from the true face state as shown in FIG. 14A to the facial expression with the mouth wide open as shown in FIG. , A change amount V2 per unit time of a distance L2 between the reference point M0 and the marker M2, and a change amount per unit time of a distance L3 between the reference point M0 and the marker M3. The amount of time V3 and the amount of change in time were measured. That is, it can be said that this is a temporal change in the relative speed of the markers M1, M2, and M3 with respect to the reference point M0.

  As a result of the above measurement, a result as shown in the graph of FIG. 15 was obtained. FIG. 15A is a graph showing the measurement results of an evaluation target that has been evaluated to have a young appearance impression by visual evaluation. From this, it can be seen that although the absolute values of the maximum values of V1 to V3 are different, the time when each of them takes the maximum value is very close. On the other hand, as shown in FIG. 15 (b), when the measurement result of the evaluation target evaluated to have an old visual impression by visual evaluation is observed, after V3 takes the maximum value, V1 and V2 are delayed with the maximum value. Was taken.

  In other words, it can be said that in a change in the expression of the evaluation target having an old-looking impression, the movement of the cheek represented by the markers M1 and M2 is later than the movement of the jaw represented by the marker M3.

  From these results, it has been found that when the followability of the movement of the cheek during the change of the facial expression is good, there is a possibility that the evaluator may be given a young-looking impression.

<Time change of the area surrounded by three or more markers>
As shown in FIG. 16, two regions were set: a jaw A1 surrounded by three markers and a cheek A2 surrounded by four markers. Then, when the time change of the area of each region at the time of the facial expression change is measured, and the result obtained by integrating these values with time is calculated as the skin expansion / contraction ratio in each region, the result as shown in the graph of FIG. Obtained.

  FIG. 17A is a graph showing the measurement results of an evaluation target that has been evaluated to have a young appearance impression by visual evaluation. FIG. 17B is a graph showing the measurement results of the evaluation target evaluated to have an old-looking impression by visual evaluation. When these are compared, the evaluation target having a younger appearance impression has a higher expansion / contraction ratio of the cheek A2 than the jaw A1, whereas the evaluation target having the older appearance impression has the cheek A2 more than the jaw A1. Was low in elasticity.

  From these results, it was found that when the skin of the cheek A2 flexibly expands and contracts as compared with the jaw A1, there is a possibility that an evaluator may be given a young-looking impression.

[3] Measurement of physical quantity by optical flow method <Acquisition of moving image>
For each of the evaluation targets of 80 women in their twenties to sixties, two-dimensional moving images including changes in facial expressions were taken. Then, grid-like grid points were set in the moving image, and a skin motion vector at each grid point in each frame was determined from the change in shading between the frames in the moving image, as shown in FIG. In addition, a moving image was presented to the evaluator, and a visual evaluation value of the age of each evaluation target was obtained.

  Note that, for analysis of the moving image and acquisition of the visual evaluation value, the facial expression with the mouth open in FIG. A moving image that changes to a natural expression and returns to a true face was used.

<Change of motion vector>
As a result of analyzing the magnitude of the motion vector during the change of the facial expression, that is, the speed of the movement of the skin, for the cheek region, a graph as shown in FIG. 20 was obtained. This shows the temporal change of the speed of the movement of the skin, with the vertical axis representing the average value of the magnitude of the motion vector in each frame and the horizontal axis representing the number of frames, that is, the time axis. Looking at this, compared to the measurement result of the evaluation target that was evaluated to have a young appearance impression by visual evaluation indicated by a solid line in the graph, it was evaluated to have an old appearance impression by the visual evaluation indicated by a broken line in the graph. The measurement result of the evaluation object obtained has a shape having a sharp peak, particularly when the expression changes as shown in FIG.

  FIG. 21 is a graph showing the result of calculating the time width of the movement of the skin of the evaluation target that looks old and the speed of the movement of the skin of the evaluation target that looks young shown in FIG. 20. Looking at this, similar to the skin physical properties indicated by the measurement results of the speed of movement of the marker at the time of the facial expression change by the motion capture moving image described earlier with reference to FIG. It can be read that the graph value is large and the change in the movement is gradual.

  In other words, instead of analysis using 3D video motion capture, which requires physically placing markers on the face to be evaluated and shooting using multiple cameras, a 2D image that can be easily shot It was suggested that similar analysis results could be obtained by analyzing the moving images by the optical flow method.

  INDUSTRIAL APPLICABILITY The present invention can be used for easily and accurately discriminating a customer's apparent age impression and the like in a case of cosmetics sales, counseling, aesthetics, and the like.

REFERENCE SIGNS LIST 1 moving image acquiring means 11 stereo moving image acquiring means 2 physical quantity measuring means 21 feature point extracting means 22 motion vector specifying means 23 motion vector analyzing means 24 feature point coordinate calculating means 25 feature point tracking means 3 correlation storage means 4 visual evaluation value Calculation means 5 Physiological quantity calculation means

Claims (11)

A visual evaluation step of presenting dynamic information of the facial expression change to the evaluator, and causing the evaluator to evaluate the visual impression of the face;
In order to determine an impression determining part that determines the appearance impression of the face, the line of sight data of the evaluator who views the dynamic information is analyzed, and an area in which the evaluator gazes in the dynamic information is analyzed. Gaze analysis process,
In the impression determination site, a physical quantity measurement step of measuring a physical quantity of a skin change caused by a facial expression change,
From among the physical quantity, and extracts a high factor correlated with visual impression of the face, characterized in that it contains a factor extraction step, the extraction method of the determinants of visual impression of the face. Wherein the visual impression of the face is age impression extraction method determinant visual impression of the face according to claim 1. The physical quantity measuring step, from the face to the motion capture moving images captured by setting a plurality of markers, characterized in that it comprises a step of measuring a quantity related to the markers as the physical quantity, according to claim 1 or claim 3. The method for extracting a determinant of a visual impression of a face according to 2 . The amount related to the marker, the position, the movement amount of the marker, the speed magnitude, the speed of direction, the acceleration magnitude, of the direction of acceleration, characterized in that it comprises at least one, according to claim 3 Method of Extracting Determinants of Appearance Impression of the Face of a Person. The method according to claim 3 , wherein the amount relating to the marker includes an amount relating to a temporal change in a distance between the plurality of markers. 6. The method according to claim 3, wherein the amount related to the marker includes an amount related to a time change of a value indicating an area and / or a shape of a surface formed by three or more of the markers. 7. A method for extracting a determinant of the appearance impression of the described face. The physical quantity measurement step has a grid point setting step of setting a plurality of grid points on the face,
The method according to any one of claims 3 to 6 , wherein the physical quantity includes a quantity related to the grid point. 8. The method according to claim 7 , wherein the quantity related to the grid points includes a quantity related to a temporal change in the magnitude and / or direction of the velocity vector for each of the grid points. . A visual evaluation value acquisition step of presenting dynamic information including the facial expression change to the evaluator to obtain a visual evaluation value of a visual impression of the face,
9. The face appearance according to claim 1 , wherein the factor extracting step includes analyzing the correlation between the physical quantity and the visual evaluation value to extract the factor. How to extract determinants of impression. Measuring a physiological quantity related to the skin, comprising a physiological quantity measuring step,
The appearance of the face according to any one of claims 1 to 9 , wherein the factor extraction step includes analyzing a correlation between the physical quantity and the physiological quantity to extract the factor. How to extract determinants of impression. The method according to claim 10 , wherein the physiological quantity is a physical property of the skin surface and / or an internal property of the impression determining part.