JP2015005281A - Face image analysis method and face image analyzer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To use a face image to accurately analyze an evaluation index of the whole face.SOLUTION: A face image analysis method includes steps of: separating a shape-standardized face image into a low-frequency component image excluding high frequency components and a high-frequency component image including the high frequency components; acquiring multidimensional base vectors to be obtained from multiple sample face images, and acquiring multiple weighting factors for the base vectors concerning a target face image through a principal component analysis to the low-frequency and high-frequency component images separated from the target face image included in the face image; and selecting multiple analysis target weighting factors in at least some analysis target base dimensions out of base dimensions owned by the base vectors, from the acquired weighting factors, and analyzing the target face image using the selected analysis target weighting factors for the target face image and a face image index equation calculated through a multiple regression analysis using the analysis target weighting factors and multiple index values concerning the sample face images.

Description

  The present invention relates to a technique for analyzing a face image.

  By performing image analysis on a face image or a part of a face, evaluation of the skin state, the appearance age of the face, etc., and a face image operation on a certain target face image are performed.

  For example, Patent Document 1 described below contributes to the stereoscopic effect of the face image and the brightness of the skin color by performing principal component analysis (PCA (Principal Component Analysis)) and multiple regression analysis on the standardized face image. A method is proposed in which a base is extracted and a face image of a certain target is generated by changing a coefficient of the extracted base. Further, Patent Document 2 below extracts a melanin component image and a hemoglobin component image by performing independent component analysis (ICA (Independent Component Analysis)) on a polarized image of a part of the face, and the histogram distribution thereof. Based on the above, we propose a method for generating a simulation image of a subject in which only the texture is close to the target image. Non-patent document 1 will be described later as a reference document.

JP 2008-276405 A JP 2009-82736 A

Norimichi Tsumura, etc., "Image-based skin color and texture analysis / synthesis by extracting hemoglobin and melanin information in the skin", acm Transactions on Graphics, Vol.22, No.3.pp.770-779 (2003), Proceedings of ACM SIGGRAPH 2003

  In the face analysis method using the partial skin image of the face as in Patent Document 2 described above, since only partial information is used, it is not suitable for obtaining some evaluation index for the entire face.

  On the other hand, PCA is often used for the feature analysis and manipulation of face images, as in Patent Document 1 above. Since PCA obtains an eigen component (base) having a large variance, it can efficiently acquire and manipulate facial features common to a plurality of people. However, since PCA is not an analysis considering local spatial frequency, various base components having different spatial frequencies are mixed in the base, and the base has a global characteristic in terms of spatial frequency. Factors that affect the evaluation index of the entire face, such as color unevenness component, uneven unevenness component, and glossy component, give different evaluation indexes to the entire face depending on its size, spatial frequency, position, etc. . Therefore, in the face image feature analysis and operation using PCA, the local spatial frequency is buried, and the evaluation index of the face image may not be analyzed accurately.

  The present invention has been made in view of such problems, and provides a technique for analyzing an evaluation index of the entire face with high accuracy using a face image. Here, the evaluation index for the entire face includes a quantitative index such as skin age and the like, and an index of the appearance impression of the entire face such as appearance age, youthfulness, and skin feeling.

  Each aspect of the present invention employs the following configurations in order to solve the above-described problems.

  The face image analysis method according to the first aspect includes a frequency component separation step of separating a face image whose shape is standardized into a low frequency component image from which a high frequency component has been removed and a high frequency component image including only the high frequency component; By obtaining a multi-order basis vector obtained from a plurality of sample face images and performing principal component analysis on each of the low-frequency component image and the high-frequency component image separated from the target face image included in the face image by the frequency component separation step. A target coefficient acquisition step for acquiring a plurality of weight coefficients of the base vector for the target face image, and a plurality of weight coefficients acquired by the target coefficient acquisition step, at least a part of the basis orders of the base vector A plurality of analysis target weight coefficients in the analysis target base order are selected, a plurality of analysis target weight coefficients for the selected target face image, and a plurality of samples are selected. By using the face image index formula calculated by multiple regression analysis using a plurality of analytes weighting coefficients and the plurality of index values regarding Le face image, it comprises an analysis step of analyzing a target face image.

  As another aspect of the present invention, a face image analysis apparatus that executes the face image analysis method according to the first aspect may be used, or the face image analysis method according to the first aspect may be performed by at least one computer. The program may be executed by the computer, or may be a computer-readable storage medium storing such a program.

  According to each aspect described above, it is possible to provide a technique for analyzing an evaluation index of the entire face with high accuracy using a face image.

It is a figure which shows notionally the hardware structural example of the face image analyzer in 1st Embodiment. It is a flowchart which shows the operation example of the face image analyzer in 1st Embodiment. It is a figure which shows notionally the process structural example of the face image analyzer in 1st Embodiment. It is a flowchart which shows the example of the operation | movement which analyzes a sample face image in the face image analysis apparatus in 2nd Embodiment. It is a flowchart which shows the example of the operation | movement which analyzes the object face image in the face image analysis apparatus in 2nd Embodiment. It is a figure which shows notionally the process structural example of the face image analyzer in 2nd Embodiment. It is a flowchart which shows the example of the operation | movement which analyzes the object face image in the face image analysis apparatus in 3rd Embodiment. It is a figure which shows notionally the process structural example of the face image analyzer in 3rd Embodiment. It is a figure which shows the concept of the frequency component isolation | separation process using a two-dimensional discrete wavelet transform. It is a figure which shows the result of 3 steps | paragraphs of two-dimensional discrete wavelet transforms (multi-resolution analysis). It is a figure which shows the result of 2 steps | paragraphs of two-dimensional discrete wavelet transforms (multiresolution analysis). It is a figure which shows the result of the principal component analysis with respect to a low frequency component image. It is a figure which shows the result of the principal component analysis with respect to a high frequency component image. It is a figure which shows the result of having modulated only the low frequency component image of a melanin pigment image. It is a figure which shows the result of having modulated only the high frequency component image of a melanin pigment image. 14 is a flowchart illustrating an example of an operation of analyzing a sample face image of the face image analysis apparatus according to the second embodiment. It is a figure which shows the concept of the two-dimensional discrete wavelet transform in Example 2. FIG. It is a figure which shows the concept of the frequency component separation process in Example 2. FIG. 10 is a flowchart illustrating an example of an operation of analyzing a target face image in the face image analysis apparatus according to the second embodiment. 10 is a diagram illustrating a part of the result of principal component analysis in Example 2. FIG. It is a figure which shows notionally the process structural example of the face image analyzer in Example 2. FIG.

  Embodiments of the present invention will be described below. In addition, each embodiment given below is an illustration, respectively, and this invention is not limited to the structure of each following embodiment.

[First Embodiment]
[Hardware configuration]
FIG. 1 is a diagram conceptually illustrating a hardware configuration example of a face image analysis apparatus (hereinafter simply referred to as an analysis apparatus) 10 according to the first embodiment. The analysis apparatus 10 in the first embodiment is a so-called computer, and includes a CPU (Central Processing Unit) 2, a memory 3, an input / output interface (I / F) 4, and the like. The memory 3 is a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk, a portable storage medium, or the like.

  The input / output I / F 4 is connected to the input unit 7, the output unit 9, a communication device that communicates with other computers and other devices, and the like. The input unit 7 is a device that receives input of a user operation such as a keyboard and a mouse. The output unit 9 is a device that provides information to a user, such as a display device or a printer. Note that the hardware configuration of the analyzer 10 is not limited.

[Operation example (Face image analysis method)]
FIG. 2 is a flowchart illustrating an operation example of the face image analysis apparatus 10 according to the first embodiment. Hereinafter, the face image analysis method according to the first embodiment will be described with reference to FIG.
The face image analysis method executed by the face image analysis apparatus 10 includes a frequency component separation step (S21), a target coefficient acquisition step (S22), and an analysis step (S23), as shown in FIG.

  The frequency component separation step (S21) separates the face image whose shape is standardized into a low-frequency component image from which the high-frequency component has been removed and a high-frequency component image that includes only the high-frequency component. For normalization of the face image, warping processing, morphing processing, or the like using affine transformation is performed. Details of the standardization of the face image will be described in the second and subsequent embodiments. In the first embodiment, the face image whose shape is standardized before the frequency component separation is performed may not be an internal reflection image as described later, and is separated by an independent component analysis as described later. It does not have to be a hemoglobin pigment image, a melanin pigment image, or a shadow image.

  The face image analysis apparatus 10 extracts a low-frequency component image and a high-frequency component image from the face image using a well-known image processing technique called frequency decomposition, multiresolution analysis, or the like. For example, the face image analysis apparatus 10 applies a filter that removes a predetermined high frequency component and a predetermined low frequency component to information on a spatial frequency component obtained by performing two-dimensional Fourier transform on the face image, and this filtered space. A low-frequency component image and a high-frequency component image can be acquired by performing inverse two-dimensional Fourier transform on the frequency component information. The face image analysis apparatus 10 can also extract a low-frequency component image and a high-frequency component image from the face image using multi-resolution analysis such as two-dimensional discrete wavelet transform. Details of the two-dimensional discrete wavelet transform will be described later. The present embodiment does not limit the method of extracting the low frequency component image and the high frequency component image from the face image. Furthermore, this embodiment does not limit the number of low-frequency component images and high-frequency component images that are separated.

  Here, the face image whose shape is standardized includes a face image to be analyzed (hereinafter also referred to as a target face image), a sample face image, and the like. The target face image may be included in the sample face image. Therefore, in (S21), the face image analysis apparatus 10 may separate the target face image whose shape is standardized into the low frequency component image and the high frequency component image, or a plurality of shapes whose shapes are standardized. The sample face image may be separated into the low-frequency component image and the high-frequency component image.

  The target coefficient acquisition step (S22) acquires a multi-order basis vector obtained from a plurality of sample face images, and the low frequency component separated in the frequency component separation step (S21) from the target face image whose shape is normalized. A plurality of weighting factors of the base vector regarding the target face image are acquired by principal component analysis (PCA) for each of the image and the high-frequency component image. Specifically, the face image analysis apparatus 10 acquires difference information between the average face of the plurality of sample face images and the target face image that is the basis of the plurality of basis vectors, and the difference information and the plurality of basis vectors. Can be obtained for the target face image. The plurality of weighting factors acquired here are a collection of weighting factors of each base order of the base vector. Therefore, the face image analyzing apparatus 10 acquires a plurality of weighting coefficients for each separation unit in (S21).

  The analysis step (S23) selects a plurality of analysis target weight coefficients from the plurality of weight coefficients acquired in (S22), and uses the selected plurality of analysis target weight coefficients and the face image index formula to Analyze the image. The analysis target weight coefficient means a weight coefficient in at least a part of the analysis target base orders among the base orders of the basis vectors. The analysis target weighting factor may be a full weighting factor of all base orders of the base vector, or may be a weighting factor of some base orders. Further, the face image index formula is a multiple regression formula calculated by multiple regression analysis using a plurality of analysis object weight coefficients and a plurality of index values obtained from a plurality of sample face images that are the basis of the basis vector. means. Details of the method of acquiring the face image index formula will be described in the second and subsequent embodiments.

  The face image analysis apparatus 10 may calculate an index value indicating the degree of fit of the target face image regarding the index used in the face image index formula as the analysis of the target face image, and based on the index, the target face image May be modulated. As described above, the index value may be a quantitative index value or a value obtained by quantifying the impression of the entire face. In the present embodiment, if the analysis is performed using the analysis target weighting coefficient of the target face image and the face image index formula, the analysis content of the target face image in the analysis process is not limited, and the evaluation index is not limited.

[Processing configuration]
FIG. 3 is a diagram conceptually illustrating a processing configuration example of the face image analysis apparatus 10 in the first embodiment. The face image analysis apparatus 10 according to the first embodiment includes a frequency component separation unit 12, a target coefficient acquisition unit 14, an analysis unit 18, and the like. Each of these processing units is realized, for example, by executing a program stored in the memory 3 by the CPU 2. Further, the program may be installed from a portable recording medium such as a CD (Compact Disc) or a memory card or another computer on the network via the input / output I / F 4 and stored in the memory 3. Good.

  The face image analyzing apparatus 10 can execute the face image analyzing method shown in FIG. 2 by using each processing unit shown in FIG. In this case, the frequency component separation unit 12 executes the above-described frequency component separation step (S21), the target coefficient acquisition unit 14 executes the above-described target coefficient acquisition step (S22), and the analysis unit 18 performs the above-described analysis. An analysis process (S23) is performed.

  Even if the face image analysis apparatus 10 holds the target face image and sample face image whose shape is standardized, and the plurality of basis vectors and the face image index formula in advance by the face image analysis apparatus 10 itself. Alternatively, it may be generated by the face image analyzing apparatus 10 itself, or may be acquired from another computer or a portable recording medium via the input / output I / F 4.

[Operation and Effect in First Embodiment]
As described above, in the first embodiment, the low-frequency component image from which the high-frequency component is removed and the high-frequency component image including only the high-frequency component are extracted from the target face image, and the low-frequency component image and the high-frequency component image are extracted. Then, principal component analysis (PCA) using a plurality of order basis vectors obtained from a plurality of sample face images is performed. As a result, the weighting coefficients of the multi-dimensional basis vectors related to the target face image are calculated for each spatial frequency component separation unit.

  Here, the low-frequency component included in the target face image can be defined as a wide range of pigment concentration distribution over the entire face or the entire facial region, and indicates a component related to the pigment concentration of the entire skin of the face. On the other hand, the high frequency component can be defined as a local pigment concentration distribution, and indicates a component related to a fine pigment concentration contained in the face. In the first embodiment, by performing PCA on each dye density distribution (high-frequency component image and low-frequency component image) having different characteristics as described above, the feature amount of each dye density distribution having different characteristics is acquired. Can do. Specifically, the basis vector obtained from the low-frequency component image represents the pigment unevenness of the entire face, and the basis vector obtained from the high-frequency component image represents the fine pigment unevenness of the face and individual characteristics. Therefore, according to the first embodiment, it is possible to use feature quantities (narrowband spatial frequency characteristics) that were difficult to consider in the prior art. Such an effect will be described in more detail in the section of the embodiment.

  In the first embodiment, a target face image is analyzed based on a plurality of analysis target weight coefficients selected from such a plurality of weight coefficients and a face image index formula based on a plurality of sample face images. . Therefore, according to the first embodiment, it is possible to perform face image analysis in a comprehensive manner taking into account the features spreading over the entire face, the fine features of the face, and the personal features. As a result, according to the first embodiment, the evaluation index of the entire face can be analyzed with high accuracy using the face image.

[Second Embodiment]
Hereinafter, the face image analysis device 10 and the face image analysis method according to the second embodiment will be described focusing on the contents different from the first embodiment. The description of the same content as in the first embodiment is omitted as appropriate. The hardware configuration of the face image analyzing apparatus 10 in the second embodiment is the same as that of the first embodiment shown in FIG.

[Operation example (Face image analysis method)]
Hereinafter, a face image analysis method according to the second embodiment will be described.
The face image analysis method according to the second embodiment executed by the face image analysis apparatus 10 includes a first step of analyzing a sample face image and a second step of analyzing the target face image. The said 1st process should just be performed once except the case where a base vector and a face image parameter | index formula are updated. The second step may be executed after the first step and at a timing at which analysis of the target face image is required.

  FIG. 4 is a flowchart illustrating an example of an operation of analyzing a sample face image in the face image analysis apparatus 10 according to the second embodiment. Hereinafter, the first step included in the face image analysis method according to the second embodiment will be described with reference to FIG.

  The image acquisition step (S41) acquires a plurality of internally reflected light images in which the entire face of each sample provider is captured. The internally reflected light image is a face image of the sample provider from which the reflected light component on the skin surface of the face is excluded. When the face image analyzer 10 has an imaging device (not shown) connected via the input / output I / F 4, for example, an internal reflected light image is acquired by the following method. Can do. A polarizing plate S that passes only s-polarized light is installed in front of the light source (between the light source and the sample provider), and a polarizing plate P that passes only p-polarized light is in front of the imaging device (the imaging device and its sample provider). In the environment installed in the middle, the imaging device images the entire face of the sample provider via the polarizing plate P. The face image analysis device 10 acquires the face images of each sample provider imaged by the imaging device via the input / output I / F 4.

  The face image analyzing apparatus 10 may acquire a plurality of internally reflected light images from a portable recording medium, another computer, or the like via the input / output I / F 4. In the present embodiment, the data format of the internally reflected light image is not limited. The image data is acquired, for example, as a file in JPEG (Joint Photographic Experts Group) format, GIF (Graphic Interchange Format) format, or the like.

  Further, the acquired internal reflection light image only needs to include the entire face, and may include portions other than the background and the head. Depending on the evaluation index, the sample provider's face may be a natural face or a face with makeup (makeup face). The sample provider is preferably determined to cover the age group and sex to be analyzed. Hereinafter, the internally reflected light image acquired in (S41) is simply referred to as a face image.

  In the normalization step (S42), a plurality of sample face images and average face images whose shapes are normalized are generated from each face image acquired in (S41). This embodiment does not limit the standardization method of the shape of the face image. For example, the face image analysis apparatus 10 performs the normalization process as follows. The face image analyzer 10 detects a plurality of feature points from the original face image, extracts a face region based on the outer edges of the detected feature points, and normalizes the extracted face region (warping process). . By normalizing the face area, the face direction, size, and position in the image are unified. Further, the face image analyzing apparatus 10 generates an average face of a plurality of sample providers by synthesizing the normalized face images of the sample providers, and averages the shapes of the normalized face images. Standardize to face shape. Such face image manipulation is performed using, for example, a known face image synthesis system called FUTON (foolproof utilities for facial image manipulation system). The face image analysis apparatus 10 generates the plurality of sample face images by excluding non-analysis regions (regions other than the skin color portion (eyes, lips, eyebrows, etc.)) from each standardized face image.

  The independent component analysis step (S43) performs independent component analysis on the average face image and a plurality of sample face images whose shapes are standardized, so that a hemoglobin pigment image, melanin is obtained from the average face image and each sample face image. A pigment image and a shadow image are extracted. Hereinafter, the hemoglobin pigment image, melanin pigment image, and shadow image extracted in (S43) may be referred to as pigment component images, respectively. Here, the pigment component decomposition method described in Non-Patent Document 1 can be used as a method of extracting three types of pigment component images from a face image using independent component analysis.

In the frequency component separation step (S44), the hemoglobin pigment image, the melanin pigment image, and the shadow image extracted in (S43) are respectively converted into a low frequency component image from which the high frequency component has been removed and a high frequency component image including only the high frequency component. To separate. The method of separating the spatial frequency component in (S44) is the same as (S21) in the first embodiment. Through the steps so far, the following low-frequency component image and high-frequency component image are acquired from the average face image and each sample face image whose shape is standardized.
-High frequency component image of hemoglobin dye image-Low frequency component image of hemoglobin dye image-High frequency component image of melanin dye image-Low frequency component image of melanin dye image-High frequency component image of shadow image-Low frequency component image of shadow image

  The basis vector acquisition step (S45) performs principal component analysis on each of the low-frequency component image and the high-frequency component image separated in (S44). Get each basis vector. In other words, the face image analyzing apparatus 10 performs principal component analysis on each of the above-described six image types obtained by the separation of (S44) from each of the average face image and all sample face images. In this principal component analysis, eigenvectors and eigenvalues are calculated from the difference information between each sample face and average face, and a multi-order basis vector is obtained from the calculated eigenvectors and eigenvalues. As a result, the face image analyzing apparatus 10 acquires six multi-order basis vectors corresponding to the six image types described above. The order of the basis vectors corresponds to, for example, the number of sample providers.

  In the sample coefficient acquisition step (S46), the weighting coefficients of the multi-order basis vectors for each of the plurality of sample face images are respectively acquired by the principal component analysis in (S45). Specifically, the face image analysis apparatus 10 acquires, for each sample face image, the weight coefficients of six multi-order basis vectors corresponding to the six image types described above.

The analysis target base order selection step (S47) selects a base order (analysis target base order) to be analyzed from the base orders of the basis vectors. The face image analysis apparatus 10 may set all the base orders as the analysis target base orders, or may set some base orders as the analysis base orders. The face image analyzing apparatus 10 performs multiple regression analysis on the relationship between the weighting coefficient group for each of the plurality of sample face images acquired in (S46) and the index value for each sample face image, and determines the determination coefficient (R ^2). ), The analysis base order is selected from all the base orders of the base vector. For example, a base order having a determination coefficient larger than a predetermined value may be selected as the analysis target base order, or a predetermined number of base orders from a base order having a large determination coefficient may be selected as the analysis base order. The analysis base order may be selected separately for each of the six multi-order base vectors calculated as described above, or may be selected in common. In addition, index values relating to each sample face image are collected in advance based on interviews with the sample provider and sensory evaluation by a predetermined evaluator, and are acquired by the face image analyzer 10 together with the internally reflected light image.

  The index formula calculation step (S48) selects and selects a plurality of analysis target weight coefficients from the plurality of weight coefficients acquired in (S46) based on the analysis target base order selected in (S47). A multiple regression analysis is performed on the relationship between the plurality of analysis target weighting coefficients and the plurality of index values related to the plurality of sample face images, thereby estimating a face image index formula that is a multiple regression formula. That is, the face image index formula is a multiple regression formula with the analysis target weight coefficient as an explanatory variable and the index value as an objective variable. By (S48), the face image analysis apparatus 10 estimates six face image index formulas corresponding to the above six image types.

  FIG. 5 is a flowchart illustrating an example of an operation of analyzing the target face image in the face image analysis apparatus 10 of the second embodiment. Hereinafter, the second step included in the face image analysis method according to the second embodiment will be described with reference to FIG.

  In the image acquisition step (S51), an internally reflected light image that captures the entire face of the subject is acquired. The method for acquiring the internally reflected light image is the same as (S41) in FIG. The internal reflected light image acquired here only needs to show the entire face of the subject, may include a part other than the background and the head, and is reflected in the internally reflected light image according to the analysis index. The face of the subject may be a natural face or a face with makeup (makeup face). Hereinafter, the internally reflected light image acquired in (S51) is referred to as the face image acquired in (S51).

  In the normalization step (S52), a target face image whose shape is normalized is generated from the face image acquired in (S51). The standardization method of the shape of the face image is basically the same as (S42). However, in (S52), the face image analysis apparatus 10 standardizes the normalized face image shape to the average face shape generated in (S42).

  The independent component analysis step (S53) extracts a hemoglobin pigment image, a melanin pigment image, and a shadow image from the target face image by performing independent component analysis on the target face image whose shape is standardized. The method of independent component analysis is the same as (S43).

The frequency component separation step (S54) is the same as (S44) except that the image to be processed is the target face image. Through the steps so far, the following low-frequency component image and high-frequency component image are respectively acquired from the target face image whose shape is standardized.
-High frequency component image of hemoglobin dye image-Low frequency component image of hemoglobin dye image-High frequency component image of melanin dye image-Low frequency component image of melanin dye image-High frequency component image of shadow image-Low frequency component image of shadow image

  In the logarithmic coefficient acquisition step (S55), weighting coefficients of a plurality of base vectors related to the target face image are acquired by principal component analysis on the low-frequency component image and the high-frequency component image separated in (S54). At this time, the face image analysis apparatus 10 performs principal component analysis using the six multi-order basis vectors corresponding to the above-described six image types acquired in (S45). Specifically, the face image analysis device 10 acquires difference information between the average face generated in (S42) and the target face image, and the difference information and the multi-order basis vectors acquired in (S45). By calculating the inner product of, it is possible to obtain the weight coefficient of the multi-dimensional basis vector regarding the target face image. Thereby, the face image analysis apparatus 10 acquires the weight coefficients of the six multi-order basis vectors corresponding to the above-described six image types, respectively, regarding the target face image.

  The analysis step (S56) selects a plurality of analysis object weight coefficients from the plurality of weight coefficients acquired in (S55) based on the analysis object base order selected in (S47), and selects the selected plurality of analysis object weight coefficients. The six index values of the target face image corresponding to the six image types described above are estimated using the analysis target weighting coefficient and the face image index formula calculated in (S48). Specifically, the face image analysis device 10 applies a plurality of analysis target weighting factors corresponding to each image type to the face image index formula corresponding to each image type, thereby providing a target corresponding to each image type. Each index value of the face image is estimated.

4 and 5, it is assumed that the sample provider is different from the subject. However, the sample provider may include the subject. That is, the target face image may be included in the plurality of sample face images. In this case, (S51) to (S55) shown in FIG. 5 are replaced with (S41) to (S44) and (S46) shown in FIG.
〔Device configuration〕
FIG. 6 is a diagram conceptually illustrating a processing configuration example of the face image analysis apparatus 10 according to the second embodiment. The face image analysis apparatus 10 according to the second embodiment includes an image acquisition unit 21, a normalization processing unit 22, an independent component analysis unit 23, a frequency component separation unit 24, a principal component analysis unit 25, a regression analysis unit 26, and a face image analysis unit. 27 etc. Each of these processing units is realized, for example, by executing a program stored in the memory 3 by the CPU 2. Further, the program may be installed from a portable recording medium such as a CD or a memory card or another computer on the network via the input / output I / F 4 and stored in the memory 3.

  The face image analyzing apparatus 10 can execute the face image analyzing method shown in FIGS. 4 and 5 as follows using each processing unit shown in FIG. The image acquisition unit 21 executes image acquisition steps (S41) and (S51). The standardization processing unit 22 executes standardization steps (S42) and (S52). The independent component analysis unit 23 performs the independent component analysis steps (S43) and (S53). The frequency component separation unit 24 executes frequency separation steps (S44) and (S54). The principal component analysis unit 25 executes a basis vector acquisition step (S45), a sample coefficient acquisition step (S46), and a target coefficient acquisition step (S55). The regression analysis unit 26 executes an analysis target base order selection step (S47) and an index formula calculation step (S48). The face image analysis unit 27 performs an analysis process (S56). The frequency component separation unit 24 corresponds to the frequency component separation unit 12 in the first embodiment, the principal component analysis unit 25 corresponds to the target coefficient acquisition unit 14 in the first embodiment, and the face image analysis unit 27 This corresponds to the analysis unit 18 in one embodiment.

[Operations and effects in the second embodiment]
Color unevenness that affects the evaluation index of the entire face includes color unevenness caused by melanin pigments such as buckwheat, mole, and stain (pigmentation), and color unevenness caused by hemoglobin pigments such as acne and cheek redness. There is. Therefore, in order to analyze the index regarding color unevenness with high accuracy, it is desirable to analyze for each different factor. For example, when considering age-related changes in the face, color unevenness caused by melanin pigments and color unevenness caused by hemoglobin pigments are likely to show different changes due to different factors. . Further, as other factors that affect the evaluation index of the entire face, there are irregularities such as wrinkles of the eyes, constriction lines, sagging of the cheeks and chin, cheek dandruff, and the like. Unevenness unevenness appears on the face image as a shadow. Therefore, in order to analyze the index related to uneven unevenness with high accuracy, it is desirable to analyze the shadow component separately from the above-described pigment component in the face image.

  Thus, as described above, in the second embodiment, the independent face analysis is applied to the average face image and the plurality of sample face images whose shapes are normalized and the target face image whose shapes are normalized, and the average face image A hemoglobin pigment image, a melanin pigment image, and a shadow image are extracted from each sample face image and target face image. Further, frequency component separation is performed on the separated component images. Thereby, according to 2nd Embodiment, compared with the form where extraction of a hemoglobin pigment image, a melanin pigment image, and a shadow image is not performed, pigment irregularity and unevenness irregularity in the whole face can be analyzed with high accuracy.

  Furthermore, in the second embodiment, an internally reflected light image is used as a face image that is a source of analysis. As a result, it is possible to prevent the face image to be processed from containing noise components such as surface reflection light components for the analysis of color unevenness and unevenness, and as a result, a normal face that is not an internally reflected light image. Compared to a form in which an image is used, analysis accuracy regarding color unevenness and unevenness unevenness can be improved.

  In the second embodiment, principal component analysis is applied to the hemoglobin pigment image, the melanin pigment image, and the shadow image obtained as described above, and a plurality of base vectors for a plurality of sample face images are acquired. The By using this basis vector, a plurality of weight coefficients of the basis vector are obtained for each of the plurality of sample face images and the target face image. As a result, according to the second embodiment, not only a wide-band spatial frequency characteristic but also a narrow-band spatial frequency characteristic can be taken into account in the face image. In addition, it is possible to perform face image analysis in consideration of exhaustive personal characteristics.

  Furthermore, in the second embodiment, a face is extracted from all the base orders of the basis vector by multiple regression analysis of the relationship between the weight coefficient group for each of the plurality of sample face images and the index value for each sample face image. A base order having a large influence on the overall evaluation index is selected as the analysis base order. Then, the face image index formula, which is a multiple regression formula, is estimated by performing multiple regression analysis on the relationship between the selected weighting factor of the analysis target base order and the plurality of index values related to the plurality of sample face images. Thereby, it is possible to obtain the face image index formula more efficiently than to obtain the face image index formula using all the base orders.

  In the second embodiment, each image type (hemoglobin) is obtained by using the basis vector and the face image index formula thus obtained from a plurality of sample face images and the weight coefficient group thus obtained from the target face image. Each index value of the target face image corresponding to each of a pigment image, a melanin pigment image, a shadow image, and a high frequency component image and a low frequency component image) is estimated. Therefore, according to the second embodiment, the features spreading over the entire face, the fine features of the face, and the personal features are comprehensively reflected for each factor of color unevenness and unevenness that can affect the evaluation index of the entire face. The obtained index value can be obtained, and as a result, the evaluation index of the entire face can be analyzed with high accuracy using the face image.

[Third Embodiment]
Hereinafter, the face image analysis device 10 and the face image analysis method according to the third embodiment will be described focusing on the contents different from the first embodiment and the second embodiment. The description of the same content as the first embodiment and the second embodiment is omitted as appropriate. The hardware configuration of the face image analyzing apparatus 10 in the third embodiment is the same as that of the first embodiment shown in FIG.

[Operation example (Face image analysis method)]
Hereinafter, a face image analysis method according to the third embodiment will be described.
The face image analysis method according to the third embodiment is different from the second embodiment in that the target face image is modulated in the analysis step included in the second step of analyzing the target face image.

  FIG. 7 is a flowchart illustrating an example of an operation of analyzing the target face image in the face image analysis apparatus 10 according to the third embodiment. Hereinafter, the second step included in the face image analysis method according to the third embodiment will be described with reference to FIG. However, in FIG. 7, steps (S51) to (S56) denoted by the same reference numerals as those in FIG. 5 are the same as those in the second embodiment. The face image analysis method in the third embodiment includes a target acquisition step (S71) and a face image modulation step (S72) in the analysis step (S56).

  In the target acquisition step (S71), the face image analyzer 10 acquires a target index value. The face image analysis apparatus 10 may acquire the target index value from a portable recording medium, another computer, or the like via the input / output I / F 4 or data input by the user using the input unit 7. Or may be stored in advance. The target index value is an index value that is a target for changing the target face image. For example, 40 years of age is set as the target index value for a subject's actual age of 30 years. When the apparent age of 25 years of the target face image is estimated as the index value in the analysis step (S56), the apparent age of 45 years is acquired as the target index value. The acquired target index value is arbitrary.

  In the face image modulation step (S72), the face image analysis apparatus 10 uses the face image index formula to calculate a plurality of analysis target weight coefficients corresponding to the difference between the target index value and the current index value of the target face image. In addition, the target face image is modulated by updating the plurality of analysis target weight coefficients for the target face image with the calculated plurality of analysis target weight coefficients, thereby generating a modulated face image. Here, as the current index value of the target face image, when the evaluation index is a quantitative index, a quantitative index value (for example, real age) of the subject shown in the target face image may be used. When the index is a qualitative index such as a face impression, the index value estimated in the analysis step (S56 in FIG. 5) may be used. Further, the current index value of the target face image may be acquired from a portable recording medium, another computer, or the like via the input / output I / F 4 or may be input by the user using the input unit 7. .

  The analysis target weighting coefficient corresponding to the difference is acquired for each image type based on the face image index formula of each of the six image types described in the second embodiment. The face image analysis device 10 adds or subtracts the analysis target weight coefficient corresponding to the difference to the analysis target weight coefficient for the target face image used in the analysis step (S56), and updates the analysis target weight coefficient, And it corresponds to the six image types by the linear sum of the weight coefficient of the base order other than the analysis target base order acquired in the target coefficient acquisition step (S55), the multi-order base vector, and the average face. A modulated image is generated. Subsequently, the face image analyzing apparatus 10 recombines the frequency component images separated in (S54) in the modulated image. This recombination is realized, for example, by performing the reverse process of the separation process of (S54). Thereby, a modulated hemoglobin dye image, a modulated melanin dye image, and a modulated shadow image are obtained.

The face image analysis apparatus 10 reintegrates the modulated hemoglobin dye image, the modulated melanin dye image, and the modulated shadow image obtained by the above recombination. For this reintegration, for example, the technique described in Non-Patent Document 1 is used. Specifically, the hemoglobin dye image modulated by ρ _m (x, y), ρ _h (x, y), and p ^log (x, y) in (Equation 10) described in Non-Patent Document 1 above. Then, by applying the modulated melanin image and the modulated shadow image, c ^log (x, y) corresponding to the modulated image is obtained, and exp (c ^log (x, y)) × 255 is obtained. By calculating, a modulated face image that is an RGB image is acquired. The modulated face image obtained here is an image obtained by modulating an internally reflected light image whose shape is standardized.

  The face image analyzing apparatus 10 applies a reverse process of the normalization step (S52) to the modulated face image, thereby obtaining a modulated internal reflection light image having a shape before normalization. get. Specifically, the face image analysis apparatus 10 restores the excluded non-analysis regions (eyes, lips, eyebrows, etc.), and restores the average face shape to the original shape.

  In the example of FIG. 7, it is exemplified that one target index value is obtained in the target acquisition step (S71), but a plurality of target index values are acquired (S71), and a face image after modulation for each target index value is obtained. May be acquired (S72).

〔Device configuration〕
FIG. 8 is a diagram conceptually illustrating a processing configuration example of the face image analysis device 10 according to the third exemplary embodiment. The face image analysis apparatus 10 according to the third embodiment further includes a face image modulation unit 28 in addition to the configuration of the second embodiment. The face image modulation unit 28 is also realized in the same manner as other processing units.

  The face image analyzing apparatus 10 can execute the face image analyzing method shown in FIGS. 4 and 7 by using each processing unit shown in FIG. The face image modulation unit 28 executes a target acquisition step (S71) and a face image modulation step (S72). Other processing units are the same as those in the second embodiment.

  In the third embodiment, the modulation of the target face image is added to the face image analysis method in the second embodiment, but the target face image is not estimated in the analysis process (analysis process), and the target value is not estimated. The face image may be modulated. In this embodiment, the face image analysis unit 27 in FIG. 8 is not necessary.

[Operations and effects in the third embodiment]
As described above, in the third embodiment, a target index value is acquired, and the original target face image is modulated into a face image corresponding to the target index value. In this face image modulation, the face image index formula, the basis vector, and the average face acquired in the first step of analyzing the sample face image are used, and the reverse procedure of the second step is sequentially executed to modulate the face image. A target face image is generated.

  Therefore, according to the third embodiment, in order to modulate the target face image using analysis information (basis vectors, face image index formulas, etc.) obtained by high-precision analysis as described in the second embodiment, A face image corresponding to the target index value can be generated with high accuracy. That is, according to the third embodiment, the change of the face image from the current index value to the target index value can be reproduced with high accuracy.

[Supplement to the above embodiments]
In each of the above-described embodiments, an example in which a high-frequency component image and a low-frequency component image are extracted from a face image in the frequency component separation processing has been described. However, the number and space of the extracted high-frequency component images and low-frequency component images The frequency separation boundary is not limited.

  For example, in the frequency component separation step (S54), the face image analyzing apparatus 10 performs a two-dimensional discrete wavelet transform on the face image in a plurality of stages, so that the diagonal low frequency component image and the pair An angular high-frequency component image, a horizontal high-frequency component image, and a vertical high-frequency component image may be generated. In this case, in the target coefficient acquisition step (S55), the face image analysis apparatus 10 determines a plurality of weighting coefficients for the target face image as a diagonal low-frequency component image, a diagonal high-frequency component image at each stage, It acquires about each of a horizontal direction high frequency component image and a vertical direction high frequency component image.

  FIG. 9 is a diagram showing the concept of frequency component separation processing using two-dimensional discrete wavelet transform. According to the example of FIG. 9, the face image analysis device 10 (frequency component separation unit 24) uses the hemoglobin pigment image, the melanin pigment image, and the shadow image extracted by the independent component analysis as face images before processing, before this processing. A low-frequency component image and a high-frequency component image are generated by performing two-dimensional discrete wavelet transform on the face image. In the two-dimensional discrete wavelet transform, the discrete wavelet transform is performed in the horizontal direction (each row) of the image, so that the image is separated into a high frequency component H and a low frequency component L in the horizontal direction. Next, discrete wavelet transform is performed on the separated images in the vertical direction (each column), whereby the images are separated in the horizontal direction and the vertical direction, respectively. The face image before processing is separated into a diagonal low-frequency component image LL, a diagonal high-frequency component image HH, a horizontal high-frequency component image HL, and a vertical high-frequency component image LH by one-stage two-dimensional discrete wavelet transform.

  When one-stage two-dimensional discrete wavelet transform is performed in the frequency component separation process, the image LL shown in FIG. 9 is extracted as the low-frequency component image, and the image HL shown in FIG. LH and HH are extracted. Furthermore, in the frequency component separation process, a two-dimensional two-dimensional discrete wavelet transform may be performed. According to the two-stage two-dimensional discrete wavelet transform, the images LL and LLLL shown in FIG. 9 are extracted as the low-frequency component images, and the images HH, HL, LH, LLHH shown in FIG. LLHL and LLLH are extracted.

When the two-dimensional two-dimensional discrete wavelet transform is performed in the second and third embodiments described above, the face image analysis device 10 (principal component analysis unit 25) has the following 24 types: The principal component analysis can be performed on each of the images. However, even in this case, the face image analysis apparatus 10 may perform principal component analysis only on the low-frequency component image LLLL and the high-frequency component image HH.
-Low frequency component image LLLL of hemoglobin dye image
-Low frequency component image LL of hemoglobin dye image
-High-frequency component image LLLH of hemoglobin dye image
-High frequency component image LLHL of hemoglobin dye image
-High frequency component image LLHH of hemoglobin dye image
-High-frequency component image HL of hemoglobin dye image
-High frequency component image LH of hemoglobin dye image
-High frequency component image HH of hemoglobin dye image
-8 images for melanin images-8 images for shadow images

  Here, according to the multi-resolution analysis of the two-dimensional discrete wavelet transform, the dimension (number of pixels) of the image (pigment density distribution) which is the original signal is reduced to a quarter per stage. That is, when the number of pixels of the face image before processing is 262144 (= 512 × 512), the number of pixels of the low-frequency component image LL after one-step conversion is 65536 (= 256 × 256), and two-step conversion is performed. The number of pixels of the subsequent low-frequency component image LLLL is 16384 (= 128 × 128), and the number of pixels of the low-frequency component image LLLLLLL after the three-stage conversion is 4096 (= 64 × 64). Here, it is desirable to perform multi-resolution analysis up to a stage where the skin area represented by one pixel of the low-frequency component image is about 4 mm (millimeters). The multi-resolution analysis may be performed up to many stages, but the resolution is lowered, so that the processing efficiency is deteriorated. For example, when the entire face (about 15 to 25 cm) is reflected in an image of 262144 pixels, the image has a resolution of about 0.5 mm per pixel, and the image after three-stage conversion has a resolution of about 4 mm. Have. In addition, when the entire face is shown in an image of (1024 × 1024) pixels, four-stage two-dimensional discrete wavelet transform may be performed.

  Therefore, the face image analysis apparatus 10 (frequency component separation unit 24) may determine the number of stages for performing the two-dimensional discrete wavelet transform based on the resolution (number of pixels) of the face image.

[Modification]
In the second embodiment and the third embodiment described above, the internal reflection light image is acquired, the independent component analysis is performed on the internal reflection light image whose shape is standardized, and the hemoglobin dye separated by the independent component analysis is obtained. Frequency component separation processing was performed on the image, the melanin pigment image, and the shadow image. However, instead of internally reflected light, a general face image is acquired, and independent component analysis may be performed on a general face image whose shape is standardized, or without performing independent component analysis, Frequency component separation processing may be performed on the internally reflected light image whose shape is standardized. In such a modification, although the analysis accuracy is lower than in the second and third embodiments, a significant effect can be obtained as compared with the prior art.

  Further, when the evaluation index is derived from a facial feature represented by surface reflected light such as a shine component, the surface reflected light image is used instead of or in addition to the internally reflected light image. May be obtained, and the above-described processing may be performed on the surface reflected light image whose shape is standardized. According to this modification, it is possible to perform the feature analysis and face image operation of the entire face in consideration of other components such as the shine component.

  Furthermore, in the second and third embodiments described above, the analysis target base order is selected. However, although the processing speed is reduced, multiple regression analysis may be performed using the weight coefficients of all base orders. Good.

  In the second and third embodiments described above, the target face image index values corresponding to the six image types are estimated, but the target face image index values are integrated. It may be estimated in units. For example, the face image analyzing apparatus 10 integrates the index values related to the high-frequency component image and the low-frequency component image, and estimates the index values of the target face image for three types of melanin pigment image, hemoglobin pigment image, and shadow image, respectively. You may make it do. Furthermore, the face image analyzing apparatus 10 may estimate one index value of the target face image.

  Examples will be given below to describe the above-described embodiments in more detail. However, the above-described embodiments are not limited by the following examples. The following examples prove the validity of the operations and effects of the above-described embodiments.

  FIG. 10 is a diagram showing the results of three-stage two-dimensional discrete wavelet transform (multi-resolution analysis). FIG. 11 is a diagram illustrating a result of one-stage two-dimensional discrete wavelet transform (multi-resolution analysis). The number of pixels of the face image before processing that is the original signal in FIGS. 10 and 11 is 262144 (= 512 × 512). However, in FIGS. 10 and 11, a masking process is performed on the image of each frequency component so that the face region can be clearly determined.

  As shown in FIGS. 10 and 11, the face image is separated into a high-frequency component image and a low-frequency component image by multi-resolution analysis. As shown in FIG. 11 (a), the low-frequency component image represents the global information of the dye density distribution, and as shown in FIGS. 11 (b), (d) and (e), the high-frequency component image. Since the component image is a difference in the pigment concentration distribution, it represents a local pigment concentration change in the face. That is, the high-frequency component image includes fine color unevenness such as moles and freckles. Further, as shown in FIG. 10, by performing multi-resolution analysis in multiple stages, feature amounts are more remarkably separated, and the low-frequency component image does not include minute changes in facial pigment density distribution. Most of them are included in the high-frequency component image. As described above, according to FIGS. 10 and 11, it is clear that a wide range change and a fine change in the dye density distribution can be separated by the multi-resolution analysis in three stages.

  FIG. 12 is a diagram illustrating a result of principal component analysis for a low-frequency component image, and FIG. 13 is a diagram illustrating a result of principal component analysis for a high-frequency component image. In FIG. 12 and FIG. 13, since 17 sample face images were used, 16 principal components (16th order basis vectors) were obtained.

  According to FIG. 12, the characteristic amount of color unevenness over the entire face or the entire face part is expressed in the main component of the low frequency component. According to FIG. 13, fine color unevenness and personal characteristics are expressed in the main component of the high-frequency component. For example, the mole portion of the first main component in FIG. 13A and the cheek portion of the fourth main component in FIG. According to FIG. 12 and FIG. 13, each main component of the high frequency component can be regarded as expressing a feature amount of fine color unevenness of the face or unique color unevenness of the individual. As a result, it is clear that it is possible to obtain different types of characteristic features of pigment unevenness by principal component analysis for each frequency component, and further, the entire face or facial part can be obtained by principal component analysis for low frequency components. It becomes clear that a main component in which the feature amount of color unevenness over the whole is expressed more prominently can be obtained.

  FIG. 14 is a diagram illustrating a result of modulating only a low-frequency component image of a melanin pigment image, and FIG. 15 is a diagram illustrating a result of modulating only a high-frequency component image of a melanin pigment image. In the example of FIGS. 14 and 15, the actual age is used as the evaluation index, the current index value of the target face image is 24 years, and four target index values of 20, 30, 40, and 50 years are set. A modulated face image corresponding to each target index value is generated. Specifically, by multiple regression analysis using a low-frequency component image obtained by a three-stage discrete wavelet transform, a face image index formula having an actual age as an index value is acquired, and using this face image index formula, The analysis target weighting coefficient corresponding to each difference between the current index value and each target index value was acquired. Then, each of the modulated low-frequency component images corresponding to each target index value is generated by the weighting factor updated with the analysis target weighting factor corresponding to each difference, and the two-dimensional inverse of the modulated image is generated. By applying the discrete wavelet transform, a modulated melanin image corresponding to each target index value was generated. Thereafter, the reconstructed melanin image, the original hemoglobin dye image, and the original shadow image were recombined to generate a modulated face image corresponding to each age.

  As shown in FIG. 14, as a result of age-modulating only the low-frequency component, the result was that the skin color of the entire face changed to black. That is, a result that can be regarded as an increase in the melanin concentration of the entire face was obtained. On the other hand, as shown in FIG. 15, as a result of age-modulating only the high-frequency component, a result of gradually generating fine color unevenness such as freckles was obtained. That is, a result that can be regarded as an increase in local melanin concentration was obtained. These results depend on the characteristics of the color unevenness feature quantity of each frequency component obtained by principal component analysis, and correspond to the results of emphasizing the characteristics by modulation. Can be considered. As described above, the frequency component separation step (frequency component separation process) can separate the feature amount of the pigment unevenness of the entire face into the low frequency component and the feature amount of the fine pigment unevenness of the face into the high frequency component. It became clear. Then, it has become clear that by performing principal component analysis and multiple regression analysis on each frequency component, it is possible to reproduce changes in the appearance of the face having the characteristics of each frequency component.

Hereinafter, the results of analyzing the sample face images of 203 people (female: 201 people, male: 2 people) in their 20s to 80s by the same method as the second embodiment and the third embodiment described above are shown. . In the following, the selected analysis target base order is exemplified as the analysis result. In this example, a three-dimensional two-dimensional discrete wavelet transform is performed on each pigment component image, and a weight coefficient of a multi-order basis vector is calculated by principal component analysis on each frequency component image. Then, based on the result of the multiple regression analysis of the relationship between the calculated weight coefficient group of each sample face image and the index value, the analysis target base order is selected in order from the base order having the largest determination coefficient (R ² ). . In the following, results are shown for the case where the actual age is used as the evaluation index and the case where the impression of youthfulness is used as the evaluation index.

Here, a part of the above results are extracted and illustrated. Specifically, (Type 1), (Type 2), (Type 3), (Type 4) and (Type 5) are selected from the following frequency component images separated for the melanin pigment image. The analysis base order is extracted.
(Type 1) Low frequency component image LLLLLLL of melanin pigment image
(Type 2) Low frequency component image LLLL of melanin pigment image
(Type 3) High frequency component image LLLLLLH of melanin pigment image
(Type 4) High frequency component image LLHL of melanin pigment image
(Type 5) High frequency component image LH of melanin pigment image

Hereinafter, for each type extracted, the analysis target base orders are listed in descending order of the coefficient of determination.
When the evaluation index (index value) is real age:
(Type 1) 1, 6, 3, 14, 5, 13, 19, 7, 46, 148, 4, 181
(Type 2) 1, 6, 3, 13, 5, 14, 19, 126, 7, 20, 4, 81, 9, 37
(Type 3) 3, 4, 9, 10, 20, 22, 38, 47, 17, 124, 61, 35, 1, 156, 84, 93
(Type 4) 1, 3, 4, 5, 17, 81, 50, 12, 15, 59, 27, 65, 124, 94, 40, 181, 191, 74, 103, 37, 61, 13, 187, 153, 9
(Type 5) 3, 6, 7, 2, 30, 25, 79, 126, 178, 70, 14, 110, 49, 36, 101, 23, 193, 5, 84, 29, 189, 153, 47, 9, 31, 156, 162, 53, 123, 91, 26, 58, 90, 44, 100, 87

As an example of the analysis result when the evaluation index is youthfulness, a part of the face image index formula obtained in the index formula calculation step (index formula calculation processing) is extracted and shown. In addition, each type shown below is the same as that when the above-mentioned evaluation index (index value) is real age.

  Next, a specific example using the surface reflected light image will be described below as a second embodiment.

  Hereinafter, the face image analysis apparatus 10 and the face image analysis method according to the second embodiment will be described focusing on contents different from the above contents. The description of the same content as above is omitted as appropriate. The hardware configuration of the face image analyzing apparatus 10 in Example 2 is the same as that of the first embodiment shown in FIG.

[Operation example (Face image analysis method)]
Hereinafter, a face image analysis method according to the second embodiment will be described.
FIG. 16 is a flowchart illustrating an example of an operation of analyzing the sample face image of the face image analysis apparatus 10 according to the second embodiment. Hereinafter, the first step included in the face image analysis method according to the second embodiment will be described with reference to FIG.

  The image acquisition step (S161) acquires a plurality of surface reflected light images in which the entire face of each sample provider is captured. A surface reflected light image is a face image of a sample provider including a reflected light component on the skin surface of the face. The surface reflected light image may include only the reflected light component, or may include the reflected light component and other components. The face image analyzing apparatus 10 may acquire the plurality of surface reflected light images described above from a portable recording medium, another computer, or the like via the input / output I / F 4. Further, when the face image analyzing apparatus 10 includes an imaging device (not shown), for example, the surface reflected light image can be acquired by the following method.

  The face image analysis apparatus 10 acquires a surface reflected light image from each polarization image obtained by projecting S-polarized light on the subject's skin and imaging S-polarized light and P-polarized light. Specifically, the face image analyzing apparatus 10 projects a polarized image obtained by projecting S-polarized light and imaging S-polarized light (SS polarized image), and polarized light obtained by projecting S-polarized light and imaging P-polarized light. The difference from the image (SP polarization image) is taken. The SS polarization image is an image having a strong surface reflection light component on the subject's skin and a small amount of the internal reflection light component, and the SP polarization image is an image having a strong internal reflection light component on the subject's skin. Therefore, by obtaining the difference between the SS polarized image and the SP polarized image, a surface reflected light image in which noise other than the surface reflected light component is further suppressed can be acquired. Further, the face image analysis apparatus 10 further generates a PS polarized image obtained by projecting P polarized light and imaging S polarized light, and a PP polarized image obtained by projecting P polarized light and imaging P polarized light. The surface reflected light image may be acquired. Thereby, the surface reflected light image which further suppressed noises other than the surface reflected light component can be acquired.

  What is shown in the acquired surface reflected light image is the same as the above-described internally reflected light image. Henceforth, the surface reflected light image acquired by (S161) is only described with a face image.

  In the normalization step (S162), a plurality of sample face images and average face images whose shapes are normalized are generated from each face image acquired in (S161). The processing content of (S162) is the same as (S42) described above except that the processing target is a surface reflected light image.

  The frequency component separation step (S163) includes only the low-frequency component image from which the high-frequency component is removed and the high-frequency component of the average face image generated in (S162) and the plurality of sample images whose shapes are normalized. Separate into high frequency component images. In the second embodiment, two-dimensional discrete wavelet transform is used as the separation method in (S163). For this conversion, for example, a Haar wavelet is used. However, the wavelet used is not limited. Hereinafter, the two-dimensional discrete wavelet transform may be abbreviated as wavelet transform.

  The surface reflected light image expresses the gloss of the entire face and uneven unevenness such as pores and wrinkles. Since the uneven unevenness that appears in high spatial frequency components is too individual-specific, the principal component analysis May be difficult to extract. Therefore, in the second embodiment, the resolution of the high-frequency component image is reduced and individual differences are absorbed, and then the feature is extracted by principal component analysis. Specifically, in (S163), wavelet transform is applied not only to diagonal low-frequency component images (LL, LLLL, etc.) but also to high-frequency component images. It is a new idea to apply the two-dimensional discrete wavelet transform to the high-frequency component image. In the spatial frequency component separation method using the two-dimensional discrete wavelet transform, as shown in FIG. 9, only diagonal low-frequency component images (LL, LLLL, etc.) are normally subject to transformation in the second and subsequent stages. This is because that.

  FIG. 17 is a diagram illustrating the concept of the two-dimensional discrete wavelet transform in the second embodiment. In the example of FIG. 17, the second-stage wavelet transform is applied not only to the diagonal low-frequency component image LL but also to the diagonal high-frequency component image HH, the horizontal high-frequency component image HL, and the vertical high-frequency component image LH. Is done. Thereby, (S163) generates each diagonal low frequency component image (CH, CD, CV) in each high frequency component (HL, HH, LH). However, (S163) may generate any one or any two of the diagonal low-frequency component images (CH, CD, CV).

  However, in each diagonal low-frequency component image in each high-frequency component, there is a risk that the feature information on the shine and unevenness of the unevenness is greatly reduced. This is because, in the process of reducing the resolution of the high-frequency component image by the wavelet transform, the edge information that appears as the positive / negative information in the adjacent pixels may be significantly reduced due to the cancellation of the positive / negative. Therefore, in the second embodiment, absolute value calculation (S164) is performed in the frequency component separation step (S163). Specifically, in (S164), an absolute value calculation is performed on the high-frequency component image obtained by the first-stage conversion.

  FIG. 18 is a diagram illustrating the concept of frequency component separation processing according to the second embodiment. In each high-frequency component image acquired by the first-stage wavelet transform, as shown by the line graph in FIG. 18, shine and unevenness appear as positive / negative information of adjacent pixels. When the second-stage wavelet transform is applied to each high-frequency component image as it is, the edge information may be greatly reduced due to the cancellation of the positive and negative. Therefore, in the second embodiment, the absolute value calculation (S164) is performed on each high-frequency component image obtained by the first-stage conversion. At this time, the code information of the pixel whose code is reversed may be held. (S163) generates each diagonal low-frequency component image in the high-frequency component by applying the second-stage wavelet transform to each high-frequency component absolute value image obtained by the absolute value calculation. As a result, it is possible to prevent a decrease in the feature information of the shine and unevenness in the process of reducing the resolution. Further, if the code information of the pixel whose code is reversed by the absolute value calculation is held, the original code can be restored when the image is recombined. Details of image recombination will be described later.

When the two-stage wavelet transform is performed in (S163), for example, the following low-frequency component image and high-frequency component image are separated from the average face image and each sample face image whose shape is standardized, respectively. The
Low frequency component image = diagonal low frequency component image LL and LLLL in the low frequency component
High frequency component image = diagonal low frequency component image CH, CD and CV in the high frequency component
However, the images HL, HH, and LH can also be used as high-frequency component images. Furthermore, the images LLHL, LLHH, and LLLH shown in FIG. 17 and high-frequency component images other than the images CH, CH, and CV in the high-frequency component may also be used as the high-frequency component image. In this case, it is desirable to apply absolute value calculation to each high-frequency component image.

In the basis vector acquisition step (S165), by performing principal component analysis on each of the low frequency component image and the high frequency component image separated in (S163), for each low frequency component image and each high frequency component image, a plurality of orders Get each basis vector. The processing content of (S165) is the same as (S45) described above.
Thereafter, the processing contents of (S166), (S167), and (S168) are the same as (S46), (S47), and (S48) described above.

  FIG. 19 is a flowchart illustrating an example of an operation of analyzing the target face image in the face image analysis apparatus 10 according to the second embodiment. Hereinafter, the second step included in the face image analysis method according to the second embodiment will be described with reference to FIG.

  In the image acquisition step (S191), a surface reflected light image in which the entire face of the subject is captured is acquired. The method of acquiring the surface reflected light image is the same as (S161) in FIG. Further, what is shown in the acquired surface reflected light image is the same as the internally reflected light image acquired in (S51) of FIG. Henceforth, the surface reflected light image acquired by (S191) is only described with a face image.

  In the normalization step (S192), a target face image whose shape is normalized is generated from the face image acquired in (S191). The method for normalizing the shape of the face image is the same as (S52) in FIG.

  The frequency component separation step (S193) is the same as (S163) described above except that the image to be processed is the target face image. (S193) includes holding the information of the pixel whose sign is inverted by the absolute value calculation (S195) together with the absolute value calculation (S194). The information held in (S195) may be only information indicating the position of the pixel whose code is inverted, or may be information indicating the position of the pixel before being inverted. However, (S195) is unnecessary when recombination described later is not performed.

(S196) is the same as (S55) in FIG.
(S197) includes a target acquisition step (S198) and a face image modulation step (S199) in addition to the processing content of (S56) in FIG. The target acquisition step (S198) is the same as (S71) in FIG.

  In the face image modulation step (S199), the face image analysis device 10 generates a target face image modulated according to the target index value acquired in (S198). However, in the second embodiment, the face image analyzing apparatus 10 modulates the target face image using the plurality of updated analysis target weight coefficients based on the pixel information held in (S195) (S194). ) To restore the sign inverted by the absolute value calculation in (1) to generate a modulated face image.

  Specifically, the face image analysis apparatus 10 updates a plurality of analysis target weight coefficients for the target face image, similarly to (S72) of FIG. The face image analysis apparatus 10 includes the updated analysis target weight coefficient, the weight coefficient of the base order other than the analysis target base order acquired in the target coefficient acquisition step (S196), the multi-order basis vector, the average Modulated images are generated for high-frequency components and low-frequency components by linear summation with the face. The face image analysis device 10 re-synthesizes the generated modulated image by performing inverse transformation of the two-dimensional discrete wavelet. At the time of the inverse transformation, the face image analyzing apparatus 10 recovers the sign inverted by the absolute value calculation in (S194). The face image analyzing apparatus 10 applies a reverse process of the normalization step (S192) to the modulated face image, thereby obtaining a modulated surface reflected light image having a shape before normalization. get. Specifically, the face image analysis apparatus 10 restores the excluded non-analysis regions (eyes, lips, eyebrows, etc.), and restores the average face shape to the original shape.

  In FIGS. 16 and 19, it is assumed that the sample provider and the subject are different from each other, but the sample provider may include the subject. That is, the target face image may be included in the plurality of sample face images. In this case, (S191) to (S196) shown in FIG. 19 are replaced with (S161) to (S164) and (S166) shown in FIG.

  FIG. 20 is a diagram illustrating a part of the result of the principal component analysis in the second embodiment. According to the example of FIG. 20, in step S193, three-stage two-dimensional discrete wavelet transform is performed. Specifically, after absolute value calculation is performed on the diagonal high-frequency component image HH, the horizontal high-frequency component image HL, and the vertical high-frequency component image LH obtained by the first-stage wavelet transform, the second-stage high-frequency component image HH is calculated. The wavelet transform is applied to HH, HL, and LH obtained by the absolute value calculation, and the diagonal low frequency component image LL, respectively. The third-stage wavelet transform is applied to the diagonal low frequency component image LLLL in the low frequency component and the diagonal low frequency component image HHLL, HLLL, and LHLL in the high frequency component, respectively. According to the example of FIG. 20, the images CH, CD, and CV are separated from the original image as high-frequency component images by three-stage wavelet transform. However, other images can also be used as the high frequency component image.

  FIG. 20 shows images of the top four principal components obtained by principal component analysis on the separated high-frequency component images CH, CD, and CV. The image shown in FIG. 20 indicates that according to the second embodiment, the feature information of the shine and unevenness can be extracted from the high frequency component of the surface reflected light image. Further, when the image is viewed, it has become clear that the following feature information can be extracted from the main components of the CH image, the CV image, and the CD image. From the main component of the CH image including the horizontal component, it is possible to extract mainly horizontal feature information such as forehead wrinkles. From the main components of the CV image including the vertical component, vertical feature information such as wrinkles between the jaws and eyebrows and cheek pores can be extracted. Feature information in an oblique direction such as wrinkles near the cheeks can be extracted from the main component of the CD image including diagonal components.

〔Device configuration〕
FIG. 21 is a diagram conceptually illustrating a processing configuration example of the face image analysis apparatus 10 according to the second embodiment. The face image analysis apparatus 10 according to the second embodiment includes an image acquisition unit 31, a normalization processing unit 32, a frequency component separation unit 33, a principal component analysis unit 34, a regression analysis unit 35, an absolute value calculation unit 36, and a pixel information holding unit 37. A face image analysis unit 38, a face image modulation unit 39, and the like. Each of these processing units is realized, for example, by executing a program stored in the memory 3 by the CPU 2. Further, the program may be installed from a portable recording medium such as a CD or a memory card or another computer on the network via the input / output I / F 4 and stored in the memory 3.

  The face image analyzing apparatus 10 can execute the face image analyzing method shown in FIGS. 16 and 19 as follows using each processing unit shown in FIG. The image acquisition unit 31 executes image acquisition steps (S161) and (S191). The normalization processing unit 32 executes normalization steps (S162) and (S192). The frequency component separation unit 33 performs a two-dimensional discrete wavelet transform. The absolute value calculator 36 performs absolute value calculation on the high frequency component image obtained by the first-stage wavelet transform. That is, the absolute value calculation unit 36 executes (S164) and (S194). The pixel information holding unit 37 holds information on the pixel whose sign is inverted in the absolute value calculation by the absolute value calculation unit 36 (S195). The frequency component separation unit 33, the absolute value calculation unit 36, and the pixel information holding unit 37 execute (S163) and (S193).

  The principal component analysis unit 34 executes a basis vector acquisition step (S165), a sample coefficient acquisition step (S166), and a target coefficient acquisition step (S196). The regression analysis unit 35 executes an analysis target base order selection step (S167) and an index formula calculation step (S168). The face image analysis unit 38 executes an analysis process (S197). The face image modulation unit 39 executes a target acquisition step (S198) and a face image modulation step (S199).

  As described above, in the second embodiment, the second-stage wavelet transform is applied to the high-frequency component image obtained by the first-stage wavelet transform. In the second embodiment, by reducing the resolution of the high-frequency component image in this way, the individual difference between the shine and unevenness appearing in the high spatial frequency component is reduced. The feature information can be extracted. Further, in the second embodiment, wavelet transform is applied to a high frequency component image in which absolute value calculation is applied to the high frequency component image. This suppresses a decrease in the feature information of the shine and unevenness due to the application of the wavelet transform. According to the second embodiment, it is possible to perform feature analysis and face image manipulation of the entire face in consideration of components such as shine and uneven unevenness.

  Furthermore, according to the second embodiment, a feature component for each direction can be extracted from high-frequency components for each direction (horizontal direction, vertical direction, and diagonal direction) obtained by wavelet transform. Thereby, according to Example 2, it is possible to analyze the evaluation index of the entire face with high accuracy by using the feature amount of the subject's skin surface such as shine and uneven unevenness in a state that is distinguished by direction, and A face image corresponding to the target index value can be generated with high accuracy.

  In Example 2 described above, the analysis step (S197) may not include the target acquisition step (S198) and the face image modulation step (S199). In this case, the analysis step (S197) only needs to estimate the index value of the target face image corresponding to each of the low frequency component image and the high frequency component image. In this case, the face image analysis device 10 may not have the face image modulation unit 39.

  Further, the number of wavelet transform steps executed in the frequency component separation steps (S163) and (S193) may be only one step. In this case, in the sample coefficient acquisition step (S166) and the target coefficient acquisition step (S196), the low frequency component image LL and the high frequency component images HH, HL, and LH to which the absolute value calculation is applied are mainly used. It is desirable to perform component analysis. Further, when a multi-stage wavelet transform is executed and an image other than the diagonal low frequency component image in the high frequency component (HHHL, HHLH, HLHL, etc.) is used as the high frequency component image, It is desirable to perform principal component analysis on an image obtained by applying absolute value calculation to the image.

  In addition, in the several flowchart used by the above-mentioned description, although several process (process) is described in order, the execution order of a process is not restrict | limited to the order of the description. The order of the steps shown in the figure can be changed within a range that does not hinder the contents. Moreover, each above-mentioned embodiment, each modification, and each Example can be combined in the range with which the content does not conflict.

2 CPU
3 Memory 10 Face image analyzer (analyzer)
12 frequency component separation unit 14 target coefficient acquisition unit 18 analysis unit 21, 31 image acquisition unit 22, 32 normalization processing unit 23 independent component analysis unit 24, 33 frequency component separation unit 25, 34 principal component analysis unit 26, 35 regression analysis Unit 27, 38 Face image analysis unit 28, 39 Face image modulation unit 36 Absolute value calculation unit 37 Pixel information holding unit

Claims (14)

A frequency component separation step of separating the face image whose shape is standardized into a low frequency component image from which a high frequency component has been removed and a high frequency component image including only the high frequency component;
A plurality of basis vectors obtained from a plurality of sample face images are obtained, and main components for each of the low-frequency component image and the high-frequency component image separated from the target face image included in the face image by the frequency component separation step are obtained. A target coefficient acquisition step of acquiring a plurality of weighting coefficients of the base vector for the target face image by component analysis;
From the plurality of weight coefficients acquired in the target coefficient acquisition step, select a plurality of analysis target weight coefficients in at least a part of the analysis target base orders among the base orders possessed by the base vector, and the selected Using a plurality of analysis target weight coefficients for a target face image and a face image index formula calculated by multiple regression analysis using a plurality of analysis target weight coefficients and a plurality of index values for the plurality of sample face images An analysis step of analyzing the target face image;
A face image analysis method including: The frequency component separation step further includes a low-frequency component image from which the high-frequency component is removed, and a high-frequency component image including only the high-frequency component of the plurality of sample face images whose shapes are standardized included in the face image. Separated into
The basis vectors for the low frequency component image and the high frequency component image by principal component analysis for each of the low frequency component image and the high frequency component image separated from the plurality of sample face images by the frequency component separation step. Each obtaining a basis vector, and
A sample coefficient acquisition step of acquiring a plurality of weight coefficients of the basis vector for each of the plurality of sample face images by the principal component analysis in the basis vector acquisition step;
The face image index is obtained by multiple regression analysis using the plurality of analysis object weight coefficients among the plurality of weight coefficients acquired by the sample coefficient acquisition step and a plurality of index values related to the plurality of sample face images. An index formula calculation step for estimating the formula;
The face image analysis method according to claim 1, further comprising: The face image index formula includes at least a first face image index formula and a second face image index formula corresponding to each separation unit in the frequency component separation step,
The analyzing step applies the plurality of analysis target weighting factors for the target face image obtained by principal component analysis on the low frequency component image to the first face image index formula, and Applying the plurality of analysis target weighting factors for the target face image obtained by principal component analysis on the high frequency component image to the second face image index formula, an index value for the target face image is obtained, Estimate for each separation unit in the frequency component separation step,
The face image analysis method according to claim 1 or 2. The analysis step includes
A target acquisition process for acquiring target index values;
Using the face image index formula, calculate a plurality of analysis target weight coefficients corresponding to the difference between the target index value and the current index value of the target face image, and calculate the plurality of analysis target weight coefficients A modulation step of modulating the target face image by updating the plurality of analysis target weight coefficients for the target face image, and generating a modulated face image;
The face image analysis method according to claim 1, comprising: The frequency component separation step performs a plurality of stages of two-dimensional discrete wavelet transform on the face image, so that for each stage, a diagonal low frequency component image, a diagonal high frequency component image, a horizontal high frequency component image, and Generate vertical high-frequency component images,
In the target coefficient acquisition step, the plurality of weight coefficients for the target face image are obtained by using the diagonal low frequency component image, the diagonal high frequency component image, the horizontal high frequency component image, and the vertical at each stage. Acquired for each of the directional high-frequency component images,
The face image analysis method according to any one of claims 1 to 4. The frequency component separation step determines the number of steps for performing a two-dimensional discrete wavelet transform based on the resolution of the face image.
The face image analysis method according to claim 5. Independent component analysis step for extracting a hemoglobin pigment image, a melanin pigment image and a shadow image from the face image,
Further including
The frequency component separation step separates each of the hemoglobin pigment image, the melanin pigment image, and the shadow image extracted by the independent component analysis step into the low-frequency component image and the high-frequency component image.
The face image analysis method according to any one of claims 1 to 6. An image acquisition step of acquiring an internally reflected light image of the subject;
From the internally reflected light image acquired in the image acquisition step, a normalization step of generating the face image with a normalized shape,
The face image analysis method according to claim 1, further comprising: An image acquisition step of acquiring a surface reflected light image of the subject;
From the surface reflected light image acquired in the image acquisition step, a normalization step of generating the face image with a normalized shape,
Further including
The frequency component separation step includes
The face image is separated into the low-frequency component image and the high-frequency component image by two-dimensional discrete wavelet transform,
Absolute value calculation is performed on the high-frequency component image obtained by the two-dimensional discrete wavelet transform.
Including that,
The face image analysis method according to any one of claims 1 to 4. The frequency component separation step performs further two-dimensional discrete wavelet transform on the high frequency component absolute value image obtained by the absolute value calculation.
The face image analysis method according to claim 9. The frequency component separation step generates a high frequency component absolute value image obtained by the absolute value calculation as a high frequency component image after separation,
The face image analysis method according to claim 9 or 10. Holding information of pixels whose sign is inverted by the absolute value calculation,
Further including
The analysis step includes
A target acquisition process for acquiring target index values;
Using the face image index formula, calculate a plurality of analysis target weight coefficients corresponding to the difference between the target index value and the current index value of the target face image, and calculate the plurality of analysis target weight coefficients A modulation step of modulating the target face image by updating the plurality of analysis target weight coefficients for the target face image, and generating a modulated face image;
Including
The modulation step recovers the code inverted by the absolute value calculation based on the held pixel information when the target face image is modulated using the updated plurality of analysis target weight coefficients. Generating the modulated face image,
The face image analysis method according to claim 9.   A face image analysis apparatus that executes the face image analysis method according to claim 1.   A program causing a face image analysis method according to any one of claims 1 to 12 to be executed by at least one computer.