JP7429683B2 - Method for determining temporal change characteristics of facial surface texture - Google Patents

Description

The present invention relates to a technique for predicting and analyzing changes in human facial surface texture over time using a trained model obtained by machine learning and facial image data.

In Patent Document 1 listed below, an image of a target person's face and a representative face image of each age group for comparison with that face are displayed, and the target person's appearance is determined based on the comparison result of the two appearances. A technique for calculating the age group of people is disclosed.
Patent Document 2 listed below discloses a technique for calculating the intensity of a spatial frequency from face image data of a subject and calculating the estimated age of the subject based on the correlation between the intensity of the spatial frequency and the age. There is.
Patent Document 3 listed below discloses a method of predicting aging using a trained model obtained by machine learning. The trained models include a shape aging model that predicts age-related changes in facial shape, a texture aging model that predicts age-related changes in facial surface texture, and a texture aging model that predicts 3D data from 2D images. A three-dimensional prediction model is included. This method extracts feature points from the target image, estimates the face orientation in the target image using the extracted feature points, and generates first 3D data based on the 3D prediction model and the estimated face orientation. Then, the shape aging model is used to generate second 3D data from the first 3D data, and the texture aging model is applied to the 2D image generated based on the first 3D data. Then, an aging texture is generated, and the aging texture is synthesized with the second three-dimensional data to generate an aging face model.

JP2021-71888A International Publication No. 2011/162050 Japanese Patent Application Publication No. 2016-194892

Many people are vaguely concerned about how their facial impressions change as they age.
On the other hand, changes in facial impressions are caused by changes in facial surface texture, and the tendency for facial impressions to change with age differs from person to person.
Therefore, it is very important from the viewpoint of beauty to enable each individual to specifically know the future change trend of his or her own facial surface texture.

The present invention provides a technique that can specifically show trends in changes over time in facial surface texture of each individual.
As used herein, "facial surface texture" corresponds to spatial distribution characteristics originating from the surface or inside of facial skin. Examples of "facial surface texture" include features related to uneven skin tone such as spots, freckles, pigment spots, melasma, uneven redness, dark circles, and dullness, as well as irregularities on the skin surface such as wrinkles, sagging, and pores. Features that create the spatial distribution of the facial surface include features related to shadows, uneven distribution of surface roughness such as skin gloss or scales, crepe wrinkles, and characteristics related to brightness and darkness.

According to the present invention, a trained model is obtained by machine learning using training data including a plurality of face image data sets before and after changes over time, and calculates changes over time feature values of facial surface texture of input face image data. a data acquisition step in which one or more processors capable of utilizing possible trained models acquire facial image data of a subject; a feature amount acquisition step of acquiring a time-varying feature amount of the subject's facial surface texture; and a dimension compression step of compressing the number of dimensions of the acquired time-varying feature amount in multiple stages to obtain a two-dimensional time-varying feature amount. and the temporal change of the subject compressed into two dimensions in the dimension compression step based on the distribution information of the two-dimensional temporal change feature amount of the facial surface texture obtained in advance with respect to the face image data group of the population. a type identifying step of specifying a time-varying characteristic type that indicates a tendency of the facial surface texture of the subject to change over time by performing cluster determination on the feature amounts, and the identified time-varying characteristic type is , is one of a plurality of temporal change characteristic types each characterized by a degree of local change and a degree of global change in facial surface texture, and the cluster determination is performed by determining a plurality of clusters based on the distribution information. A cluster corresponding to the temporal change characteristic amount of the subject is determined from among a plurality of clusters in which the mutually different temporal change characteristic types are respectively associated, and in the type identification step, the cluster corresponding to the temporal change characteristic amount of the subject is determined. A method for determining a temporal change characteristic of a facial surface texture is provided, which identifies a temporal change characteristic type corresponding to a cluster.
Further, according to the present invention, there is provided a temporal change characteristic determination device that can utilize the learned model and can execute the temporal change characteristic determination method. Furthermore, it is also possible to provide a program that causes a computer to execute the method for determining characteristics over time, and a computer-readable recording medium on which such a program is recorded can also be provided. This recording medium includes non-transitory tangible media.

According to the above aspect, it is possible to provide a technique that can specifically indicate future change trends in the facial surface texture of each individual.

1 is a diagram conceptually illustrating an example of a hardware configuration of an information processing device capable of executing a method for determining a temporal change characteristic of facial surface texture according to an embodiment of the present invention; FIG. 3 is a diagram illustrating a process flow of a method for determining temporal change characteristics of facial surface texture according to the first embodiment. It is a graph which shows the application result of the principal component analysis to the 10-dimensional time-varying component score in an Example. It is a graph showing the distribution of two-dimensional time-varying component scores in an example. It is a graph showing clustering of the distribution of two-dimensional time-varying component scores in an example. It is a graph showing the age distribution of three clusters in an example. It is a graph which shows the discriminant of the distribution of two-dimensional time-varying component scores in an example. It is a figure showing a part of distribution of two-dimensional time-varying component scores in an example, and an image showing the characteristics of the first principal component of the time-varying component scores. It is a figure which shows a part of distribution of a two-dimensional time-varying component score in an Example, and the image showing the characteristic of the second principal component of the time-varying component score.

Hereinafter, an example of a preferred embodiment of the present invention (hereinafter referred to as the present embodiment) will be described. Note that this embodiment described below is an example, and the present invention is not limited to the configurations listed below.

[Hardware configuration example]
A method for determining temporal change characteristics of facial surface texture (hereinafter referred to as the present method) according to the present embodiment is executed by one or more processors included in one or more information processing apparatuses.
FIG. 1 is a diagram conceptually showing an example of the hardware configuration of an information processing apparatus 10 that can execute this method.
The information processing device 10 is a so-called computer, and includes a CPU 11, a memory 12, an input/output interface (I/F) 13, a communication unit 14, and the like. The information processing device 10 may be a stationary PC (Personal Computer), a portable PC, a smartphone, a tablet, or other mobile terminal.

The CPU 11 is a so-called processor, and may include, in addition to a general CPU (Central Processing Unit), an application-specific integrated circuit (ASIC), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), and the like. The memory 12 is a RAM (Random Access Memory), a ROM (Read Only Memory), or an auxiliary storage device (hard disk, etc.).
The input/output I/F 13 can be connected to user interface devices such as the display device 15 and the input device 16. The display device 15 is a device, such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display, that displays a screen corresponding to drawing data processed by the CPU 11 or the like. The input device 16 is a device such as a keyboard, a mouse, etc. that receives user operation input. The display device 15 and the input device 16 may be integrated and realized as a touch panel.
The communication unit 14 communicates with other computers via a communication network and exchanges signals with other devices such as printers. A portable recording medium or the like may also be connected to the communication unit 14.

The hardware configuration of the information processing device 10 is not limited to the example shown in FIG. Information processing device 10 may include other hardware elements not shown. Further, the number of each hardware element is not limited to the example of FIG. 1 either. For example, the information processing device 10 may include multiple CPUs 11. Further, the information processing device 10 may be realized by a plurality of computers each having a plurality of housings.

The information processing device 10 can execute this method by having the CPU 11 execute a computer program stored in the memory 12. It can also be stated that the CPU 11 can perform the method through execution of a computer program stored in the memory 12. This computer program is installed via the input/output I/F 13 or the communication unit 14 from a portable recording medium such as a CD (Compact Disc), a memory card, etc. or from another computer on the network, and is stored in the memory 12. be done.

The information processing device 10 (CPU 11) can use a trained model obtained by machine learning using teacher data.
The "trained model" here is a model obtained by machine learning using teacher data, that is, supervised learning, and is expressed as an AI (Artificial Intelligence) model, Machine Learning (ML) model, etc. It is possible.
The trained model used in this embodiment may be a regression equation obtained by regression analysis, or may be a neural network model obtained by principal component analysis, deep learning, etc. The data structure, learning algorithm, etc. of the model are not limited. For example, the trained model is realized by a combination of a computer program and a parameter, a combination of a plurality of functions and parameters, or the like. When a trained model is constructed using a neural network, and the input layer, intermediate layer, and output layer are considered as units of one neural network, the trained model may refer to one neural network, or it may refer to multiple neural networks. It may also refer to a combination of neural networks. Further, the trained model may be configured by a combination of multiple regression equations, or may be configured by one multiple regression equation.
The trained model may be stored in the memory 12 within the information processing device 10, or may be stored in the memory of another computer that the information processing device 10 can access via communication.

In this way, the information processing device 10 can be described as a device that can use a trained model and can execute a method for determining a temporal change characteristic of facial surface texture.
Hereinafter, the trained model used in this method will be referred to as an AI model. Furthermore, in the following description, the CPU 11 will be described as the main body that executes this method.

[Method for determining temporal change characteristics of facial surface texture]
The details of this method will be explained below using FIG. 2. FIG. 2 is a diagram showing the processing flow of the method for determining temporal change characteristics of facial surface texture according to the present embodiment.

The AI model used in this embodiment is a trained model obtained by machine learning using training data that includes multiple face image datasets before and after changes over time, and is a trained model that is obtained by machine learning using training data that includes multiple face image data sets before and after changes over time. It is possible to calculate the change feature amount.
Here, "time-varying characteristic amount of facial surface texture of facial image data" means that the facial surface texture of a human face reflected in facial image data changes over time from when the human face is imaged until a predetermined time or a predetermined period has elapsed. This is information that shows what kind of change will occur and the characteristics of that change.
The time-varying feature amount is, for example, facial surface texture information of a human face reflected in input facial image data, and a predetermined period of time or a predetermined period (for example, several hours or several days) predicted by the AI model from the input facial image data. , several weeks, several years, etc.). However, there are no restrictions on the specific data format or data content of the temporal change feature amount.
The facial surface texture information may be information indicating facial surface texture, and may be image data that is a set of pixel values for each pixel, or may be data in a format other than image data. In this embodiment, principal component scores encoded by principal component analysis of image data are used as the facial surface texture information. The principal component score is a group of numerical values indicating the weights of low- to high-order feature vectors (eigenfaces) obtained by principal component analysis. The number of dimensions of the principal component score depends on the number of training data used in machine learning of the AI model, so it is preferably large, but it is desirable to suppress the amount of data so that it does not become larger than the image data format. It will be done. Therefore, the number of dimensions of the principal component score is, for example, 50 dimensions, 100 dimensions, etc.

The AI model in this embodiment calculates the principal component score of the facial surface texture of the human face reflected in the input facial image data, and calculates the principal component score for the predetermined time period of the human face using a machine-learned predictive transformation matrix. Alternatively, the principal component scores of the facial surface texture after a predetermined period of time have elapsed are predicted, and the difference between these principal component scores is calculated as the time-varying feature amount.
When the time-varying feature quantity is calculated by the difference between facial surface textures, and when the facial surface texture information is represented by image data that is a set of pixel values for each pixel, the difference can be calculated using any well-known method. It can be calculated using this method. For example, by comparing facial surface texture images pixel by pixel or block by block and calculating the difference value (signed or absolute value) or ratio of shading values, or by comparing and determining the shading values, difference data can be obtained. can be extracted. The extracted difference data may contain unnecessary information such as electronic noise and skin unevenness information due to illumination that differs from the original facial surface texture, so frequency filters and spatial filters are also applied. By applying this to the extracted difference data, unnecessary information may be excluded. Further, in order to extract only necessary information from the difference data, extraction may be performed through morphological transformation.

The length of the elapsed time of the temporal change feature may be a length of time that allows the change in the facial surface texture of the human face to be visually recognized, and may be several hours, days, weeks, months, or several hours. For example, it can be set to the morning and night of the same day (12 hours), one week, the period from summer to winter (six months), three years, five years, ten years, etc. However, the length of the elapsed time is not particularly limited.
Further, the predetermined time or the predetermined period (length of elapsed time) may be set by a face image data set before and after changes over time that is used as teacher data. For example, if the face image data set used as training data is a set of face image data that shows a human face after 5 years and face image data that shows a human face 5 years ago, then (length of elapsed time) can be considered as 5 years.
The face image data used as training data is preferably image data of a bare face, but may be image data of a face with makeup. In addition, it is preferable that the training data is formed only from facial image datasets of the same person before and after changes over time, but it is also possible to The data set may include face image data sets of two people, or may include face image data of either one before or after change over time and the other face image data generated by image processing on that face image data. may be formed.
However, the facial image data used in the training data has the orientation of the photographed face normalized, and the position of specific parts of each individual's face (eyes, nose, mouth, etc.) has been normalized. It is desirable that Further, the face image data used as the teacher data may be not only a visible light image but also an image captured in a wavelength band other than visible light such as infrared light.

The AI model may be one that ultimately calculates the time-varying feature amount of the facial surface texture of the input facial image data, or may be one that calculates the time-varying feature amount as intermediate information before the final output. There may be.
In addition, various well-known techniques can be used for the data structure and learning method of the AI model. For example, the generation method and learning method of the texture aging model disclosed in Patent Document 3 mentioned above can be used. . For example, both a texture aging model using principal component analysis and a texture aging model using WAVELET transformation may be used, or either one may be used. Further, the face image data applied to the teacher data and the AI model may be converted to an image in a cylindrical coordinate system other than a rectangular coordinate system.

As shown in FIG. 2, this method includes steps (S21) to (S26).
Step (S21) is a data acquisition step of acquiring facial image data of the subject. In the following description, the face of the subject reflected in the face image data acquired in step (S21) may be referred to as the original face.
The face image data is acquired as an image file in, for example, JPEG (Joint Photographic Experts Group) format, BMP (Bitmap image) format, TIFF (Tagged Image File Format) format, GIF (Graphic Interchange Format) format, or the like. However, the data format of the image is not limited, and may be a format in which brightness information of each pixel is listed. Further, the acquired image may be a color image, or a grayscale image obtained by converting a color image into a grayscale. The CPU 11 may acquire facial image data from a camera that has captured a facial image, or may acquire facial image data from another computer or a portable recording medium.

In the face image data acquired in step (S21), it is preferable that the state of the subject in the image matches that of the face image used in the training data of the AI model. For example, when the face image of the teacher data shows the facial surface texture seen from the front, it is preferable that the face image data similarly shows the facial surface texture seen from the front of the subject.
Therefore, in step (S21), facial image data may be acquired by subjecting the acquired facial image data to normalization processing to match the facial image of the AI model's teacher data. In this normalization process, for example, the size and position of the face in the image data, the position of specific parts of the face (eyes, nose, mouth, etc.) are normalized, the background is removed, and the like.
When normalizing the positions of specific parts of the face (eyes, nose, mouth, etc.), in the step (S21), a predetermined image recognition process is performed on the acquired facial image data to identify the eyes, nose, mouth, etc. Specific parts such as the nose, mouth, and eyebrows may be automatically recognized. In this case, in the step (S21), it is also possible to check whether the acquired facial image data can be used based on the presence or absence of a specific part, positional relationship, etc. indicated by the automatic recognition result.

Step (S22) is a feature amount acquisition step of acquiring a time-varying feature amount of the subject's facial surface texture calculated by the trained AI model based on the face image data acquired in step (S21).
When the AI model is stored in the memory 12 in the information processing device 10, the CPU 11 applies the facial image data acquired in step (S21) to the AI model as input facial image data. The time-varying feature amount can be obtained from the AI model. On the other hand, if the AI model is stored in the memory of another computer that the information processing device 10 can access via communication, the CPU 11 transmits the facial image data acquired in step (S21) to the other computer. By sending this, the time-varying feature amount calculated by the AI model can be acquired from the computer via communication.
In this embodiment, the time-varying feature acquired in step (S22) is the principal component score of the original face reflected in the input face image data, and the predetermined time predicted by the AI model from the principal component score of the original face. Alternatively, the difference data may be used as difference data with the principal component score of the subject's face after a predetermined period of time (for example, 5 years) has elapsed.

Step (S23) is a dimension compression step in which the number of dimensions of the time-varying feature obtained in step (S22) is compressed in multiple stages to obtain a two-dimensional time-varying feature.
In the step (S23), the number of dimensions of the time-varying feature obtained in the step (S22) may be compressed to two dimensions, and the number of stages of dimension compression may be two or more, and the compression dimension of each stage may be compressed to two dimensions. The number is arbitrary.
For example, when two stages of compression are performed in the step (S23), in the first stage of compression, the element groups of the temporal change features acquired in the step (S22) are compressed so that they each have the same number of elements. By dividing it into a predetermined number of dimensions, calculating the Euclidean norm of the element for each divided dimension region, and using it as an element in that dimension region, the time-varying feature is compressed into a predetermined number of dimensions, and the second stage of compression is performed. Then, a two-dimensional time-varying feature is obtained by performing dimension compression processing on the predetermined number of dimensions of the time-varying feature.

In the example described later, in the first stage of compression, a 100-dimensional time-varying feature quantity (100 element groups) is divided into 10 element groups each, and 10 elements are By calculating the Euclidean norm of the element group and using it as an element in that dimension domain, a 10-dimensional time-varying feature quantity is obtained. In the next second stage of compression, principal component analysis is applied to the 10-dimensional time-varying features to calculate the weights of the first principal component (feature vector) and the second principal component (feature vector). Obtain two-dimensional time-varying features to be used as elements.
In this example, the number of divisions in the first stage compression is preferably 3 or more and 20 or less. According to such a number of divisions, the accuracy of the time-varying feature amount can be maintained with the weight of the first principal component and the weight of the second principal component obtained by the principal component analysis in the second stage of compression.

In addition, when the number of stages of dimension compression is three stages, the number of dimensions of the time-varying feature quantity is compressed in stages using Euclidean norm calculation in the first and second stages of compression, Principal component analysis similar to that described above can be used in the third stage of compression.
Further, the compression method at each stage is not limited to only Euclidean norm calculation and principal component analysis, and other known dimension compression methods such as factor analysis can also be used.

The step (S24) is based on the distribution information of the two-dimensional time-varying characteristic amount of the facial surface texture obtained in advance for the face image data group of the population, and the image of the subject compressed into two dimensions in the step (S23). This is a step of performing cluster determination on time-varying feature quantities.
All or part of the "population face image data group" here may be all or part of the training data of the AI model, or other face image data groups not included in the training data. It may be.
When a group of image data included in the training data of the AI model is used, the "two-dimensional temporal change feature of the facial surface texture acquired in advance" is the facial surface texture of the facial image dataset before and after the temporal change. It can be obtained by calculating the difference between pieces of information in advance. On the other hand, when other facial image data groups that are not included in the training data of the AI model are used, the "two-dimensional time-varying feature amount of the facial surface texture acquired in advance" is based on each facial image data. can be obtained in advance by applying this to a trained AI model.

"Distribution information of two-dimensional time-varying features of facial surface texture acquired in advance" is a mapping of a group of time-varying features acquired in advance to a two-dimensional coordinate system with each dimension of the time-varying features as an axis. It may be a feature value distribution diagram obtained by using the above method, or it may be a discriminant function or a decision boundary equation obtained by discriminant analysis of the group of time-varying feature values obtained in advance. Any known discriminant algorithm such as Fisher's linear discriminant analysis method can be used for the discriminant analysis.

Therefore, the cluster determination in step (S24) is performed on a plurality of clusters based on the distribution information of the temporal change feature obtained in advance, and in which mutually different temporal change characteristic types are associated with each other. From this, clusters corresponding to the subject's time-varying feature values are determined.
If the distribution information is the above-mentioned feature distribution map, this feature distribution map is divided into multiple clusters, and by mapping the subject's time-varying feature values to the feature distribution map, It is possible to determine clusters corresponding to the subject's time-varying feature values. In addition, if the distribution information is a discriminant function that can distinguish each cluster, by inputting the subject's time-varying feature quantity into this discriminant function, the cluster corresponding to the subject's time-varying feature quantity can be determined. can do.

There are multiple types of temporal change characteristics that indicate trends over time. Each type of change is characterized by a greater or lesser degree of change (large).
A local change in the facial surface texture is a change in the distribution of small color unevenness in the facial surface texture or a change in the shading distribution due to the structure (on the order of several mm or smaller (for example, on the order of several hundred μm)). Examples of changes include changes in color unevenness such as age spots, freckles, pigment spots, liver spots, uneven redness, and dark circles.Changes in shadow distribution due to small structures include changes in surface roughness such as scales and crepe wrinkles. This corresponds to changes in brightness and darkness caused by uneven distribution, or changes in shadows caused by irregularities on the skin surface such as fine wrinkles, pores, upper eyelid grooves, lower eyelid grooves, and geniolabial grooves.
A global change in the facial surface texture is a large uneven color distribution of the facial surface texture, a change in the shading distribution (on the order of several cm) due to the structure, or a change in the gloss distribution, and as a change in the shading distribution due to this large structure. For example, this corresponds to changes in shadows caused by irregularities on the skin surface such as large folds, wrinkles, nasolabial folds, sagging skin, and firmness of the skin, or moss (depression) on the cheeks and temples. The change in shadow due to swelling of orbital fat is also equivalent. A large change in color unevenness distribution corresponds to, for example, a change in skin dullness or large age spots.

Step (S25) is a type specifying step of specifying a time-varying characteristic type that indicates a tendency of the subject's facial surface texture to change over time, according to the result of the cluster determination in step (S24). Specifically, among the plurality of time-varying characteristic types, the time-varying characteristic type associated with the cluster determined in step (S24) is specified as the time-varying characteristic type of the subject.

The present inventors have demonstrated that distribution information obtained by performing multiple stages of dimensional compression on the multidimensional time-varying feature values of facial surface textures collected for multiple people is It has been newly discovered that facial surface texture can be classified into clusters, and each cluster is characterized by more or less local changes and more or less global changes in facial surface texture.
Then, as described above, by determining the cluster to which the two-dimensional temporal change feature of the subject's facial surface texture belongs based on the distribution information, local Identify time-varying characteristic types characterized by more or less global changes and more or less global changes.

Step (S26) outputs image data of the temporal change feature amount of the facial surface texture of the subject obtained from the AI model in step (S22), along with the information on the temporal change characteristic type specified in step (S25). This is the output process.
The output in the step (S26) may be realized by displaying the information of the temporal change characteristic type and the image data of the temporal change characteristic amount on the display device 15, or by printing it on a printer device. Alternatively, it may be realized by sending the information to a portable recording medium or another computer via the communication unit 14.
Further, as an output format, the CPU 11 may display (output) the information of the temporal change characteristic type and the image data of the temporal change characteristic amount side by side so that they can be compared, or may switch between them. may be displayed or may be displayed in a superimposed manner.

The information on the temporal change characteristic type to be output is, for example, a character string that describes the characteristics of the temporal change characteristic type characterized by the degree of local change and the degree of global change in the facial surface texture. Ru. Further, identification information of the type of characteristics changing over time may be output.
The image data of the temporal change feature amount may be any data that can indicate the characteristic of the temporal change of the facial surface texture of a human face together with the position information of the facial surface (for each part of the facial surface). For example, as will be described later as an example, two types of image data representing the first and second elements of a two-dimensionally compressed time-varying feature are generated as image data of a time-varying feature. Good too. The image data in this example may be generated from the image data of the time-varying feature amount before compression as exemplified in the embodiment, or may be generated from the image data of the time-varying feature amount after the first stage of compression. Alternatively, it may be generated from the image data of the two-dimensional time-varying feature after final compression.
If the AI model can output the time-varying feature amount and the position information of the face surface in a form in which they are associated with each other, image data of the time-varying feature amount may be obtained by combining them.
In this way, by outputting image data that can show the characteristics of changes over time in the facial surface texture of a human face together with the position information on the face surface, it is possible to present trends in changes over time in the facial surface texture for each part of the face.

Therefore, by outputting image data of the temporal change feature amount of the subject's facial surface texture obtained from the AI model along with information on the temporal change characteristic type, it is possible to determine the local and global changes in the facial surface texture. It is possible to present not only the temporal change tendency characterized by the degree of change, but also the positional information of the facial surface where such a temporal change tendency occurs.

[Modified example]
The contents of the above-described embodiments can be modified as appropriate without causing any problems.
For example, in the above-described embodiment, image data of the time-varying characteristic amount is output together with the information of the time-varying characteristic type in step (S26), but only the information of the time-varying characteristic type may be output.

Moreover, the following process may be performed instead of step (S26). That is, the above-described method for determining the temporal change characteristic of facial surface texture may further include the following steps (S31) and (S32) in addition to steps (S21) to (S25).

In the step (S31), the CPU 11 extracts from the memory 12 one or both of the skin care information and the beauty treatment information corresponding to the temporal change characteristic type specified in the step (S25).
The memory 12 stores in advance one or both of skin care information and beauty treatment information for a person determined to have a plurality of temporal change characteristic types in association with identification information of each of the plurality of temporal change characteristic types. There is. Based on the identification information of the temporal change characteristic type specified in step (S25), the CPU 11 can extract one or both of the skin care information and the beauty treatment information corresponding to the temporal change characteristic type from the memory 12.

The skin care information is information related to skin care, and includes information on skin care products or skin care methods. Skin care products are any products that can be used for skin care, such as skin care cosmetics such as facial cleansers, lotions, serums, etc., products used for removing makeup (absorbent products, etc.), and skin care products. There may be food and drinks for this purpose. Foods and drinks for skin care also include nutritional supplements called supplements that are dissolved in water and ingested. A skin care method is a skin beauty method and excludes medical procedures. Skin care methods can include various methods such as diet, exercise menu, lifestyle-related methods such as what to do after taking a bath or before going to bed, and how to use skin care products.
The beauty treatment information is information regarding beauty treatments for a person's face (excluding medical treatments), and includes information such as beauty massage.
However, the specific contents of the extracted skin care information and beauty treatment information are not limited in any way.

In the step (S32), the CPU 11 outputs one or both of the skin care information and the beauty treatment information extracted in the step (S31).
The output in step (S32) may be realized by displaying one or both of skin care information and beauty treatment information on the display device 15, or may be realized by printing on a printer device, or may be realized by printing on a printer device. It may also be realized by sending the information to a portable recording medium or another computer via the communication unit 14.
Further, in the step (S32), the CPU 11 selects, in addition to one or both of the skin care information and the beauty treatment information, one or both of the information on the temporal change characteristic type identified in the step (S25) or the image data of the temporal change characteristic amount. Both may be further output.

According to such a modified example, by providing an image of the subject's own face, the subject can determine the type of change tendency (temporal change characteristics) of the subject's own facial surface texture over time (the passage of a predetermined time or a predetermined period of time). Not only can you know your type (type), you can also know skin care information and beauty treatment information suitable for that type.
Skin care information and beauty treatment information suitable for a time-varying characteristic type are information that reduces local changes and/or global changes in facial surface texture depending on the time-varying characteristic type. For example, if the temporal change characteristic type indicates relatively more global and local changes in facial surface texture, then When skin care information or beauty treatment information is output and the temporal change characteristic type indicates that there are relatively many global changes in the facial surface texture and relatively few local changes, the global change in the facial surface texture is Skin care information or beauty treatment information that causes fewer changes is output.

Some or all of the embodiments and modifications described above may also be specified as follows. However, the embodiments and modifications described above are not limited to the following description.

<1>
A trained model that is obtained by machine learning using training data that includes multiple facial image datasets before and after changes over time, and that is capable of calculating temporal change features of facial surface texture of input facial image data. One or more processors that can be used
a data acquisition step of acquiring face image data of the subject;
a feature amount acquisition step of acquiring a time-varying feature amount of the facial surface texture of the subject calculated by the learned model based on the acquired facial image data;
a dimension compression step of compressing the number of dimensions of the acquired time-varying feature amount in multiple stages to obtain a two-dimensional time-varying feature amount;
The time-varying feature of the subject compressed into two dimensions in the dimensional compression step based on the distribution information of the two-dimensional time-varying feature of the facial surface texture obtained in advance with respect to the face image data group of the population. a type identification step of identifying a temporal change characteristic type that indicates a temporal change tendency of the facial surface texture of the subject by performing cluster determination on the subject;
Run
The specified time-varying characteristic type is one of a plurality of time-varying characteristic types each characterized by a degree of local change and a degree of global change in facial surface texture,
The cluster determination includes selecting a cluster corresponding to the temporal change characteristic amount of the subject from among a plurality of clusters based on the distribution information and each of which is associated with a mutually different temporal change characteristic type. Determine,
In the type identification step, a time-varying characteristic type corresponding to the determined cluster is identified;
A method for determining the characteristics of facial surface texture changes over time.

<2>
The plurality of clusters include at least a first cluster, a second cluster, and a third cluster,
The first cluster has a first temporal change characteristic type indicating that, with respect to facial surface texture, the global change is less than the second cluster and the third cluster, and the local change is less than the second cluster. is associated with
Regarding facial surface texture, the second cluster has more global changes than the first cluster and less than the third cluster, and has more local changes than the first cluster and the third cluster. is associated with the second aging characteristic type that indicates
The third cluster is a third temporal change characteristic type indicating that, with respect to facial surface texture, global changes are more than the first cluster and the second cluster, and local changes are less than the second cluster. is associated with
The method for determining characteristics over time according to <1>.
<3>
In the dimensional compression step, two stages of compression are performed,
In the first stage of compression in the dimensional compression process, the element group of the time-varying feature is divided into a predetermined number of elements so that each has the same number of elements, and the Euclidean norm of the element is calculated for each divided dimensional area. compress the time-varying feature amount to the predetermined number of dimensions by making it an element of the dimension domain,
In the second stage of compression in the dimensional compression step, the two-dimensional time-varying feature quantity is obtained by a dimensional compression process on the time-varying feature quantity of the predetermined number of dimensions.
The method for determining characteristics over time according to <1> or <2>.
<4>
the one or more processors,
an output step of outputting image data of the temporal change feature amount of the facial surface texture of the subject calculated by the learned model, together with information on the temporal change characteristic type specified in the type identification step;
The method for determining characteristics over time according to any one of <1> to <3>, further comprising performing the following.
<5>
The one or more processors can refer to an information storage unit that stores one or both of skin care information and beauty treatment information in association with each of the plurality of temporal change characteristic types,
the one or more processors,
extracting, from the information storage unit, one or both of skin care information and beauty treatment information corresponding to the time-varying characteristic type identified in the type identifying step;
outputting one or both of the extracted skin care information or beauty treatment information;
The method for determining characteristics over time according to any one of <1> to <4>, further comprising performing the following.
<6>
A device for determining characteristics over time that can utilize the trained model and is capable of executing the method for determining characteristics over time according to any one of <1> to <5>.
<7>
A computer program capable of causing one or more processors that can use the learned model to execute the method for determining a temporal change characteristic according to any one of <1> to <5>, or a computer program for the one or more processors that can use the learned model. A recording medium that records data in a readable manner.

The above-mentioned contents will be explained in more detail by giving examples below. However, the description of the following examples is not intended to limit the above content in any way.

In this example, a face image data set in which the faces of 106 Japanese women between the ages of 16 and 74 were captured before and after five years was prepared as the teacher data.
The AI model is similar to the embodiment described above, and calculates the principal component score composed of the weights of the 100-dimensional feature vector (eigenface) by performing principal component analysis on the input facial image data. By applying the principal component scores to the predictive transformation matrix, the principal component scores of the facial surface texture after changes over time are calculated, and the differences between these principal component scores are calculated as the changes over time feature quantity. Therefore, the calculated temporal change feature amount is 100-dimensional data.
In this embodiment, the time-varying feature amount calculated by the AI model is referred to as a time-varying component score.
Through machine learning using the above training data, this AI model is able to calculate the principal component score of the facial surface texture after 5 years, and is a model that can calculate the temporal change feature amount before and after 5 years. It was learned as.

In this example, by applying face image data of the faces of each of the 106 people in the teacher data taken five years ago to the AI model learned in this way, the temporal change component scores of the 106 people are calculated. was calculated.
The number of dimensions of each of the temporal change component scores for the 106 people calculated in this way was compressed in two stages, and two-dimensional temporal change component scores were obtained. Specifically, in the first stage of dimension compression, the elements (weights) of each temporal change component score are divided into 10 dimensions in order from the lowest degree of the feature vector (eigenface), and the By calculating the Euclidean norm of a group of 10 (10 dimensions) elements (weights) and using them as elements in that dimensional domain, the 10-dimensional time-varying component scores are determined. In the second stage of dimension reduction, principal component analysis is applied to the 10-dimensional time-varying component scores, and the weight of the first principal component (feature vector) and the weight of the second principal component (feature vector) are A two-dimensional time-varying component score was determined.

At this time, the principal component of the 10-dimensional time-varying component score (Euclidean norm) obtained in the first step of dimensional compression is compared to the first principal component and second principal component obtained in the second step of dimensional compression. Regarding the loadings, in the first principal component, the contribution of the low-order dimension region from the 1st to the 5th dimension is large, and in the second principal component, the contribution of the high-order dimension region from the 6th dimension to the 8th dimension is large. It was shown that the contribution was large. The fact that the degree height of the 10-dimensional temporal change component score (Euclidean norm) obtained in the first stage of dimension compression corresponds to the degree height of the eigenface before dimension compression means that This is clear from the process of generating change component scores. In general, eigenfaces have the characteristic that the lower the degree, the more global features (larger features) are selectively extracted, and the higher the degree, the more local features (smaller features) are selectively extracted. One principal component was considered to indicate global changes in facial surface texture, and the second principal component was considered to indicate local changes in facial surface texture.

FIG. 3 is a graph showing the results of applying principal component analysis to the 10-dimensional time-varying component scores in the example, and FIG. 3 shows the contribution rate and cumulative contribution rate of each principal component.
The vertical axis of Figure 3 shows the contribution rate and cumulative contribution rate (%), and the horizontal axis shows the number of each principal component (principal components are identified by numbers from 1 to 10 from low order to high order). ) is shown.
According to Figure 3, the results of the principal component analysis performed in the second stage of dimensional compression of the time-varying feature values show that the cumulative contribution rate of the first and second principal components accounts for 75% or more. I understand.
Therefore, as mentioned above, the two-dimensional time-varying component score, which has the weight of the first principal component and the weight of the second principal component as elements, is determined in the second stage of dimension compression of the temporal change feature. It can be said that the feature values are sufficiently expressed.

FIG. 4 is a graph showing the distribution of two-dimensional time-varying component scores in the example.
When the two-dimensional time-varying component scores obtained for each of the 106 people as described above were mapped onto a graph with the first principal component on the horizontal axis and the second principal component on the vertical axis, the results are shown in Figure The distribution shown in Figure 4 was shown.

FIG. 5 is a graph showing clustering of the distribution of two-dimensional time-varying component scores in the example.
By subjectively clustering each plot showing each person's time-varying component score in the distribution graph shown in FIG. 4, the results were divided into three clusters as shown in FIG. 5. In addition, since the validity of this clustering was confirmed by evaluating the distance between plots, it can be said that the clustering here was performed by qualitative evaluation using the distance between plots in addition to subjective judgment.
It can be said that each cluster has the following characteristics.
In cluster 1, the variance of the values of the first principal component is small, the variance of the values of the second principal component is large, and the values of the first principal component and the second principal component of the center of gravity of the cluster both show positive values. .
In cluster 2, the values of the first principal component and the second principal component are clustered with the same degree of variance, and the value of the second principal component at the center of gravity of the cluster shows a negative value.
In cluster 3, the value of the first principal component and the value of the second principal component show a negative correlation, and the value of the first principal component of the center of gravity of the cluster shows a negative value, and the value of the second principal component shows a negative correlation. The value indicates a positive value.
Further, cluster 1, cluster 2, and cluster 3 have an overall shape like a smile curve in this order.

FIG. 6 is a graph showing the age distribution of three clusters in the example.
When we took statistics on the age of each person in the two-dimensional temporal change component score for each cluster shown in Figure 5, we found that the age composition of each cluster was significantly different from each other, as shown in Figure 6. Do you get it. Cluster 1 has an average age of 31 years old and has a large number of temporal change component scores for the generation under 34 years old, and cluster 2 has an average age of 47 years old and has a large number of temporal change component scores for the generation from 35 to 44 years old. Cluster 3 has many scores, and cluster 3 has an average age of 62 years and a large number of temporal change component scores for the 55 and older generation, indicating that each cluster is classified according to the age structure. Do you get it.

Therefore, the three clusters are considered to capture the characteristics of facial surface texture changes that tend to occur in each age group.
Therefore, the horizontal axis (the first element (first principal component) of the two-dimensional time-varying component score) in the distribution graphs shown in FIGS. The vertical axis (the second element (second principal component) of the two-dimensional temporal change component score) is the local change (about several mm or smaller (for example, about several hundred μm)) of the facial surface texture. It was thought to indicate a change. Regarding the first principal component, the larger the component, the smaller the global change; , local changes were considered to be large.
In addition, the horizontal axis shows changes in the distribution of major color unevenness such as skin dullness and large age spots, major structural changes such as wrinkles, sagging, and firmness of the skin, or changes in the distribution of gloss, and the vertical axis shows changes in the distribution of color unevenness such as skin dullness and large age spots. Small uneven color changes such as freckles, pigment spots, melasma, uneven redness, dark circles, uneven distribution of surface roughness such as scales, crepe wrinkles, small wrinkles, pores, upper eyelid grooves, lower eyelid grooves, mental labial grooves, etc. It can also be said to show changes in shading distribution due to structure.

FIG. 7 is a graph showing a discriminant for the distribution of two-dimensional time-varying component scores in the example.
Based on the subjective clustering results shown in FIG. 5, a discriminant for determining the three clusters was generated as a formula for determining the boundaries of each cluster. In this example, two linear discriminants (Eq1 and Eq2) shown in FIG. 7 were determined.
The linear discriminant Eq1 has a positive slope, and the linear discriminant Eq2 has a negative slope. The value of the principal component (vertical axis) shows a positive value.
According to these two discriminants, the area above discriminant Eq1 and below discriminant Eq2 is determined to be cluster 1, the area below discriminant Eq1 and below discriminant Eq2 is determined to be cluster 2, and the area below discriminant Eq1 is determined to be cluster 2. Moreover, the area above discriminant Eq2 can be determined to be cluster 3.

FIG. 8 is a diagram showing a part of the distribution of the two-dimensional time-varying component scores in the example and an image showing the characteristics of the first principal component of the time-varying component score. FIG. 7 is a diagram showing an image showing a part of the distribution of the time-varying component scores and the characteristics of the second principal component of the time-varying component scores. Hereinafter, the image representing the characteristics of the first principal component of the two-dimensional temporal change component score will be referred to as the first principal component corresponding image, and the image representing the characteristics of the second principal component of the two-dimensional temporal change component score will be referred to as the second principal component image. Sometimes referred to as a principal component corresponding image.
8 and 9 show the distribution of the time-varying component scores for 17 people aged 45 to 54 out of the 106 time-varying component scores shown in FIG. 4 and the like.
Furthermore, in FIG. 8, for 8 of the 17 people, images corresponding to the first principal component of the two-dimensional time-varying component scores are shown at the end of the leader line from the plot of the time-varying component scores of each person. In FIG. 9, images corresponding to the second principal components of the two-dimensional time-varying component scores for the same eight people are shown at the end of the leader line from the plot of the time-varying component scores of each person. ing.

As shown in FIGS. 8 and 9, the first principal component corresponding image and the second principal component corresponding image include positional information of each part of the face surface (eyes, nose, mouth, cheeks, etc.). In this example, by using each order feature vector (eigenface) used in the principal component analysis performed by the AI model, the first principal component corresponding image containing the position information of each part of the face surface and the second principal A component corresponding image can be generated.
Specifically, the first principal component corresponding image shown in FIG. After converting the pixel values into absolute values for the entire time-varying difference image synthesized from the component scores and the eigenfaces of these orders (after converting the pixel values into the magnitude of change), By performing a process of converting the pixel value to 0 (a process of filling it in black), it was generated so that pixels (parts) with high intensity were emphasized.
The scores of the 11th to 30th principal components in the 100-dimensional time-varying component score become the 2nd to 3rd elements (Euclidean norm) in the first stage of compression, and the 2nd to 3rd Since the element has a high contribution to the first element in the two-dimensional time-varying component score obtained in the second stage of compression, the first principal component corresponding image generated as described above has a two-dimensional time-varying component score. It can be said that it represents the characteristics of the first principal component of the score.
The second principal component corresponding image shown in FIG. 9 shows the 51st to 70th principal component scores and their After converting the pixel values into absolute values for the entire time-varying difference image synthesized from the eigenfaces of the order (after setting the pixel values to the magnitude of change), pixel values below a certain intensity are set to 0. By performing conversion processing (blacking out processing), pixels (parts) with high intensity were generated so as to be emphasized.
The scores of the 51st to 70th principal components in the 100-dimensional time-varying component score become the 6th to 7th elements (Euclidean norm) in the first stage of compression, and the 6th to 7th Since the element has a high contribution to the second element of the two-dimensional time-varying component score obtained in the second stage of compression, the second principal component corresponding image generated as described above has a two-dimensional time-varying component score. It can be said that this represents the characteristics of the second principal component of the score.
However, if the first principal component corresponding image and the second principal component corresponding image represent the characteristics of the first principal component or second principal component of the two-dimensional time-varying component score, a method other than the above method may be used. may be generated.

The first principal component corresponding image shown in FIG. 8 shows changes in a relatively larger area (global area) than the second principal component corresponding image shown in FIG. It can be seen that the second principal component corresponding image shown in FIG. 8 shows changes in a relatively smaller area (local area) than the first principal component corresponding image shown in FIG.
Furthermore, according to the first principal component corresponding image, it can be seen that cluster 1 has fewer global changes than clusters 2 and 3, and cluster 3 has more global changes than clusters 1 and 2. .
According to the second principal component corresponding image, clusters 1 and 3 have fewer local changes than cluster 2, and conversely, cluster 2 has more local changes than clusters 1 and 3. Recognize.

From the above, cluster 1 can be associated with the first temporal change characteristic type, which indicates that there are fewer global changes than clusters 2 and 3, and fewer local changes than cluster 2, regarding facial surface texture. , Cluster 2 is of the second temporal characteristic type indicating more global changes than Cluster 1 and less than Cluster 3, and more local changes than Clusters 1 and 3 regarding facial surface texture. Cluster 3 may be associated with a third temporal characteristic type indicating more global changes than Clusters 1 and 2 and less local changes than Cluster 2 with respect to facial surface texture. I can do it.

Therefore, according to the present example, the subject's 100-dimensional time-varying component score is obtained by applying the subject's face image data to the AI model, and the number of dimensions of the 100-dimensional time-varying component score is reduced to 2. A two-dimensional time-varying component score is obtained by stepwise compression, and the two-dimensional time-varying component score is applied to two linear discriminants to determine the cluster to which the subject's time-varying component score belongs. It has been proven that it is possible.
In addition, it was demonstrated that each cluster represents a classification according to age group, and is characterized by more or less global changes and more or less local changes in facial surface texture.
Based on the clusters determined for the subject, a temporal change characteristic type characterized by more or less local changes and more or less global changes in the facial surface texture is identified with respect to the subject's facial surface texture. It has also been demonstrated that it is possible.

10 Information processing device (temporal change characteristic determination device)
11 CPU
12 Memory 13 Input/output I/F
14 Communication unit 15 Display device 16 Input device

Claims (6)

A trained model that is obtained by machine learning using training data that includes multiple facial image datasets before and after changes over time, and that is capable of calculating temporal change features of facial surface texture of input facial image data. One or more processors that can be used
a data acquisition step of acquiring face image data of the subject;
a feature amount acquisition step of acquiring a time-varying feature amount of the facial surface texture of the subject calculated by the learned model based on the acquired facial image data;
a dimension compression step of compressing the number of dimensions of the acquired time-varying feature amount in multiple stages to obtain a two-dimensional time-varying feature amount;
The time-varying feature of the subject compressed into two dimensions in the dimensional compression step based on the distribution information of the two-dimensional time-varying feature of the facial surface texture obtained in advance with respect to the face image data group of the population. a type identification step of identifying a temporal change characteristic type that indicates a temporal change tendency of the facial surface texture of the subject by performing cluster determination on the subject;
Run
The specified time-varying characteristic type is one of a plurality of time-varying characteristic types each characterized by a degree of local change and a degree of global change in facial surface texture,
The cluster determination includes selecting a cluster corresponding to the temporal change characteristic amount of the subject from among a plurality of clusters based on the distribution information and each of which is associated with a mutually different temporal change characteristic type. Determine,
In the type identification step, a time-varying characteristic type corresponding to the determined cluster is identified;
A method for determining the characteristics of facial surface texture changes over time. The plurality of clusters include at least a first cluster, a second cluster, and a third cluster,
The first cluster has a first temporal change characteristic type indicating that, with respect to facial surface texture, the global change is less than the second cluster and the third cluster, and the local change is less than the second cluster. is associated with
Regarding facial surface texture, the second cluster has more global changes than the first cluster and less than the third cluster, and has more local changes than the first cluster and the third cluster. is associated with the second aging characteristic type that indicates
The third cluster is a third temporal change characteristic type indicating that, with respect to facial surface texture, global changes are more than the first cluster and the second cluster, and local changes are less than the second cluster. is associated with
The method for determining characteristics over time according to claim 1. In the dimensional compression step, two stages of compression are performed,
In the first stage of compression in the dimensional compression process, the element group of the temporal change feature is divided into a predetermined number so that each has the same number of elements, and the Euclidean norm of the element is calculated for each divided dimensional area. compress the time-varying feature amount to the predetermined number of dimensions by making it an element of the dimension domain,
In the second stage of compression in the dimensional compression step, the two-dimensional time-varying feature quantity is obtained by a dimensional compression process on the time-varying feature quantity of the predetermined number of dimensions.
The method for determining characteristics over time according to claim 1 or 2. the one or more processors,
an output step of outputting image data of the temporal change characteristic quantity of the facial surface texture of the subject calculated by the learned model, together with information on the temporal change characteristic type specified in the type identification step;
The method for determining characteristics over time according to any one of claims 1 to 3, further comprising performing the following. The one or more processors can refer to an information storage unit that stores one or both of skin care information and beauty treatment information in association with each of the plurality of temporal change characteristic types,
the one or more processors,
extracting, from the information storage unit, one or both of skin care information and beauty treatment information corresponding to the time-varying characteristic type identified in the type identifying step;
outputting one or both of the extracted skin care information or beauty treatment information;
The method for determining characteristics over time according to any one of claims 1 to 4, further comprising performing the following. A temporal change characteristic determination device that can utilize the learned model and can execute the temporal change characteristic determination method according to any one of claims 1 to 5.

Images