JP7495292B2 - Method for predicting facial changes over time - Google Patents

Description

The present invention relates to a technology for predicting and analyzing changes in a person's face over time using a trained model obtained through machine learning and facial image data.

The following Patent Document 1 discloses a technology for analyzing evaluation indices for the entire face with high precision by extracting a hemoglobin pigment image, a melanin pigment image, and a shadow image from a facial image by independent component analysis, and then applying frequency component separation and principal component analysis to these extracted images to obtain evaluation indices for melanin pigment factors and hemoglobin pigment factors that cause color unevenness, as well as for each factor of unevenness such as wrinkles and sagging.
The following Patent Document 2 discloses a method for predicting aging using a trained model obtained by machine learning. The trained model includes a shape aging model for predicting changes in face shape due to aging, a texture aging model for predicting changes in face surface texture due to aging, and a three-dimensional prediction model for predicting three-dimensional data from a two-dimensional image. This method extracts feature points of a target image, estimates a face direction in the target image using the extracted feature points, generates first three-dimensional data based on the three-dimensional prediction model and the estimated face direction, generates second three-dimensional data from the first three-dimensional data using the shape aging model, applies a texture aging model to the two-dimensional image generated based on the first three-dimensional data to generate an aging texture, and synthesizes the aging texture with the second three-dimensional data to generate an aging face model.
The following non-patent document 1 discloses a method for removing shadows from a face by pigment vector analysis, and for extracting hemoglobin and melanin components by applying independent component analysis to a face image from which the shadows have been removed.

JP 2015-5281 A JP 2016-194892 A

Norimichi Tsumura et al., "Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin," ACM Transactions on Graphics, Vol. 22, No. 3, pp. 770-779(2003). (Proceedings of ACM SIGGRAPH 2003)

The aging prediction method described above can predict aging texture from a two-dimensional face image, and other analysis methods can process face images to change the face surface texture or obtain a face evaluation index. These current methods only focus on changing the face surface texture of a face image or obtaining some evaluation index from a face image.
In the midst of this, the inventors came up with a new idea: by predicting what kind of texture differences initial differences in facial surface texture will manifest as over time, and presenting this in an easy-to-understand manner, it would be possible for users to objectively and easily grasp the effects of skin beautification methods and products (excluding medical procedures).

The present invention relates to a technique for predicting how differences in initial facial surface texture will appear as texture differences after changes over time, and presenting these in an easily understandable manner.
In this specification, the term "facial surface texture" refers to a spatial distribution feature derived from the surface or interior of facial skin. Examples of "facial surface texture" include features that generate a spatial distribution of the facial surface, such as uneven skin color, such as redness or blemishes, skin gloss, and uneven or rough skin surface, such as wrinkles, pores, and scales. Other examples include features that generate a spatial distribution of the facial surface, such as the state of adhesion of a coating film on skin to which a makeup cosmetic has been applied, uneven skin color, gloss, and uneven surface unevenness caused by sebum secretion.

In order to solve the above-mentioned problems, the aspects of the present invention employ the following configurations.
An aspect of the present invention relates to a method for predicting changes over time in a face, executed by one or more processors, that can utilize a trained model obtained by machine learning using training data including multiple two-dimensional facial image data sets before and after the passage of time, and that is capable of predicting at least facial surface texture information after changes over time from two-dimensional facial image data. This method for predicting changes in face over time includes a data acquisition step of acquiring two-dimensional facial image data of a subject; a first prediction step of acquiring first time-changed facial information including facial surface texture information after a change in the original face of the subject over time predicted by applying the acquired two-dimensional facial image data to the trained model; an image manipulation step of acquiring two-dimensional manipulated facial image data showing the image-manipulated face of the subject by applying a specific image processing to the acquired two-dimensional facial image data that changes at least a portion of the facial surface texture from the original face of the subject reflected in the facial image data; a second prediction step of acquiring second time-changed facial information including facial surface texture information after a change in the image-manipulated face of the subject predicted by applying the acquired two-dimensional manipulated facial image data to the trained model; and an output step of outputting information capable of comparing at least the facial surface texture regarding the original face and the image-manipulated face after the change over time to an output unit using the first time-changed facial information and the second time-changed facial information.
Further, another aspect of the present invention relates to, for example, a facial change over time prediction device (information processing device, computer) that executes the facial change over time prediction method according to the above aspect, to a program that causes a computer to execute the facial change over time prediction method according to the above aspect, and to a computer-readable storage medium on which such a program is recorded. This recording medium includes a non-transitory tangible medium.

The above aspect provides a technology that predicts how differences in initial facial surface texture will appear as texture differences after changes over time, and presents these in an easy-to-understand manner.

1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device capable of executing a face time-change prediction method according to an embodiment of the present invention. 4 is a diagram showing a process flow of a face time-dependent change prediction method according to the first embodiment; FIG. 11 is a diagram showing a processing flow relating to a modified example of the face time-dependent change prediction method according to the first embodiment. FIG. 11 is a diagram showing a process flow of a face time-dependent change prediction method according to the second embodiment. FIG. 11 is a diagram showing a process flow of a face time-dependent change prediction method according to the second embodiment. FIG. 13 is a diagram showing a composite image of various texture images and various homologous models. FIG. 13 is a diagram showing a part of a process flow of a face time-dependent change prediction method according to a third modified example.

A preferred embodiment of the present invention (hereinafter, referred to as the present embodiment) will be described below. Note that the present embodiment described below is merely an example, and the present invention is not limited to the configuration described below.

The method for predicting a change over time of a face according to this embodiment (hereinafter referred to as the present method) is executed by one or more processors included in one or more information processing devices.
FIG. 1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device 10 capable of executing the present method.
The information processing device 10 is a so-called computer, and includes a CPU 11, a memory 12, an input/output interface (I/F) 13, a communication unit 14, etc. The information processing device 10 may be a stationary PC (Personal Computer), or may be a portable terminal such as a portable PC, a smartphone, or a tablet.

The CPU 11 is a so-called processor, and in addition to a general CPU (Central Processing Unit), may also include an application specific integrated circuit (ASIC), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), etc. The memory 12 is a RAM (Random Access Memory), a ROM (Read Only Memory), or an auxiliary storage device (such as a hard disk).
The input/output I/F 13 can be connected to user interface devices such as a display device 15 and an input device 16. The display device 15 is a device such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display that displays a screen corresponding to drawing data processed by the CPU 11 or the like. The input device 16 is a device such as a keyboard, a mouse, etc. that accepts input of user operations. The display device 15 and the input device 16 may be integrated and realized as a touch panel.
The communication unit 14 communicates with other computers via a communication network and exchanges signals with other devices such as a printer, etc. A portable recording medium, etc. can also be connected to the communication unit 14.

The hardware configuration of the information processing device 10 is not limited to the example of FIG. 1. The information processing device 10 may include other hardware elements not shown. Furthermore, the number of each hardware element is not limited to the example of FIG. 1. For example, the information processing device 10 may have multiple CPUs 11. Furthermore, the information processing device 10 may be realized by multiple computers consisting of multiple housings.

The information processing device 10 can execute the present method by the CPU 11 executing a computer program stored in the memory 12. This computer program is installed from a portable recording medium such as a CD (Compact Disc) or memory card, or from another computer on a network, via the input/output I/F 13 or the communication unit 14, and is stored in the memory 12.

The information processing device 10 (CPU 11) can use a trained model obtained by machine learning using teacher data.
The "trained model" here is a model obtained by machine learning using training data, i.e., supervised learning, and can be expressed as an AI (Artificial Intelligence) model, a Machine Learning (ML) model, etc.
The trained model used in this embodiment may be a regression equation obtained by regression analysis, or may be a neural network model obtained by principal component analysis, deep learning, or the like, and the data structure and learning algorithm of the model are not limited. For example, the trained model is realized by a combination of a computer program and a parameter, or a combination of a plurality of functions and parameters. The face image is subjected to dimensional reduction and coding processing using, for example, principal component analysis. When the trained model is constructed with a neural network and the input layer, intermediate layer, and output layer are regarded as a unit of one neural network, the trained model may refer to one neural network or a combination of multiple neural networks. The trained model may also be composed of a combination of multiple multiple regression equations, or may be composed of one multiple regression equation.
The trained model may be stored in memory 12 within the information processing device 10, or may be stored in the memory of another computer that the information processing device 10 can access via communication.

In this way, the information processing device 10 can be described as a facial change over time prediction device that is capable of using a trained model and executing a facial change over time prediction method.
Hereinafter, the trained model used in this method will be referred to as an AI model. In addition, in the following description, this method will be described as being executed by the CPU 11.

[First embodiment]
A method for predicting change over time of a face according to a first embodiment (hereinafter, referred to as a first method) will be described below with reference to Fig. 2. Fig. 2 is a diagram showing a process flow of the method for predicting change over time of a face according to the first embodiment.

The AI model used in the first embodiment is obtained by machine learning using training data including multiple two-dimensional facial image data sets before and after time has passed, and is capable of predicting facial surface texture information after changes over time from two-dimensional facial image data.
Here, "the AI model is capable of predicting facial surface texture information after changes over time" means that facial surface texture information after changes over time can be obtained based on information output from the AI model.
The facial surface texture information generated using the AI model may be image data or data in a format other than image data as long as it indicates the texture of the facial surface. In this embodiment, the facial surface texture information will be described as image data.
In addition, the changes over time predicted by the AI model are changes over time such as age, season, day, morning and evening, and the length of time that has passed is not limited. The AI model may predict aging changes after 5 or 10 years, changes after one week or one month, changes after application of cosmetics, beauty treatments (excluding medical procedures), or beauty or health foods and beverages, or changes after a certain time has passed since applying makeup cosmetics. The changes over time predicted by such an AI model can be set by a face image data set before and after the passage of time that is used as training data.
The two-dimensional face image data included in the training data may be image data of an uncovered face or image data of a made-up face. For example, the training data may be a data set of image data of a made-up face immediately after applying makeup and image data of the made-up face after makeup has come off over time.
In addition, the training data is preferably formed only from facial image datasets of the same person before and after the passage of time, but may also include facial image datasets of two different people who may correspond before and after the passage of time, for example, because they have similar facial features or facial texture features, or a dataset may be formed from facial image data of either one person before or after the passage of time and facial image data of the other person generated by image processing of that facial image data.
However, it is desirable that the facial image data used in the training data is image data in which the orientation of the face is normalized and the positions of specific facial features (eyes, nose, mouth, etc.) of each individual in the population are normalized. Furthermore, the facial image data used as the training data may be not only visible light images, but also images captured in wavelength bands other than visible light, such as infrared light.

Various well-known technologies can be used for the data structure and learning method of the AI model, such as the generation method and learning method of the texture aging model disclosed in the above-mentioned Patent Document 2. For example, both a texture aging model using principal component analysis and a texture aging model using WAVELET transformation may be used, or either one may be used. In addition, the teacher data and the two-dimensional facial image data applied to the AI model may be converted into images in a cylindrical coordinate system other than a rectangular coordinate system.

The first method includes steps (S21) to (S27) as shown in FIG.
Step (S21) is a data acquisition step for acquiring two-dimensional face image data of a subject (hereinafter, may be referred to as an original face image BOP). In the following description, the face of the subject shown in the face image data acquired in step (S21) may be referred to as an original face.
The data of the original face image BOP is acquired as an image file in, for example, JPEG (Joint Photographic Experts Group) format, BMP (Bitmap image) format, TIFF (Tagged Image File Format) format, GIF (Graphic Interchange Format) format, or the like. However, the data format of the image is not limited, and may be a format in which brightness information of each pixel is listed. The acquired image may be a color image, or a grayscale image obtained by grayscaling a color image. The CPU 11 may acquire the original face image BOP from a camera that captured the face image, or may acquire the original face image BOP from another computer or a portable recording medium.

In the original face image BOP acquired in step (S21), it is preferable that the state of the subject of the image is consistent with that of the face image used in the training data of the AI model. For example, if the face image of the training data shows a face surface texture viewed from the front, it is preferable that the original face image BOP also shows the face surface texture viewed from the front of the subject.
Therefore, in step (S21), the acquired face image data may be subjected to a normalization process to match the acquired face image data with the face image data of the AI model, thereby acquiring the original face image BOP. In this normalization process, for example, the size and position of the face in the image data, the position of specific features of the face (eyes, nose, mouth, etc.) are normalized, and the background is removed.
When normalizing the positions of specific facial features (eyes, nose, mouth, etc.), in step (S21), a predetermined image recognition process may be performed on the acquired facial image data to automatically recognize the specific features such as the eyes, nose, mouth, eyebrows, etc. In this case, in step (S21), it is also possible to check whether the acquired facial image data is usable or not based on the presence or absence and positional relationship of the specific features indicated by the result of the automatic recognition.

Step (S22) is an image manipulation step of acquiring two-dimensional manipulated face image data (hereinafter, sometimes referred to as manipulated face image BCP) showing the image-manipulated face of the subject by applying specific image processing to the two-dimensional face image data acquired in step (S21).
The specific image processing applied in step (S22) is image processing that changes at least a portion of the facial surface texture from the original face of the subject depicted in the original face image BOP, and the specific processing content is not limited as long as it is processing that can impart a facial surface texture change to the original face image BOP of the subject that is suitable for the purpose of confirming differences after changes over time.
For example, if the purpose is to check the difference over time as the unevenness of skin tone increases or decreases, image processing that increases the redness of the skin, image processing that reduces the redness of the skin, image processing that creates blemishes, image processing that removes blemishes, etc. can be applied.
In addition, if the purpose is to check the difference over time as the skin gloss increases or decreases, image processing that increases the skin gloss or image processing that decreases the skin gloss can be applied.
In addition, if the purpose is to check the differences over time due to increases and decreases in the unevenness or roughness of the skin surface, image processing that forms surface wrinkles, image processing that eliminates surface wrinkles, image processing that enlarges pores, image processing that shrinks pores, etc. can be applied.

As an example of image processing that can realize such a change in facial surface texture, an example of the image processing is to perform independent component analysis on the original face image BOP to extract a hemoglobin pigment image, a melanin pigment image, and a shadow image, separate each pigment image into a high-frequency component image and a low-frequency component image using a multi-resolution analysis (such as a two-dimensional discrete wavelet transform), perform a predetermined modulation process on the necessary frequency component images, and then recombine them. For example, a manipulation face image BCP in which the redness of the skin is increased can be generated by performing a contrast enhancement process on the low-frequency component image of the hemoglobin pigment image. In addition, a manipulation face image BCP in which the redness of the skin is reduced can be generated by performing a filtering process that reduces the contrast on the low-frequency component image of the hemoglobin pigment image. In addition, a manipulation face image BCP in which blemishes are formed or blemishes are eliminated can be generated by performing the same process on the high-frequency component image of the melanin pigment image. In addition, a manipulation face image BCP in which the gloss is increased or decreased can be generated by changing the skewness of the density histogram of the shadow image. In addition, by using an edge emphasis filter, a Gaussian filter, or the like, it is possible to generate an operation face image BCP in which pores are enlarged or reduced.
Furthermore, the specific image processing applied in step (S22) may be image processing in which the two-dimensional face image data acquired in step (S21) is displayed on the display device 15, and the original face image BOP is changed by a user operation on the displayed face image. In this case, the user can make a desired change, such as erasing a specific irregularity such as a specific blemish or a nasolabial fold, by operating the input device 16 while visually checking the part to be changed. For such image processing, image processing adopted in existing image processing software may be used.

As described above, there are various types of change modes that change the facial surface texture. Therefore, the first method may further include a step of identifying a change mode that changes the facial surface texture in the original face of the subject from among the multiple change modes. In this case, each change mode is associated with an image process, and in step (S22), the CPU 11 can apply a specific image process corresponding to the identified change mode from among the multiple types of image processes to the two-dimensional face image data acquired in step (S21).
The CPU 11 may display information indicating a plurality of supported change modes on the display device 15, and allow the user to select a desired change mode. Then, a specific image process corresponding to the selected change mode may be applied to the original face image BOP, thereby generating an operated face image BCP.
This makes it possible to generate an manipulated face image BCP in which various characteristics of the facial surface texture, such as redness, spots, glossiness, wrinkles, pores, etc., of the skin are changed from the original face image BOP. As a result, as described below, it is possible to predict and present what kind of texture difference will appear as after changes over time due to differences in various characteristics of the facial surface texture. In addition, by identifying the change mode according to the efficacy of each cosmetic product, beauty treatment (excluding medical treatment), or food and drink for beauty or health, it becomes possible to allow the user to understand the efficacy of those products.
However, the CPU 11 may generate a plurality of operation face images BCP corresponding to a plurality of change modes by applying a plurality of specific image processes corresponding to a plurality of change modes to the original face image BOP. The CPU 11 may further change the operation face image BCP by displaying the operation face image BCP on the display device 15 and having the user indicate the degree of texture change. After displaying the operation face image BCP, the CPU 11 may also have the user select whether or not to proceed with the process with the operation face image BCP.

Step (S23) is a first prediction step for acquiring facial surface texture information (first time-varying facial information) of the subject's original face predicted after change over time by applying the two-dimensional facial image data (original facial image BOP) acquired in step (S21) to an AI model. In this embodiment, in step (S23), a predicted original facial image AOP is generated using the first time-varying facial information. In other words, the predicted original facial image AOP indicates a facial surface texture obtained by changing the facial surface texture of the subject's original face shown in the original facial image BOP over time based on a prediction by the AI model.

Step (S24) is a second prediction step of acquiring face surface texture information (second time-varying face information) of the image-manipulated face of the subject predicted by applying the two-dimensional manipulated face image data (manipulated face image BCP) acquired in step (S22) to an AI model. In this embodiment, in step (S24), a predicted manipulated face image ACP is generated using the second time-varying face information. In other words, the predicted manipulated face image ACP indicates a face surface texture obtained by changing the face surface texture of the image-manipulated face of the subject shown in the manipulated face image BCP over time based on a prediction by the AI model.

Step (S25) is a step of acquiring face surface texture difference information before and after a time-dependent change in the original face of the subject, and step (S26) is a step of acquiring face surface texture difference information before and after a time-dependent change in the image-operated face of the subject. In this embodiment, step (S25) acquires difference information between the original face image BOP acquired in step (S21) and the predicted original face image AOP acquired in step (S23), and step (S26) acquires difference information between the operation face image BCP acquired in step (S22) and the predicted operation face image ACP acquired in step (S24).
The facial surface texture difference information acquired in steps (S25) and (S26) may be any information indicating the difference between the facial surface textures, and in this embodiment, difference image data is acquired as the difference information. Hereinafter, the difference information acquired in step (S25) may be referred to as difference image OD, and the difference information acquired in step (S26) may be referred to as difference image CD.

Any known method can be used to obtain the facial surface texture difference information. For example, the difference information can be obtained by extracting difference data, extracting necessary information from the difference data, encoding the necessary information, and visualizing the encoded necessary information.
For example, the difference data can be extracted by comparing the images on a pixel-by-pixel or block-by-block basis to calculate the difference value (signed or absolute value) or ratio of the gray values, or by comparing and discriminating the gray values. The extracted difference data may contain unnecessary information, such as electronic noise or uneven skin information caused by lighting different from the original facial surface texture. Therefore, the unnecessary information can be removed by applying a frequency filter or a spatial filter to the extracted difference data. In addition, in order to extract only the necessary information from the difference data, extraction via morphological transformation may be performed.
The necessary information extracted in this way is coded by binarization, multi-value conversion (grading), etc., and the coded necessary information can be visualized by imaging or labeling it as differences in color or brightness.

Step (S27) is an output step of outputting facial surface texture difference information before and after the change over time of the original face of the subject acquired in step (S25) and facial surface texture difference information before and after the change over time of the image-manipulated face of the subject acquired in step (S26) in a manner that allows comparison. In this embodiment, a difference image OD regarding the original face of the subject and a difference image CD regarding the image-manipulated face of the subject are output in a manner that allows comparison.
The output in step (S27) may be realized by displaying the difference information (difference image) on the display device 15, by printing it on a printer device, or by sending it to a portable recording medium or to another computer via the communication unit 14.
In addition, as an output format, the CPU 11 may display (output) both sets of difference information (difference images) side by side so that they can be compared, or may display either one in a switchable manner, or may display them superimposed.

In addition to the difference images OD and CD, the CPU 11 may also display the original face image BOP and the operation face image BCP on the display device 15, and may further display a predicted original face image AOP and a predicted operation face image ACP on the display device 15.
In step (S27), the CPU 11 can also output a first superimposed texture information obtained by superimposing facial surface texture difference information before and after the change over time of the original face on the facial surface texture information of the original face, and a second superimposed texture information obtained by superimposing facial surface texture difference information before and after the change over time of the image-manipulated face on the facial surface texture information of the image-manipulated face, so as to be contrasted. In this case, in this embodiment, the CPU 11 can output a first superimposed image obtained by superimposing the difference image OD and the original face image BOP, and a second superimposed image obtained by superimposing the difference image CD and the original face image BOP or the manipulated face image BCP, so as to be contrasted.
According to this, difference information before and after the change over time is superimposed on the facial surface texture of the original face or the image-manipulated face, thereby improving the comprehension of the difference.

In this way, by outputting facial surface texture difference information for a subject before and after a change in the original face over time and facial surface texture difference information for before and after a change in the image-manipulated face over time in a manner that allows for comparison, the predicted changes in facial surface texture over time for each of the original face and the image-manipulated face can be presented in an easy-to-understand manner, and the changes in facial surface texture between the original face and the image-manipulated face can be presented in an easy-to-understand manner to allow for easy understanding of what kind of texture differences will appear as after the change over time.
Incidentally, the facial surface texture information after the predicted time-dependent change (the predicted original facial image AOP and the predicted operated facial image ACP) may indicate a texture state that is undesirable for the subject, and the subject may feel uncomfortable when he or she sees it. On the other hand, the facial surface texture difference information indicates a difference, making it difficult to recognize the facial surface texture itself.
Therefore, by outputting the difference information between the predicted facial surface texture information after time-dependent changes (the predicted original facial image AOP and the predicted operated facial image ACP) and not outputting the information, the subject is not made to feel uncomfortable, and the above-mentioned effect can be further enhanced.

In addition, by simply preparing an original face image BOP that shows the subject's face, it is possible to virtually generate an manipulated face image BCP by image processing, in which the characteristics of the facial surface texture, such as redness, blemishes, glossiness, wrinkles, pores, etc., of the skin are changed according to the purpose, without doing anything directly to the subject's face. Therefore, the subject can easily obtain the above-mentioned effects, and can easily understand the efficacy of each cosmetic product, beauty treatment (excluding medical procedures), or beauty or health food and drink.

[First Modification]
In the first embodiment described above, in steps (S25) and (S26), the difference information before and after the change over time for each of the facial surface textures of the original face and the image-operated face of the subject is obtained, but the difference information of the facial surface texture between the original face and the image-operated face before and after the change over time may be obtained. That is, facial surface texture difference information before the change over time indicating the difference between the facial surface texture information before the change over time of the original face corresponding to the two-dimensional facial image data and the facial surface texture information before the change over time of the image-operated face corresponding to the two-dimensional operated face image data, and facial surface texture difference information after the change over time indicating the difference between the facial surface texture information after the change over time of the original face and the facial surface texture information after the change over time of the image-operated face may be obtained. Specifically, difference information (difference image) is obtained between the original face image BOP obtained in step (S21) and the operated face image BCP obtained in step (S22), and difference information (difference image) is obtained between the predicted original face image AOP obtained in step (S23) and the predicted operated face image ACP obtained in step (S24).
In this case, in step (S27), in addition to or instead of the output described in the first embodiment, facial surface texture difference information before and after the change over time is output so as to be comparable.

In addition, in the first embodiment and the first modified example described above, the difference information is output in step (S27) so as to be comparable, but in addition to or instead of the difference information, facial surface texture information of the original face and the image-operated face after changes over time may be output so as to be comparable.
3 is a diagram showing a process flow relating to a modified example of the method for predicting a change over time of a face according to the first embodiment. In the example of Fig. 3, a predicted original face image AOP and a predicted operated face image ACP are output so as to be comparable to each other.
In this way, in step (S27), it is sufficient to output information that allows comparison of at least the facial surface textures of the original face and the image-manipulated face after changes over time.

[Second embodiment]
In the above-described first embodiment and its modified example, prediction of changes over time in facial surface texture information and acquisition of differences between pieces of facial surface texture information are performed based on two-dimensional facial image data (original facial image BOP) of a subject.
In the second embodiment, in addition to facial surface texture information, three-dimensional facial shape information of the subject is processed. The following describes a method for predicting changes over time of a face according to the second embodiment (hereinafter referred to as the second method), focusing on differences from the first embodiment. In the following description, the same contents as those of the first embodiment will be omitted as appropriate.

4 and 5 are diagrams showing the flow of processing in the face time-dependent change prediction method according to the second embodiment, with FIG. 4 showing the first half of the flow and FIG. 5 showing the latter half of the flow.
The AI model used in the second embodiment is obtained by machine learning using training data including multiple sets of two-dimensional facial image data before and after time and multiple sets of three-dimensional facial shape information, and is capable of predicting three-dimensional facial shape information after changes over time and facial surface texture information that matches that facial shape information from two-dimensional facial image data.
Here, "the AI model is capable of predicting facial shape information and facial surface texture information after changes over time" means that facial shape information and facial surface texture information after changes over time can be obtained based on information output from the AI model.
The three-dimensional face shape information used in the second embodiment may be information indicating the three-dimensional shape of the face, and the format thereof is not limited. However, the three-dimensional face shape information is preferably modeled information so that data of each individual is expressed by the same number of data points of the same topology in order to enable statistical processing. Furthermore, it is more preferable that the three-dimensional face shape information is expressed by a homology model defined so that each data point has the same anatomical meaning.
For this reason, the three-dimensional face shape information in the second embodiment is a homology model that standardizes and expresses the face shape of each individual using polygons with the same number of vertices and the same topological geometric structure that are anatomically associated, and is shown as three-dimensional coordinate data. In the following description, the three-dimensional face shape information may also be referred to as a homology model.

As shown in FIG. 4, the AI model in the second embodiment includes at least a three-dimensional prediction model, a texture time-varying model, and a shape time-varying model.
The three-dimensional prediction model can be obtained by machine learning using a pair of two-dimensional face image data and corresponding three-dimensional face shape information (homology model) as training data, and can predict the three-dimensional face shape information (homology model) from the two-dimensional face image data. Various well-known methods can be adopted for the data structure, generation method, and learning method of this three-dimensional prediction model, and for example, a three-dimensional prediction model generated by the method disclosed in the above-mentioned Patent Document 2 can be used.
The shape change over time model can be obtained by machine learning using a pair of three-dimensional face shape information (homology model) of the same person before and after the passage of time in a population of multiple people as training data, and it is possible to predict three-dimensional face shape information (homology model) after the change over time from the three-dimensional face shape information (homology model) predicted by the three-dimensional prediction model. Various well-known methods can be adopted for the data structure, generation method, and learning method of this shape change over time model, and for example, a three-dimensional prediction model generated by the method disclosed in the above-mentioned Patent Document 2 can be used.
The texture time-varying model is similar to the texture time-varying model in the above-mentioned first embodiment, except that the texture time-varying model in the second embodiment is obtained by machine learning using, as training data, facial surface texture information (facial images) that indicates the facial surface texture of each individual included in the population and that is normalized so as to fit the homology model of the face, and is capable of predicting facial surface texture information after time-varying from such facial surface texture information (facial images).
The changes over time predicted by the texture change model and shape change model in the second embodiment are as described in the first embodiment, and are changes that occur over time, such as with age, season, day, morning and evening, etc., and there is no limit to the length of time that has passed.

The second method includes steps (S41) to (S51) as shown in FIG.
Step (S41) and step (S42) are the same as step (S21) and step (S22) in the first embodiment. That is, step (S41) is a data acquisition step for acquiring two-dimensional face image data (original face image BOP) of the subject, and step (S42) is an image manipulation step for acquiring two-dimensional manipulated face image data (manipulated face image BCP) showing the image-manipulated face of the subject by applying specific image processing to the two-dimensional face image data acquired in step (S41).

Step (S43) is a step of acquiring first three-dimensional face information including facial shape information and facial surface texture information of the subject's original face before changes over time, predicted by applying the two-dimensional face image data (original face image BOP) acquired in step (S41) to a three-dimensional prediction model of the AI model.
In this embodiment, in step (S43), the original face texture image BOT and the original face homology model BOF are generated using the first three-dimensional face information. Specifically, first, the original face homology model BOF is generated from the original face image BOP using a three-dimensional prediction model. The original face homology model BOF is three-dimensional face shape information related to the original face of the subject, and is shown as mesh data (three-dimensional coordinate data). Next, the original face texture image BOT is generated from the original face image BOP using the generated original face homology model BOF. The original face texture image BOT is face surface texture information related to the original face of the subject, and is image information normalized to fit the original face homology model BOF. In this normalization, for example, the position of each feature point (specific part) of the original face image BOP is matched to the original face homology model BOF, texture information not included in the original face image BOP is complemented, and conversion to a cylindrical coordinate system is performed.

Step (S44) is a step of acquiring second three-dimensional face information including facial shape information and facial surface texture information before the image-operated face of the subject predicted by applying the two-dimensional manipulated face image data (operated face image BCP) acquired in step (S42) to a three-dimensional prediction model of the AI model.
In this embodiment, in step (S44), the operation face texture image BCT and the operation face homology model BCF are generated using the second three-dimensional face information. Specifically, first, the operation face homology model BCF is generated from the operation face image BCP using a three-dimensional prediction model. The operation face homology model BCF is three-dimensional face shape information related to the image operation face of the subject, and is shown as mesh data (three-dimensional coordinate data). Next, the operation face texture image BCT is generated from the operation face image BCP using the generated operation face homology model BCF. The operation face texture image BCT is face surface texture information related to the image operation face of the subject, and is image information normalized to fit the operation face homology model BCF. In this normalization, for example, the position of each feature point (specific part) of the operation face image BCP is matched to the operation face homology model BCF, texture information not included in the operation face image BCP is complemented, and conversion to a cylindrical coordinate system is performed.

In this way, the original face texture image BOT and the original face homology model BOF are three-dimensional facial information of the subject's original face before it changes over time, generated from two-dimensional facial image data of the subject's original face using the three-dimensional prediction model of the AI model, and the manipulated face texture image BCT and the manipulated face homology model BCF are three-dimensional facial information of the subject's image-manipulated face before it changes over time, generated from two-dimensional facial image data (manipulated face image data) of the subject's image-manipulated face using that three-dimensional prediction model.

Step (S45) is a step of acquiring first time-varying face information including face shape information and face surface texture information of the subject's original face after the time-varying from the first three-dimensional face information acquired in step (S43) including face shape information and face surface texture information before the time-varying. In this embodiment, the original face texture image BOT acquired in step (S43) is applied to the texture time-varying model of the AI model to generate a predicted original face texture image AOT, and the original face homology model BOF acquired in step (S43) is applied to the shape time-varying model of the AI model to generate a predicted original face homology model AOF.

Step (S46) is a step of acquiring second time-varying face information including face shape information and face surface texture information after the image-manipulated face of the subject changes over time from the second three-dimensional face information including face shape information and face surface texture information of the image-manipulated face of the subject acquired in step (S44) before the change over time. In this embodiment, the manipulated face texture image BCT acquired in step (S44) is applied to the texture time-varying model of the AI model to generate a predicted manipulated face texture image ACT, and the manipulated face homology model BCF acquired in step (S44) is applied to the shape time-varying model of the AI model to generate a predicted manipulated face homology model ACF.

Step (S47) is a step of acquiring face surface texture difference information before and after a time-dependent change in the original face of the subject, and step (S48) is a step of acquiring face surface texture difference information before and after a time-dependent change in the image-operated face of the subject. In this embodiment, step (S47) acquires difference information between the original face texture image BOT acquired in step (S43) and the predicted original face texture image AOT acquired in step (S45), and step (S48) acquires difference information between the operation face texture image BCT acquired in step (S44) and the predicted operation face texture image ACT acquired in step (S46).
The facial surface texture difference information acquired in steps (S47) and (S48) may be any information indicating the difference between the facial surface textures, and in this embodiment, difference image data is acquired as the difference information. Hereinafter, the difference information acquired in step (S47) may be referred to as a difference texture image TOD, and the difference information acquired in step (S48) may be referred to as a difference texture image TCD.
The method of acquiring the difference information in step (S47) and step (S48) is similar to that in step (S25) and step (S26) in the first embodiment.

Step (S49) is a step of combining facial surface texture difference information (difference texture image TOD) before and after the time-dependent change of the subject's original face obtained in step (S47) with the original face homology model BOF obtained in step (S43) or the predicted original face homology model AOF obtained in step (S45).
Step (S50) is a step of synthesizing facial surface texture difference information (difference texture image TCD) before and after the time-dependent change of the image-manipulated face of the subject obtained in step (S48) with the manipulated face homology model BCF obtained in step (S44) or the predicted manipulated face homology model ACF obtained in step (S46).
As a result, the three-dimensional facial image data generated in step (S49) shows a three-dimensional facial image having a three-dimensional facial shape of the original face before or after the change over time, and having difference information of the facial surface texture before and after the change over time of the original face as texture information. The facial image shown by this facial image data is represented as a three-dimensional difference image DOD.
On the other hand, the three-dimensional face image data generated in step (S50) shows a three-dimensional face image having a three-dimensional face shape before the image-manipulated face changes over time or a three-dimensional face shape after the image-manipulated face changes over time, and having difference information of the face surface texture before and after the image-manipulated face changes over time as texture information. The face image shown by this face image data is expressed as a three-dimensional difference image DCD.

Step (S51) is an output step of outputting the three-dimensional difference image DOD acquired in step (S49) and the three-dimensional difference image DCD acquired in step (S50) so as to be comparable. The output method and output form in step (S51) are the same as those in step (S27) in the first embodiment.
In this way, by outputting the difference information in the facial surface texture before and after the change over time for each of the original face and the image-manipulated face as three-dimensional difference images DOD and DCD that show the three-dimensional facial shape, it is possible to easily grasp the difference in the facial surface texture before and after the change over time.
The CPU 11 can output the three-dimensional difference images DOD and DCD as multi-viewpoint images. Examples of multi-viewpoint images include free viewpoint images in which the viewpoint can be freely changed by a user operation using the input device 16, and stereoscopic displays using a three-dimensional display, virtual reality (VR), augmented reality (AR), and the like.
In this way, by outputting the difference information as a multi-viewpoint video, the differences in facial surface texture can be confirmed from various viewpoints.
As a result, it is possible to present in an easy-to-understand manner how the difference in facial surface texture between the original face and the image-manipulated face appears as a difference in texture after a change over time.
In addition, by outputting the facial surface texture information after time-dependent changes predicted as the texture of the three-dimensional facial image (the predicted original facial texture image AOT and the predicted operated facial texture image ACT) as difference information rather than as is, the possibility of causing discomfort to the subject can be reduced.

Meanwhile, various three-dimensional face images can be generated by synthesizing various texture images and various homologous models acquired as described above, as shown in Fig. 6. Fig. 6 is a diagram showing a synthesis image of various texture images and various homologous models.
Specifically, by combining the original face texture image BOT and the original face homology model BOF, a three-dimensional face image BOD of the subject's original face before it changed over time can be generated, and by combining the manipulated face texture image BCT and the manipulated face homology model BCF, a three-dimensional face image BCD of the subject's image manipulated face before it changed over time can be generated.
In addition, by combining the predicted original face texture image AOT and the predicted original face homology model AOF, a three-dimensional face image AOD of the subject's original face after it has changed over time can be generated, and by combining the predicted manipulated face texture image ACT and the predicted manipulated face homology model ACF, a three-dimensional face image ACD of the subject's image manipulated face after it has changed over time can be generated.

In addition to the three-dimensional difference images DOD and DCD, the CPU 11 may display on the display device 15 three-dimensional facial images BOD and BCD of the original face and the image-manipulated face before the change over time so that they can be compared, and may further display on the display device 15 three-dimensional facial images AOD and ACD of the original face and the image-manipulated face after the change over time so that they can be compared.
In addition, in step (S51), the CPU 11 can also output a first superimposed difference face image that has been three-dimensionalized using the three-dimensional facial shape information of the original face (original face texture image BOT or predicted original face texture image AOT) by superimposing facial surface texture difference information (difference texture image TOD) before and after the change in the original face over time on the facial surface texture information of the original face (original face texture image BOT or predicted original face texture image AOT) and a second superimposed difference face image that has been three-dimensionalized using the three-dimensional facial shape information of the image manipulated face (manipulated face homology model BCF or predicted manipulated face homology model ACF) by superimposing facial surface texture difference information (difference texture image TCD) before and after the change in the image manipulated face over time on the facial surface texture information of the image manipulated face (manipulated face texture image BCT or predicted manipulated face texture image ACT) so that they can be compared.
According to this, difference information before and after the change over time is superimposed on the facial surface texture of the original face or the image-manipulated face, thereby improving the comprehension of the difference.

[Second Modification]
In the second embodiment described above, the three-dimensional difference image DOD and the three-dimensional difference image DCD are outputted in step (S51) so as to be contrasted, but instead of such a three-dimensional difference image, a three-dimensional face image AOD of the subject's original face after time-dependent change and a three-dimensional face image ACD of the subject's image-manipulated face after time-dependent change may be outputted so as to be contrasted (see FIG. 6). In this case, the three-dimensional face images AOD and ACD may each be displayed on the display device 15 as a multi-viewpoint video.
In addition, the predicted source face texture image AOT and the predicted operated face texture image ACT may be displayed on the display device 15 so that they can be compared, and the predicted source face homology model AOF and the predicted operated face homology model ACF can be compared.
This makes it possible to omit steps (S47) and (S48), and to compare the facial surface textures and facial shapes of the original face and the image-manipulated face after changes over time, respectively.

Moreover, difference information of the facial surface texture information and difference information of the facial shape information between the original face and the image-operated face before and after the change over time may be obtained. In this embodiment, difference information between the predicted original face texture image AOT and the predicted operated face texture image ACT is obtained, and difference information between the original face texture image BOT and the operated face texture image BCT is obtained, and the difference information may be output so as to be comparable.

[Third Modification]
Also, as shown in Fig. 7, in addition to or instead of the facial surface texture difference information, difference information of three-dimensional facial shape information before and after the change over time may be acquired. Fig. 7 is a diagram showing a part of the process flow of the facial change prediction method according to the third modified example.
In the third modified example, difference information of the face shape before and after the change over time of the original face of the subject is generated (S71), and difference information of the face shape before and after the change over time of the image-manipulated face of the subject is generated (S72). In this embodiment, difference information between the original face homology model BOF and the predicted original face homology model AOF is generated in step (S71), and difference information between the manipulated face homology model BCF and the predicted manipulated face homology model ACF is generated in step (S72).

The difference between homologous models can be calculated using, for example, a method using normal distance or a method using shortest distance. In the method using normal distance, the intersection point between the average normal from the polygon vertices of one homologous model and the face (triangle) of the other homologous model is calculated, and the distance to the shortest intersection point is calculated. In the method using shortest distance, the distance from the polygon vertices of one homologous model to the polygon vertices of the other homologous model is calculated, and the vertex with the shortest distance is searched for. However, the method of obtaining the difference between homologous models is not limited to such examples, and any well-known method can be used.

The facial shape difference information thus generated can be used as image information, and can be output as a three-dimensional facial image by combining it with a homologous model of the original face or the image-manipulated face. The three-dimensional facial image showing the facial shape difference information can also be displayed on the display device 15 as a multi-viewpoint video.
In this embodiment, the difference information between the original face homology model BOF and the predicted original face homology model AOF generated in the step (S71) is combined with the original face homology model BOF or the predicted original face homology model AOF to generate a three-dimensional face shape difference image DOK (S73). Also, the difference information between the operation face homology model BCF and the predicted operation face homology model ACF generated in the step (S72) is combined with the operation face homology model BCF or the predicted operation face homology model ACF to generate a three-dimensional face shape difference image DCK (S74).
In step (S51), in addition to or instead of the three-dimensional difference images DOD and DCD, a three-dimensional face shape difference image DOK and a three-dimensional face shape difference image DCK may be output so as to be comparable. In this case, the face shape difference images DOK and DCK may each be displayed on the display device 15 as a multi-viewpoint video.
This makes it possible to grasp not only the facial surface texture but also how differences in facial shape manifest themselves over time.

[Other Modifications]
The step (S22) of the first embodiment and the step (S42) of the second embodiment, i.e., the image manipulation step of acquiring two-dimensional manipulated face image data showing the image-manipulated face of the subject from two-dimensional face image data, can be modified as follows.
The first and second methods may further include a step of identifying areas of the subject's original face where the facial surface texture is to be changed, and the image manipulation steps (S22) and (S42) may be configured to acquire manipulated face image data showing an image manipulated face in which the facial surface texture of the identified areas has been changed from the subject's original face.

The specified area is not limited as long as it is a part of the face, but examples include wrinkles at the corners of the eyes, nasolabial folds, age spots in specific areas, pores around the nose, and uneven redness on the cheeks.
In the step of specifying the area to be changed, the CPU 11 may display on the display device 15 a facial image showing the original face based on the facial image data acquired in step (S21) or step (S41), and specify the area selected by a user operation as the area to be changed in facial surface texture. The CPU 11 may also display on the display device 15 a list of cosmetics, beauty treatments (excluding medical procedures), beauty or health foods and beverages, etc., and specify the area to be affected by the treatment or product selected from the list by the user operation as the area to be changed in facial surface texture.

The CPU 11 can also identify areas where the facial surface texture is to be changed by using at least facial surface texture information of the subject's original face after time-dependent changes, obtained using an AI model. In this case, in the first method, step (S23) is executed after step (S21) and before step (S22), and in the second method, steps (S43) and (S45) are executed after step (S41) and before step (S42). This allows facial surface texture information of the subject's original face after time-dependent changes (predicted original facial image AOP or predicted original facial texture image AOT) to be acquired before steps (S22) and (S42).

The identification method may be one that uses at least facial surface texture information after the subject's original face has changed over time, and examples of the identification method include the following.
The CPU 11 may display the facial surface texture information of the original face (the predicted original face image AOP or the predicted original face texture image AOT) as it is on the display device 15, and specify a part selected by a user operation from the facial surface texture information as a part for changing the facial surface texture. In the second method, the display device 15 may display a three-dimensional face image AOD after the time-dependent change of the original face of the subject, which is obtained by combining the predicted original face texture image AOT and the predicted original face homology model AOF.
In addition, facial surface texture difference information (difference image OD or difference texture image TOD) before and after the change in the original face over time, which is generated using facial surface texture information of the original face of the subject after the change over time, or a three-dimensional difference image DOD obtained by combining the difference texture image TOD with the original face homology model BOF or the predicted original face homology model AOF, may be displayed on the display device 15 to allow the user to select the area to be changed.
Furthermore, the CPU 11 may automatically identify the area where the facial surface texture is to be changed based on the facial surface texture difference information. For example, by referring to the facial surface texture difference information, the CPU 11 may detect an area where the difference is large or an area where a difference of a predetermined size exists over a wide range, and identify the detected area as the area where the facial surface texture is to be changed.
In this way, by using facial surface texture information after the subject's original face has changed over time to identify the areas where the facial surface texture is to be changed, it is possible to obtain operated facial image data in which the texture of the areas that are significantly affected by changes over time has been changed.

In the image manipulation steps (S22) and (S42), specific image processing may be applied to the two-dimensional face image data so that the texture of the specified part is changed. For example, a local area corresponding to the specified part is extracted from the entire area of the face surface texture shown in the two-dimensional face image data, and specific image processing (filtering, etc.) is applied to the local area, whereby manipulated face image data in which the face surface texture of the specified part has been changed can be obtained. However, there is no limitation on the method of partially changing the face surface texture.
In this way, according to this variant, it is possible to create differences in facial surface texture for desired parts of the face, and it becomes easy to understand what kind of texture differences these differences will manifest as after changes over time.

Some or all of the above-described embodiments and variations may also be specified as follows. However, the above-described embodiments and variations are not limited to the following descriptions.

<1>
one or more processors capable of utilizing a trained model obtained by machine learning using training data including a plurality of two-dimensional face image data sets before and after the passage of time, the trained model being capable of predicting at least face surface texture information after a change over time from the two-dimensional face image data;
a data acquisition step of acquiring two-dimensional facial image data of a subject;
A first prediction step of acquiring first time-varying face information including facial surface texture information after a time-varying change of the original face of the subject predicted by applying the acquired two-dimensional facial image data to the trained model;
an image manipulation step of acquiring two-dimensional manipulated face image data showing an image-manipulated face of the subject by applying a specific image processing to the acquired two-dimensional face image data, the specific image processing changing at least a part of a facial surface texture from the original face of the subject shown in the facial image data;
A second prediction step of acquiring second time-varying face information including face surface texture information after time-varying of the image-manipulated face of the subject predicted by applying the acquired two-dimensional manipulated face image data to the trained model; and
an output step of outputting, to an output unit, information that enables comparison of at least face surface textures regarding the original face and the image-operated face after the change over time, using the first time-change face information and the second time-change face information;
A method for predicting changes over time in a face.

＜2＞
A step of identifying a portion of the original face of the subject that changes the facial surface texture;
Further comprising:
In the image manipulation step, the manipulated face image data is obtained, which shows the image manipulated face obtained by changing the facial surface texture of the specified part from the original face.
The method for predicting a change in a face over time according to <1>.
＜3＞
In the identification step, the part is identified by using at least facial surface texture information included in the first time-varying face information acquired in the first prediction step.
The method for predicting a change in a face over time according to <2>.
＜4＞
identifying a change in the original face of the subject that changes the facial surface texture;
Further comprising:
In the image manipulation step, a specific image processing corresponding to the specified change mode among a plurality of types of image processing is applied to the acquired two-dimensional face image data.
The method for predicting a change in a face over time according to any one of <1> to <3>.
＜5＞
acquiring facial surface texture difference information before and after the time-dependent change of the original face, which corresponds to the acquired two-dimensional facial image data, indicating a difference between facial surface texture information before the time-dependent change of the original face and facial surface texture information after the time-dependent change of the original face, and facial surface texture difference information before and after the time-dependent change of the image manipulated face, which corresponds to the acquired two-dimensional manipulated facial image data, indicating a difference between facial surface texture information before the time-dependent change of the image manipulated face and facial surface texture information after the time-dependent change of the image manipulated face, or acquiring facial surface texture difference information before the time-dependent change, which corresponds to the acquired two-dimensional facial image data, indicating a difference between facial surface texture information before the time-dependent change of the original face, which corresponds to the acquired two-dimensional manipulated facial image data, and facial surface texture difference information after the time-dependent change, which indicates a difference between facial surface texture information after the time-dependent change of the original face and facial surface texture information after the time-dependent change of the image manipulated face,
In the output step, facial surface texture difference information before and after the change over time of the original face and facial surface texture difference information before and after the change over time of the image-operated face are output to the output unit so as to be contrasted, or facial surface texture difference information before the change over time and facial surface texture difference information after the change over time are output to the output unit so as to be contrasted.
The method for predicting a change over time in a face according to any one of <1> to <4>.
＜6＞
the trained model is a model obtained by machine learning using the training data further including a set of three-dimensional face shape information corresponding to the two-dimensional face image data set, and is capable of predicting, from the two-dimensional face image data, three-dimensional face shape information after a change over time and face surface texture information that matches the face shape information;
the first time-varying face information acquired in the first prediction step further includes three-dimensional face shape information after the original face has changed over time,
The second time-varying face information acquired in the second prediction step further includes three-dimensional face shape information after the image-operated face has changed over time,
In the output step, information enabling comparison of a facial surface texture and a facial shape of the original face and the image-manipulated face after the change over time is output to the output unit.
The method for predicting a change in a face over time according to any one of <1> to <5>.
＜７＞
In the output step, a multi-viewpoint image showing a facial surface texture and a facial shape of the original face after the time-dependent change and a multi-viewpoint image showing a facial surface texture and a facial shape of the image-manipulated face after the time-dependent change are output to the output unit so as to be contrasted.
The method for predicting a change in a face over time according to <6>.
＜8＞
the trained model is capable of predicting, from two-dimensional face image data, three-dimensional face shape information and face surface texture information that matches the face shape information, respectively, before and after a time-dependent change;
A step of acquiring first three-dimensional face information including face shape information and face surface texture information of the subject's original face before time-dependent changes predicted by applying the acquired two-dimensional face image data to the trained model;
A step of acquiring second three-dimensional face information including face shape information and face surface texture information before a time-dependent change of the image-manipulated face of the subject predicted by applying the acquired two-dimensional manipulated face image data to the trained model;
Further comprising:
In the output process, the first three-dimensional face information and the second three-dimensional face information are further used to output to the output unit a multi-viewpoint video showing a difference in facial surface texture before and after the time change of the original face and a facial shape of the original face before or after the time change, and a multi-viewpoint video showing a difference in facial surface texture before and after the time change of the image-manipulated face and a facial shape of the image-manipulated face before or after the time change in a manner that allows comparison.
The method for predicting a change in a face over time according to <6> or <7>.
＜9＞
In the output step, difference information on the face shape before and after the change over time of the original face and difference information on the face shape before and after the change over time of the image-manipulated face are output to the output unit so as to be comparable.
The method for predicting a change in a face over time according to <8>.
＜10＞
A facial change over time prediction device that can use the trained model and can execute the facial change over time prediction method described in any one of <1> to <9>.

In the process flows shown in the figures, each step is assigned a symbol, but the order in which the steps are executed in this embodiment is not limited to the order of the numbers in the symbols. In this embodiment, the order in which the steps are executed can be determined as appropriate to the extent that does not cause any problems in terms of content. In addition, the above-mentioned embodiments and modified examples can be combined to the extent that the content is not contradictory.

10 Information processing device (face change over time prediction device)
11 CPU
12 Memory 13 Input/Output I/F
14 Communication unit 15 Display device 16 Input device

Claims (10)

one or more processors capable of utilizing a trained model obtained by machine learning using training data including a plurality of two-dimensional face image data sets before and after the passage of time, the trained model being capable of predicting at least face surface texture information after a change over time from the two-dimensional face image data;
a data acquisition step of acquiring two-dimensional facial image data of a subject;
A first prediction step of acquiring first time-varying face information including facial surface texture information after a time-varying change of the original face of the subject predicted by applying the acquired two-dimensional facial image data to the trained model;
an image manipulation step of acquiring two-dimensional manipulated face image data showing an image-manipulated face of the subject by applying a specific image processing to the acquired two-dimensional face image data, the specific image processing changing at least a part of a facial surface texture from the original face of the subject shown in the facial image data;
A second prediction step of acquiring second time-varying face information including face surface texture information after time-varying of the image-manipulated face of the subject predicted by applying the acquired two-dimensional manipulated face image data to the trained model; and
an output step of outputting, to an output unit, information that enables comparison of at least face surface textures regarding the original face and the image-operated face after the change over time, using the first time-change face information and the second time-change face information;
A method for predicting changes over time in a face. A step of identifying a portion of the original face of the subject that changes the facial surface texture;
Further comprising:
In the image manipulation step, the manipulated face image data is obtained, which shows the image manipulated face obtained by changing the facial surface texture of the specified part from the original face.
The method for predicting facial changes over time according to claim 1 . In the identification step, the part is identified by using at least facial surface texture information included in the first time-varying face information acquired in the first prediction step.
The method for predicting facial changes over time according to claim 2 . identifying a change in the original face of the subject that changes the facial surface texture;
Further comprising:
In the image manipulation step, a specific image processing corresponding to the specified change mode among a plurality of types of image processing is applied to the acquired two-dimensional face image data.
The method for predicting a change over time of a face according to claim 1 . acquiring facial surface texture difference information before and after the time-dependent change of the original face, which corresponds to the acquired two-dimensional facial image data, indicating a difference between facial surface texture information before the time-dependent change of the original face and facial surface texture information after the time-dependent change of the original face, and acquiring facial surface texture difference information before and after the time-dependent change of the image manipulated face, which corresponds to the acquired two-dimensional manipulated facial image data, indicating a difference between facial surface texture information before the time-dependent change of the image manipulated face and facial surface texture information after the time-dependent change of the image manipulated face, or acquiring facial surface texture difference information before the time-dependent change, which corresponds to the acquired two-dimensional facial image data, indicating a difference between facial surface texture information before the time-dependent change of the original face, which corresponds to the acquired two-dimensional manipulated facial image data, and facial surface texture difference information after the time-dependent change, which indicates a difference between facial surface texture information after the time-dependent change of the original face and facial surface texture information after the time-dependent change of the image manipulated face,
In the output step, facial surface texture difference information before and after the change over time of the original face and facial surface texture difference information before and after the change over time of the image-operated face are output to the output unit so as to be contrasted, or facial surface texture difference information before the change over time and facial surface texture difference information after the change over time are output to the output unit so as to be contrasted.
The method for predicting a change over time of a face according to claim 1 . the trained model is a model obtained by machine learning using the training data further including a set of three-dimensional face shape information corresponding to the two-dimensional face image data set, and is capable of predicting, from the two-dimensional face image data, three-dimensional face shape information after a change over time and face surface texture information that matches the face shape information;
the first time-varying face information acquired in the first prediction step further includes three-dimensional face shape information after the time-varying face of the original face is obtained,
The second time-varying face information acquired in the second prediction step further includes three-dimensional face shape information after the image-operated face has changed over time,
In the output step, information enabling comparison of a facial surface texture and a facial shape of the original face and the image-manipulated face after the change over time is output to the output unit.
The method for predicting a facial change over time according to claim 1 . In the output step, a multi-viewpoint image showing a facial surface texture and a facial shape of the original face after the time-dependent change and a multi-viewpoint image showing a facial surface texture and a facial shape of the image-manipulated face after the time-dependent change are output to the output unit so as to be contrasted with each other.
The method for predicting facial changes over time according to claim 6. the trained model is capable of predicting, from two-dimensional face image data, three-dimensional face shape information and face surface texture information that matches the face shape information, respectively, before and after a time-dependent change;
A step of acquiring first three-dimensional face information including face shape information and face surface texture information of the subject's original face before time-dependent changes predicted by applying the acquired two-dimensional face image data to the trained model;
A step of acquiring second three-dimensional face information including face shape information and face surface texture information before a time-dependent change of the image-manipulated face of the subject predicted by applying the acquired two-dimensional manipulated face image data to the trained model;
Further comprising:
In the output process, the first three-dimensional face information and the second three-dimensional face information are further used to output to the output unit a multi-viewpoint video showing a difference in facial surface texture before and after the time change of the original face and a facial shape of the original face before or after the time change, and a multi-viewpoint video showing a difference in facial surface texture before and after the time change of the image-manipulated face and a facial shape of the image-manipulated face before or after the time change in a manner that allows comparison.
The method for predicting a facial change over time according to claim 6 or 7. In the output step, difference information on the face shape before and after the change over time of the original face and difference information on the face shape before and after the change over time of the image-manipulated face are output to the output unit so as to be comparable.
The method for predicting facial changes over time according to claim 8. A facial change over time prediction device that can use the trained model and executes the facial change over time prediction method described in any one of claims 1 to 9.

Images