WO2017006615A1 - Système de prédiction de vieillissement, procédé de prédiction de vieillissement, et programme de prédiction de vieillissement - Google Patents

Système de prédiction de vieillissement, procédé de prédiction de vieillissement, et programme de prédiction de vieillissement Download PDF

Info

Publication number
WO2017006615A1
WO2017006615A1 PCT/JP2016/063227 JP2016063227W WO2017006615A1 WO 2017006615 A1 WO2017006615 A1 WO 2017006615A1 JP 2016063227 W JP2016063227 W JP 2016063227W WO 2017006615 A1 WO2017006615 A1 WO 2017006615A1
Authority
WO
WIPO (PCT)
Prior art keywords
aging
model
dimensional
prediction
texture
Prior art date
Application number
PCT/JP2016/063227
Other languages
English (en)
Japanese (ja)
Inventor
永田　毅
和敏 松崎
秀正 前川
和彦 今泉
Original Assignee
みずほ情報総研 株式会社
警察庁科学警察研究所長が代表する日本国
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from JP2015137942A external-priority patent/JP5950486B1/ja
Application filed by みずほ情報総研 株式会社, 警察庁科学警察研究所長が代表する日本国 filed Critical みずほ情報総研 株式会社
Priority to KR1020177025018A priority Critical patent/KR101968437B1/ko
Priority to CN201680016809.0A priority patent/CN107408290A/zh
Publication of WO2017006615A1 publication Critical patent/WO2017006615A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present invention relates to an aging prediction system, an aging prediction method, and an aging prediction program for performing an aging simulation of a face image.
  • Patent Document 1 In beauty, a system for predicting changes in the shape of a face or body after aging has been studied (for example, see Patent Document 1).
  • a polygon mesh is constructed from an image acquired by scanning a face, the polygon mesh is re-parameterized, and a base mesh and a displacement image are calculated.
  • This displacement image is divided into a plurality of tiles, and the statistical value of each tile is measured.
  • the displacement image is deformed by changing the statistical value, and the deformed displacement image is combined with the base mesh to synthesize a new face.
  • An object of the present invention is to provide an aging prediction system, an aging prediction method, and an aging prediction program for efficiently and accurately performing an aging simulation of a face image.
  • an aging prediction system that solves the above problems includes a shape aging model that predicts changes due to aging of the face shape, a texture aging model that predicts changes due to aging of the texture of the face, A model storage unit that stores a three-dimensional prediction model that predicts three-dimensional data from a two-dimensional image, and a control unit that is configured to be connected to an input unit and an output unit and that predicts aging.
  • the control unit acquires a prediction target image from the input unit, extracts a feature point of the prediction target image, estimates a face direction in the prediction target image using the extracted feature point, and Generating first three-dimensional data based on the predicted prediction model and the estimated face orientation; generating second three-dimensional data from the first three-dimensional data using the shape aging model; Applying the texture aging model to the two-dimensional image generated based on the first three-dimensional data to generate an aging texture, and for the second three-dimensional data, An aging texture is synthesized to generate an aging face model, and a prediction process for outputting the generated aging face model to the output unit is executed.
  • an aging face model is produced
  • control unit obtains a face orientation angle designated for output from the input unit, and uses the generated aging face model to generate a two-dimensional face image of the face orientation angle. It may be further configured to generate and output the generated two-dimensional face image to the output unit. As a result, an image of an aging face with a designated face orientation can be output.
  • the two-dimensional image is a first two-dimensional image
  • the control unit generates a second two-dimensional image based on the acquired three-dimensional face sample data
  • the second 2D image A feature point is specified in a three-dimensional image, and the feature point is used to generate normalized sample data obtained by normalizing the three-dimensional face sample data.
  • the shape aging model and the It may be further configured to generate a texture aging model and execute a learning process for storing the generated shape aging model and the generated texture aging model in the model storage unit. Thereby, an aging model can be generated based on actual sample data.
  • the learning process may include generating the three-dimensional prediction model using the normalized sample data and storing it in the model storage unit. Thereby, a three-dimensional prediction model can be generated based on actual sample data.
  • the model storage unit stores a first texture aging model calculated using principal component analysis and a second texture aging model calculated using wavelet transform.
  • the control unit performs a first wavelet coefficient obtained by performing a wavelet transform on an image obtained by applying a first texture aging model to the first two-dimensional image, and a first wavelet coefficient for the first two-dimensional image.
  • the texture aging model to be applied may be further specified according to a result of comparison with the second wavelet coefficient to which the texture aging model of 2 is applied. This makes it possible to use the second texture aging model that is predicted by using existing stains, wrinkles, and the like, so that a more appropriate aging model can be generated.
  • a shape aging model that predicts changes in facial shape due to aging a texture aging model that predicts changes in facial texture due to aging, and 3D data from 2D images
  • an aging prediction system including a model storage unit storing a three-dimensional prediction model, and a control unit configured to be connected to an input unit and an output unit I will provide a.
  • the control unit acquires a prediction target image from the input unit, extracts a feature point of the prediction target image, estimates a face direction in the prediction target image using the extracted feature point, and Generating first three-dimensional data based on the predicted prediction model and the estimated face orientation; generating second three-dimensional data from the first three-dimensional data using the shape aging model; Applying the texture aging model to the two-dimensional image generated based on the first three-dimensional data to generate an aging texture, and for the second three-dimensional data, A prediction process for generating an aging face model by synthesizing aging textures and outputting the generated aging face model to the output unit is executed.
  • a shape aging model that predicts changes due to aging of the face shape a texture aging model that predicts changes due to aging of the texture of the face, and 3D data from 2D images are predicted.
  • a non-transitory computer-readable storage medium is provided.
  • the control unit acquires a prediction target image from the input unit, extracts a feature point of the prediction target image, and uses the extracted feature point to perform the prediction Estimating the face orientation in the target image, generating first three-dimensional data based on the three-dimensional prediction model and the estimated face orientation, and using the shape aging model, the first three-dimensional Generating second three-dimensional data from the data, applying the texture aging model to the two-dimensional image generated based on the first three-dimensional data, generating an aging texture, Synthesizing the aging texture with the second three-dimensional data to generate an aging face model, and executing a prediction process for outputting the generated aging face model to the output unit .
  • the flowchart of the process sequence of the learning process of the three-dimensional conversion of this embodiment The flowchart of the process sequence of the prediction verification process of the three-dimensional conversion of this embodiment. It is explanatory drawing explaining the verification result of the three-dimensional conversion of this embodiment, (a) is input data, (b) is a correct answer, (c) is predicted only by a two-dimensional feature point, (d) is 2 Face image when predicted with dimension feature points and image.
  • the flowchart of the process sequence of the prediction verification process of the texture aging of this embodiment Explanatory drawing explaining each main component in the prediction verification of the texture aging of this embodiment.
  • the aging of the face due to aging is learned using the three-dimensional face data before and after aging.
  • An aging simulation for predicting an aging face image is performed using the photographed two-dimensional face image.
  • an aging prediction system 20 is used.
  • An input unit 10 and an output unit 15 are connected to the aging prediction system 20.
  • the input unit 10 is a means for inputting various types of information, and includes a keyboard, a pointing device, an input interface for acquiring data from a recording medium, and the like.
  • the output unit 15 is a means for outputting various types of information, and includes a display or the like.
  • the aging prediction system 20 is a computer system for performing aging prediction processing.
  • the aging prediction system 20 includes a control unit 21, an aging data storage unit 22, a snapshot data storage unit 23, and a model storage unit 25.
  • the control unit 21 includes control means (CPU, RAM, ROM, etc.), and processes to be described later (a learning management stage, a first learning stage, a second learning stage, a third learning stage, a principal component analysis stage, a machine learning stage, (Aging management stage, first treatment stage, second treatment stage, third treatment stage, etc.).
  • the control unit 21 has a learning management unit 210, a first learning unit 211, a second learning unit 212, a third learning unit 213, a principal component analysis unit 214a, and a machine learning unit.
  • 214b an aging manager 215, a first processor 216, a second processor 217, and a third processor 218.
  • the learning management unit 210 executes a process of learning the secular change of the face in the aging simulation. As will be described later, the learning management unit 210 stores centroid calculating means for calculating the centroid position of both eyes using face feature points. Furthermore, the learning management unit 210 creates in advance data related to a generic model (basic model) used in a homologous modeling process for shapes to be described later, and stores the data in a memory.
  • This generic model is a model related to the face that represents the general characteristics of Japanese people. In the present embodiment, a generic model having 10741 mesh points and 21256 polygons (triangles) is used. This generic model includes mesh points corresponding to face feature points, and identification information for specifying each face feature point is set.
  • the learning management unit 210 stores a predetermined texture average calculation rule for calculating the average of the coordinates of each vertex of the normalized mesh model from the coordinates of the face feature points in the texture homology modeling process described later. ing.
  • the learning management unit 210 includes a learning memory for recording a cylindrical coordinate system image, a cylindrical coordinate system coordinate, and a homologous model used for learning.
  • the first learning unit 211 executes a first learning process for creating a model for predicting three-dimensional face data represented by a three-dimensional cylindrical coordinate system from two-dimensional face data (face image).
  • the 2nd learning part 212 performs the 2nd learning process which produces the model for predicting the change by aging about the texture (texture) of a face image.
  • the 3rd learning part 213 performs the 3rd learning process which produces the model for predicting the change by aging about face shape.
  • the principal component analysis unit 214a performs principal component analysis processing in response to instructions from the learning management unit 210 and the learning units 211 to 213.
  • the machine learning unit 214b performs a process of calculating an explanatory variable (a feature amount used at the time of prediction) using a dependent variable (a prediction target feature amount) according to an instruction from the learning units 211 to 213.
  • the aging management unit 215 executes processing for generating a face image after aging using a two-dimensional face image.
  • the aging management unit 215 acquires a two-dimensional face image to be predicted, and uses the first to third processing units 216 to 218 to perform an aging simulation of the texture and shape.
  • the first processing unit 216 executes processing for generating three-dimensional face data represented by a three-dimensional cylindrical coordinate system from the processing target two-dimensional face image.
  • the second processing unit 217 executes a process of predicting a change due to aging for the texture (texture) of the face image.
  • the texture aging process is executed using a texture aging model using principal component analysis and a texture aging model using wavelet (WAVELET) transformation.
  • the second processing unit 217 stores the weighting coefficient w used for this processing in the memory.
  • the weight coefficient w is a value for determining which model is to be emphasized when using a model using principal component analysis and a model using wavelet transform. In the present embodiment, “1” is used as the weighting coefficient w.
  • the 3rd process part 218 performs the process which estimates the change by aging about a face shape using 3D face data. Further, the aging data storage unit 22 stores three-dimensional face data (aging) before and after aging (10 years in this embodiment) for a predetermined number of learning subjects (samples used for learning). Data) is recorded. By using this aging data, changes before and after aging can be grasped. In this embodiment, data for about 170 people is used as the secular change data.
  • snapshot data storage unit 23 three-dimensional face data (snapshot data) obtained by photographing more learning subjects is recorded. This snapshot data is taken only once, and no aging data is recorded in the snapshot data storage unit 23. In this embodiment, data for about 870 people is used as snapshot data.
  • the model storage unit 25 stores a model (an algorithm for calculating a result variable using an explanatory variable) used when performing various predictions in an aging simulation.
  • a model an algorithm for calculating a result variable using an explanatory variable used when performing various predictions in an aging simulation.
  • data related to an angle prediction model, a three-dimensional prediction model, a texture aging model, and a shape aging model are stored.
  • model data for predicting the direction (angle with respect to the face front) where the face of the two-dimensional face image to be processed is photographed is stored.
  • the angle prediction model data is calculated and recorded by the first learning unit 211.
  • model data for converting a front-facing two-dimensional face image into three-dimensional face data is stored.
  • the three-dimensional prediction model data is calculated and recorded by the first learning unit 211.
  • model data for predicting the texture after aging is stored for the facial texture.
  • the texture aging model data is calculated and recorded by the second learning unit 212.
  • texture aging model data using principal component analysis and texture aging model data using wavelet transform are stored.
  • model data for predicting the shape after aging is stored for the face shape. This shape aging model data is calculated and recorded by the third learning unit 213.
  • the control unit 21 of the aging prediction system 20 generates the cylindrical coordinate system image D2 and the cylindrical coordinate system coordinate data D3 using the three-dimensional face data D1 stored in the aging data storage unit 22.
  • the cylindrical coordinate system image D2 is two-dimensional image data created by projecting three-dimensional face data onto a cylindrical coordinate system and interpolating into a “900 ⁇ 900” equidistant mesh.
  • the cylindrical coordinate system coordinate data D3 is data relating to the three-dimensional coordinates of each point of the “900 ⁇ 900” image generated by projecting the three-dimensional face data onto the cylindrical coordinate system.
  • the control unit 21 generates face feature point data D4 using the cylindrical coordinate system image D2 and the cylindrical coordinate system coordinate data D3.
  • the face feature point data D4 is data relating to the coordinates of the face feature points in the cylindrical coordinate system image D2. Details of the facial feature points will be described later.
  • the control unit 21 uses the cylindrical coordinate system coordinate data D3 to normalize the face feature point data D4 and the cylindrical coordinate system image D2.
  • a normalized cylindrical coordinate system image D5, normalized face feature point data D6, and three-dimensional mesh data D7 (homology model) are generated. Details of this normalization processing will be described later.
  • the homologous model is three-dimensional coordinate data in which three-dimensional data about a face is expressed by a mesh and converted so that the vertices of corresponding meshes in different data are anatomically the same position.
  • control unit 21 generates a two-dimensional face image D8 photographed from an arbitrary angle using the normalized cylindrical coordinate system image D5 and the three-dimensional mesh data D7 (homology model).
  • the control unit 21 changes the 2D face image to the 3D face data using the 2D face image D8 photographed from an arbitrary angle, the normalized face feature point data D6, and the 3D mesh data D7 (homology model).
  • the first learning process for the conversion is performed. Details of the first learning process will be described later.
  • control part 21 performs the 2nd learning process about texture aging using the normalized cylindrical coordinate system image D5. Details of the second learning process will be described later. Moreover, the control part 21 performs the 3rd learning process about three-dimensional shape aging using the three-dimensional mesh data D7 (homology model). Details of the third learning process will be described later.
  • control unit 21 executes conversion processing into a cylindrical coordinate system (step S1-1). Details of this processing will be described later with reference to FIGS.
  • control unit 21 executes face feature point extraction processing (step S1-2). Details of this processing will be described later with reference to FIG.
  • control unit 21 executes normalization processing of face feature points (step S1-3). Details of this processing will be described later with reference to FIGS.
  • control unit 21 executes homology modeling processing (step S1-4). Here, a face shape homology model and a texture homology model are generated. Details of these processes will be described later with reference to FIGS. 9 and 10.
  • control unit 21 executes a process for generating a normalized cylindrical coordinate system image (step S1-5). Details of this processing will be described later with reference to FIG. ⁇ Conversion processing to cylindrical coordinate system> Next, the conversion process to the cylindrical coordinate system (step S1-1) will be described with reference to FIGS.
  • the learning management unit 210 of the control unit 21 performs a missing portion interpolation process (step S2-1). Specifically, the learning management unit 210 checks whether there is a missing part in the three-dimensional face data. When the missing part is detected, the learning management unit 210 performs interpolation of the missing part using the peripheral information of the missing part. Specifically, the missing portion is compensated using a known interpolation method based on a predetermined range of data adjacent to the periphery of the missing portion.
  • the three-dimensional face data shown in FIG. 5B is represented by a cylindrical coordinate system having a radius and a cylindrical direction angle in the cylindrical coordinate system as two axes.
  • this three-dimensional face data some data around the chin and ears are missing.
  • the images of these portions are generated by interpolation processing using surrounding images and are made up for.
  • the learning management unit 210 of the control unit 21 executes a cylindrical coordinate system image generation process (step S2-2). Specifically, the learning management unit 210 projects the three-dimensional face data that has undergone the missing portion interpolation processing onto a cylindrical coordinate system (two-dimensional mapping). The learning management unit 210 interpolates the projected face image data into a “900 ⁇ 900” equally spaced mesh to generate a two-dimensional face image in a cylindrical coordinate system. The learning management unit 210 records the generated two-dimensional face image in the learning memory as a cylindrical coordinate system image D2.
  • FIG. 5C is a two-dimensional face image in which the three-dimensional face data projected onto the cylindrical coordinate system is represented by two axes (cylinder height and circumferential angle).
  • the learning management unit 210 of the control unit 21 executes cylindrical coordinate system coordinate generation processing (step S2-3). Specifically, the learning management unit 210 projects each coordinate (X, Y, Z) of the three-dimensional face data onto the cylindrical coordinate system, and for each point of the “900 ⁇ 900” image described above, Generate coordinate data (radial direction, angle, height). The learning management unit 210 records the generated two-dimensional face image in the learning memory as cylindrical coordinate system coordinate data D3.
  • the face feature point is a characteristic position (for example, the outermost point of the eyebrows, the innermost point of the eyebrows) in the face parts (eyebrows, eyes, nose, mouth, ears, cheeks, lower jaw, etc.) constituting the face , Mouth corner point, etc.).
  • 33 face feature points are used.
  • the face feature points may be arbitrarily added / deleted by the user. In this case, processing described later is performed using the added / deleted face feature points.
  • FIG. 6 shows the facial feature points (32) used in this embodiment with numbers.
  • the feature point number “33” is calculated as “the midpoint of a straight line connecting the center of gravity of the feature points of both eyes and the nose vertex” using the other feature points.
  • the learning management unit 210 of the control unit 21 specifies face feature points from the generated cylindrical coordinate system coordinates, and calculates the position (coordinates).
  • automatic extraction is performed using a well-known AAM (Active Appearance Models) method used for facial expression tracking, facial recognition, and the like.
  • AAM Active Appearance Models
  • a target object here, a face
  • this model is fitted to an input image to extract feature points of the target object.
  • the learning management unit 210 displays a face image in which the extracted face feature points are associated with the extraction position on the output unit 15.
  • the position of each face feature point is movably arranged.
  • the person in charge confirms the position of the facial feature point on the facial image displayed on the output unit 15 and corrects it if necessary.
  • face feature point confirmation or correction completion input is performed on the face image
  • the learning management unit 210 associates the cylindrical coordinate system coordinates of each face feature point with the number of each face feature point. D4 is generated and stored in the learning memory.
  • step S1-3 the facial feature point normalization process
  • the learning management unit 210 of the control unit 21 executes normalization processing by multiple regression analysis using the extracted face feature point data (step S3-1).
  • rotation on the XYZ axes is obtained from the feature points of the face by multiple regression analysis, and the face orientation is matched.
  • the size of the face is normalized so that the distance between the centers of the eyes becomes a predetermined value (64 mm in this embodiment). Details of this processing will be described later with reference to FIG.
  • the learning management unit 210 of the control unit 21 executes an average feature point calculation process (step S3-2). Specifically, the learning management unit 210 calculates the average coordinates of each feature point using the coordinates of 33 face feature point data for each learning target person. Thereby, the coordinates of each feature point in the average face of the learning subject are calculated.
  • the learning management unit 210 of the control unit 21 executes normalization processing by procrustes analysis (step S3-3). Specifically, the learning management unit 210 uses the least square method to minimize the sum of the square distance between the average coordinates calculated in step S3-2 and each facial feature point. Move, rotate, and resize. In this case, 25 points (feature point numbers 1 to 6, 10, 14 to 22, 24 in FIG. 6) excluding the tragus points (feature point numbers 7 and 13), mandibular corner points (feature point numbers 8 and 12), and the like. To 27 and 28 to 32) are used. As a result, the facial feature points are adjusted to be close to the average face.
  • step S3-1 normalization processing by multiple regression analysis
  • the learning management unit 210 of the control unit 21 executes the process of specifying the center of gravity of the feature points of both eyes (step S4-1). Specifically, the learning management unit 210 identifies facial feature points related to eyes in the facial feature point data. Next, the learning management unit 210 calculates the position of the center of gravity of both eyes using the center of gravity calculating means for the coordinates of the extracted face feature points. The learning management unit 210 specifies the calculated position of the center of gravity of both eyes as the origin.
  • the learning management unit 210 of the control unit 21 executes an inclination correction process around the X axis and the Y axis (step S4-2). Specifically, the learning management unit 210 sets the Z coordinate as an objective variable and the X and Y coordinates as explanatory variables for a set of face feature points excluding the face outline and nose vertex with the position of the center of gravity of both eyes as the origin. Multiple regression analysis.
  • a regression plane RPS is calculated by multiple regression analysis.
  • the learning management unit 210 rotates the regression plane RPS about the X and Y axes so that the calculated normal line NV of the regression plane RPS is parallel to the Z axis.
  • the learning management unit 210 of the control unit 21 executes an inclination correction process around the Z axis (step S4-3). Specifically, the learning management unit 210 uses a set of face feature points for calculating a face centerline. In this embodiment, a set of the center of gravity of the eyes, the apex of the nose, the lower end of the nose, the upper end of the upper lip, the lower end of the lower lip, and the chin tip coordinates are used as the facial feature points.
  • a regression line RL is calculated for this set using the Y coordinate as an objective variable and the X coordinate as an explanatory variable.
  • the learning management unit 210 rotates around the Z axis so that the calculated slope of the regression line RL is parallel to the Y axis.
  • the learning management unit 210 of the control unit 21 executes scaling processing (step S4-4). Specifically, the learning management unit 210 performs enlargement or reduction so that the distance between the centers of the eyes is 64 mm.
  • the learning management unit 210 of the control unit 21 executes a process of matching facial feature point coordinates (step S5-1). Specifically, the learning management unit 210 uses the generic model mesh point identification information stored in the memory to match the coordinates of each normalized facial feature point with the identified facial feature point of the mesh point.
  • the learning management unit 210 of the control unit 21 executes a shape fitting process (step S5-2). Specifically, the learning management unit 210 matches the shape of each facial part in the generic model in which each facial feature point is matched with the normalized shape of each facial part.
  • the learning management unit 210 of the control unit 21 executes a triangulation process (step S5-3). Specifically, the learning management unit 210 calculates the coordinates of each mesh point corresponding to each polygon (triangle) of the generic model in the normalized shape of each face part matched with the shape of the generic model ( The shape homology model) is stored in the learning memory.
  • the number of mesh points can be reduced by using a mesh model in which the mesh around the face parts such as eyes, nose and mouth is made fine and the other area meshes are widened. it can. Note that the forehead portion is deleted because the presence of bangs adversely affects the statistical processing.
  • the learning management unit 210 of the control unit 21 executes an average calculation process for the coordinates of each vertex of the normalized mesh model (step S6-1). Specifically, the learning management unit 210 calculates the average coordinates of each mesh point (vertex) from the normalized coordinates of each face feature point using a texture average calculation rule stored in advance.
  • the learning management unit 210 of the control unit 21 executes a process of transforming the texture on the cylindrical coordinate system two-dimensional polygon into an averaged two-dimensional polygon (step S6-2). Specifically, the learning management unit 210 transforms the texture (pixel information) on the polygon of each two-dimensional face data in the cylindrical coordinate system into the average coordinates calculated in step S6-1, and the texture at the deformed average coordinates. Is stored in the learning memory.
  • the learning management unit 210 of the control unit 21 calculates the texture average at each average coordinate, thereby calculating a texture homology model and stores it in the learning memory.
  • FIG. 10B shows an average face obtained by averaging textures transformed into average coordinates of each mesh model.
  • step S1-5 a process for generating a normalized cylindrical coordinate system image (step S1-5) will be described.
  • the cylindrical coordinate system image generated in step S2-2 cannot be analyzed as it is because the position of the face parts (eyes, nose, mouth, etc.) differs for each data. Therefore, the cylindrical coordinate system image is normalized so that the positions of the face parts of each data are aligned.
  • This image normalization mesh model uses 33 face feature points and pastes the mesh in a grid pattern on a cylindrical coordinate system.
  • a mesh model having 5588 mesh points and 10862 polygons (triangles) is used.
  • the learning management unit 210 of the control unit 21 calculates the average value of the image normalized mesh model and the average value of the texture of each polygon for all data.
  • an average face is generated from the calculated texture of the average value.
  • the mesh constituting the face matches the average face mesh. Therefore, the cylindrical coordinate system image is normalized.
  • the first learning unit 211 of the control unit 21 repeatedly executes the following processing for each predetermined processing target angle.
  • the first learning unit 211 of the control unit 21 performs a rotation process to a specified angle (step S7-1). Specifically, the first learning unit 211 rotates the three-dimensional homologous model to a predetermined target angle.
  • the first learning unit 211 stores the rotation angle when rotated to the predetermined target angle in the learning memory.
  • the first learning unit 211 of the control unit 21 executes a conversion process from the three-dimensional homologous model to the two-dimensional homologous model (step S7-2). Specifically, the learning management unit 210 generates a two-dimensional homology model by projecting the rotated three-dimensional homology model onto the XY plane.
  • the first learning unit 211 of the control unit 21 executes a two-dimensional feature point specifying process (step S7-3). Specifically, the first learning unit 211 of the control unit 21 specifies coordinates corresponding to face feature points in the three-dimensional homology model in the calculated two-dimensional homology model. The first learning unit 211 stores the identified face feature points in the learning memory as two-dimensional feature points.
  • the first learning unit 211 of the control unit 21 performs a process of excluding feature points hidden behind the face (step S7-4).
  • the first learning unit 211 includes a facial feature point on the photographing side (viewpoint side) and a facial feature point on the back side in the three-dimensional homologous model among the feature points of the two-dimensional homologous model. Identify The first learning unit 211 deletes the face feature points on the back side from the learning memory, and stores only the two-dimensional feature points on the photographing side.
  • the above processing is executed by repeating a loop for each angle to be processed.
  • the machine learning unit 214b of the control unit 21 executes machine learning processing (step S7-6). Specifically, the machine learning unit 214b uses “rotation angle ( ⁇ , ⁇ )” as a dependent variable (feature to be predicted) and “two-dimensional all data” as an explanatory variable (feature used in prediction). "The principal component score of the feature point divided by the standard deviation” is used. The machine learning unit 214b executes machine learning processing using the dependent variable and the explanatory variable. The first learning unit 211 records the angle prediction model generated by the machine learning unit 214b in the model storage unit 25.
  • ⁇ Machine learning process> the machine learning process will be described with reference to FIGS. 13 and 14.
  • another feature vector y (a feature amount used for prediction, which is an explanatory variable) is predicted from a certain feature vector x (a prediction target feature amount that is a dependent variable).
  • a model for predicting y i, j from x s (n), j is obtained by learning the relationship between y and x using multiple regression analysis. Specifically, “a i, s (n) ” and “b i ” in the following equation are calculated.
  • variable increase / decrease method (“stepwise method”)
  • the machine learning unit 214b of the control unit 21 executes an initial value setting process (step S8-1). Specifically, the machine learning unit 214b initializes a very large value to the minimum value (bic_min) of the Bayes information amount criterion stored in the memory, and resets the variable set (select_id) to an empty set.
  • the machine learning unit 214b of the control unit 21 executes a setting process for the current Bayes information amount standard (step S8-2). Specifically, the machine learning unit 214b substitutes the minimum value (bic_min) based on the Bayes information amount into the minimum value (bic_min_pre) based on the current Bayes information amount.
  • I is a dimension number selected as a processing target. In this iterative process, it is determined whether or not the principal component number i to be processed is a component to be added to the variable set (addition target component).
  • the machine learning unit 214b of the control unit 21 performs a determination process as to whether or not the minimum value of the correlation between the principal component number i and the variable set (select_id) is smaller than the maximum correlation coefficient Cmax (step). S8-3). Specifically, the machine learning unit 214b of the control unit 21 calculates a correlation coefficient between the principal component number i to be processed and the variable set (select_id) stored in the memory. The machine learning unit 214b compares the calculated correlation coefficient with the maximum correlation coefficient Cmax.
  • the machine learning unit 214b of the control unit 21 adds an additional target component to the variable set (select_id).
  • the multiple regression analysis is performed using the obtained variables, and the Bayes information criterion and the t value calculation process of the added variable are executed (step S8-4).
  • the machine learning unit 214b of the control unit 21 calculates a Bayesian information amount criterion by performing multiple regression analysis using a variable obtained by adding the principal component number i to be processed to the variable set stored in the memory.
  • the t value is calculated by a known t test.
  • the machine learning unit 214b of the control unit 21 performs a determination process as to whether or not the minimum value and t value of the Bayes information criterion satisfy the condition (step S8-5).
  • the calculated minimum value of the Bayes information amount criterion is larger than the current Bayes information amount criterion, and the t value is “2” or more.
  • the machine learning unit 214b of the control unit 21 sets the minimum value of the Bayes information amount criterion and Update processing of the principal component number is executed (step S8-6). Specifically, the machine learning unit 214b of the control unit 21 substitutes the Bayes information criterion (bic) for the minimum value (bic_min) of the Bayes information criterion. Furthermore, the main component number i to be processed is stored as an additional target component (add_id).
  • Step S8-3 when the minimum value of the correlation between the principal component number i and the variable set (select_id) is greater than or equal to the maximum correlation coefficient Cmax (“NO” in step S8-3), the machine learning unit 214b of the control unit 21 Steps S8-4 to S8-6 are skipped.
  • step S8-5 If either the minimum value of the Bayes information criterion or the t value does not satisfy the condition (“NO” in step S8-5), the machine learning unit 214b of the control unit 21 performs the process in step S8-6. To skip.
  • the machine learning unit 214b of the control unit 21 determines whether or not the minimum value of the Bayes information amount criterion has been updated. Processing is executed (step S8-7). Specifically, the machine learning unit 214b determines whether or not the minimum value (bic_min) of the Bayes information amount reference matches the current minimum value (bic_min_pre) of the Bayes information amount reference set in step S8-2. Determine. The machine learning unit 214b determines that the minimum value of the Bayes information criterion is not updated when they match, and determines that the minimum value of the Bayes information criterion is updated when they do not match.
  • step S8-7 when the minimum value of the Bayes information criterion is updated (in the case of “YES” in step S8-7), the machine learning unit 214b of the control unit 21 executes variable update processing (step S8-8). . Details of this variable update processing will be described with reference to FIG.
  • the machine learning unit 214b of the control unit 21 executes a process for determining whether or not the variable update is successful (step S8-9). Specifically, the machine learning unit 214b determines the variable update success based on the flags (variable update success flag, variable update failure flag) recorded in the memory in the variable update process described later.
  • step S8-9 when the variable update success flag is recorded in the memory and it is determined that the variable update is successful (in the case of “YES” in step S8-9), the machine learning unit 214b of the control unit 21 performs step S8— Steps 2 and after are repeated.
  • variable update process (step S8-8) will be described with reference to FIG.
  • this variable set is specified as the explanatory variable.
  • the machine learning unit 214b executes a new variable set setting process (step S9-1). Specifically, the machine learning unit 214b adds a component to be added (add_id) to the variable set stored in the memory to generate a new variable set (select_id_new).
  • the machine learning unit 214b repeats the following steps S9-2 to S9-4 in an infinite loop.
  • the machine learning unit 214b performs a multiple regression analysis using a new variable set (select_id_new), and executes a Bayes information criterion (bic) and t value calculation processing for all variables (step S9-2).
  • the machine learning unit 214b calculates a Bayes information criterion by multiple regression analysis using a new variable set. Further, the t value is calculated by a known t test.
  • the machine learning unit 214b determines whether or not the minimum t value among the t values of each variable of the new variable set is smaller than 2 (step S9-3).
  • the machine learning unit 214b executes the process of deleting the variable having the minimum t value from the new variable set. (Step S9-4). Specifically, the machine learning unit 214b deletes the variable for which the minimum t value has been calculated in the new variable set from the new variable set.
  • step S9-2 The processes after step S9-2 are repeatedly executed.
  • the machine learning unit 214b executes a process of determining whether or not the Bayes information amount criterion is smaller than the current minimum value of the Bayes information amount criterion (step S9-5). Specifically, the machine learning unit 214b determines whether or not the Bayes information amount criterion (bic) is smaller than the current minimum value (bic_min_pre) of the Bayes information amount criterion set in step S8-2.
  • the machine learning unit 214b executes a variable update success process (Step S9). -6). Specifically, the machine learning unit 214b substitutes a new variable set (select_id_new) for the variable set (select_id). Further, the machine learning unit 214b records a variable update success flag in the memory.
  • the machine learning unit 214b executes variable update failure processing (Step S9—). 7). Specifically, the machine learning unit 214b records a variable update failure flag in the memory.
  • the principal component analysis unit 214a of the control unit 21 performs a three-dimensional shape principal component analysis process in advance using a three-dimensional mesh (homology model) (step S10-1). Specifically, the principal component analysis unit 214a performs principal component analysis on the three-dimensional mesh points of each data. As a result, the two-dimensional mesh point is expressed by the following equation when expressed by an average value, a principal component score, and a principal component vector.
  • the first learning unit 211 of the control unit 21 executes the rotation process to the specified angle (step S10-2). Specifically, the first learning unit 211 displays a screen for designating the rotation angle on the output unit 15. Here, for example, the front (0 degree) and the side (90 degrees) are designated.
  • the rotation angle is input, the first learning unit 211 rotates the three-dimensional homologous model according to the input rotation angle.
  • the first learning unit 211 of the control unit 21 executes a process for generating a two-dimensional homologous model from the three-dimensional homologous model (step S10-3). Specifically, the first learning unit 211 generates a two-dimensional homology model by projecting the rotated three-dimensional homology model onto a two-dimensional plane (XY plane).
  • the first learning unit 211 of the control unit 21 executes a two-dimensional image generation process (step S10-4).
  • a gray two-dimensional homology model is used.
  • the first learning unit 211 generates a grayed image based on the luminance in each mesh of the generated two-dimensional homologous model.
  • the principal component analysis unit 214a of the control unit 21 executes a principal component analysis process of the two-dimensional image (step S10-5). Specifically, the principal component analysis unit 214a performs principal component analysis on the two-dimensional image generated in step S10-4 and expresses it as follows.
  • the first learning unit 211 of the control unit 21 executes a two-dimensional feature point specifying process (step S10-6). Specifically, the first learning unit 211 specifies the coordinates of the facial feature points in the calculated two-dimensional homology model, as in step S7-3. The first learning unit 211 stores the specified coordinates in the memory.
  • the first learning unit 211 of the control unit 21 executes a feature point exclusion process that hides behind the face (step S10-7). Specifically, the first learning unit 211 deletes the face feature points hidden behind the face from the memory, as in step S7-4.
  • the principal component analysis unit 214a of the control unit 21 executes a principal component analysis process of the two-dimensional feature points (step S10-8). Specifically, the principal component analysis unit 214a executes principal component analysis processing using the facial feature points stored in the memory, as in step S7-5.
  • the machine learning unit 214b of the control unit 21 executes machine learning processing in the same manner as in step S7-6 (step S10-9). Specifically, the machine learning unit 214b executes machine learning processing using the dependent variable and the explanatory variable.
  • the principal component score of the three-dimensional mesh point divided by the standard deviation is used as the dependent variable
  • the two-dimensional feature point of all data and the principal component score of the two-dimensional image of all data are used as explanatory variables.
  • “Divided by standard deviation” is used.
  • the first learning unit 211 records the three-dimensional prediction model generated by the machine learning unit 214b in the model storage unit 25.
  • the maximum correlation coefficient is secured to approximately 0.2 or more for the principal components up to the 100th, but decreases to a value having little correlation in the principal components after the 200th. .
  • the correlation tends to be small as compared with the two-dimensional feature point, but the maximum correlation coefficient is secured to about 0.1 in the 200th and subsequent principal components.
  • the F value is a parameter indicating the validity of the model.
  • the t value is a parameter indicating the validity of each variable. Although it is considered that the F value and the t value are each “2” or more, it is found that a value of “2” or more is secured in any component.
  • the coefficient of determination is a parameter indicating the explanatory power of the model, and the value indicates the rate at which the model explains the prediction target data. Specifically, when the value is “1”, it is “all predictable”, and when the value is “0”, it indicates “not predicted at all”.
  • the coefficient of determination is ensured to be approximately 50% or more for the main components up to the 40th, but less than 20% for the main components near the 100th.
  • the coefficient of determination was ensured to be approximately 50% or more for the principal components up to the 50th, and exceeded 20% even for the principal component near the 100th. As a result, it can be seen that the accuracy in the case of using the two-dimensional feature point and the image is improved as compared with the case of only the two-dimensional feature point.
  • the maximum correlation coefficient Cmax is set to “0.15” as a variable selection criterion.
  • the validity verification processing of the prediction model data to be converted from the two dimensions to the three dimensions is executed.
  • the first learning unit 211 of the control unit 21 executes a process of creating a prediction model machine-learned with the remaining [n-1] human data excluding the processing target data j (step S11-1). Specifically, the first learning unit 211 generates the three-dimensional conversion model by executing the first learning process described above using the data of [n ⁇ 1] people.
  • the first learning unit 211 of the control unit 21 performs an estimation process using a prediction model for the three-dimensional mesh point of the processing target data j (step S11-2). Specifically, the first learning unit 211 uses the processing target data j as input data and applies the generated three-dimensional conversion model to calculate a three-dimensional mesh point.
  • the first learning unit 211 of the control unit 21 performs a comparison process between the three-dimensional data (correct answer) of the processing target data j and the estimated result (step S11-3). Specifically, the first learning unit 211 compares the three-dimensional face data generated in step S11-1 with the three-dimensional face data of the processing target data j. As a result of the comparison, the shift amount of each point of the three-dimensional mesh is recorded in the memory. In this case, the average prediction error of the principal component scores was less than “0.22”. Since the variance of the principal component score is normalized to “1”, it can be seen that the estimation can be performed with high accuracy. It has also been found that the prediction using only the two-dimensional feature points and the prediction using the two-dimensional image can improve accuracy.
  • the average prediction error of the 3D mesh points is “1.546391 mm” when only 2D feature points are used, and “1.477514 mm” when 2D feature points and 2D images are used. .
  • the prediction with only the two-dimensional feature points and the prediction with the two-dimensional image can improve accuracy.
  • FIG. 17A shows a two-dimensional face image (input data) before aging.
  • FIG. 17B is a face image (correct answer) after 10 years of the face image shown in FIG.
  • FIG. 17C shows an aging face image predicted using only the two-dimensional feature points of the face image shown in FIG.
  • FIG. 17D is an aging face image predicted using the two-dimensional feature points and images of the face image shown in FIG.
  • the second learning process for texture aging is executed using FIG.
  • a texture aging process using principal component analysis and a texture aging process using wavelet transform are executed.
  • the texture aging process using wavelet transform will be described.
  • ⁇ Aging process of texture using principal component analysis> using the normalized cylindrical coordinate system image generated in step S1-5, a model for predicting the texture change due to aging in the three-dimensional face data is calculated by machine learning.
  • the control unit 21 executes principal component analysis processing of the normalized cylindrical coordinate system image (step S12-1). Specifically, the second learning unit 212 of the control unit 21 acquires aging data from the aging data storage unit 22 and snapshot data from the snapshot data storage unit 23.
  • the principal component analysis unit 214a of the control unit 21 performs principal component analysis on a cylindrical coordinate system image (a cylindrical coordinate system image of acquired secular change data and snapshot data). In this case, the principal component analysis unit 214a determines the direction of the principal component vector using the snapshot data before (or after) the aging data, and uses the aging data to determine the principal component. Score is calculated.
  • each data is expressed as an average value, a principal component score, and a principal component vector as follows.
  • j is an aging index
  • control unit 21 executes machine learning processing as in step S7-6 (step S12-2).
  • the machine learning unit 214b of the control unit 21 uses “an aging change difference vector of texture principal component scores normalized per unit year” as a dependent variable, and “pre-aging” as an explanatory variable.
  • the “principal score of texture” is used.
  • the second learning unit 212 records the texture aging model generated by the machine learning unit 214b in the model storage unit 25.
  • the cumulative contribution ratio up to the 35th principal component calculated in texture aging using the principal component analysis shown in FIG. 18 exceeds 95%, and the contribution ratio of the 25th and subsequent principal components is 0.1%. Is less than.
  • Each main component is shown in FIG. It can be seen that the higher the contribution ratio, the higher the frequency component.
  • FIG. 21 shows an image reconstructed by changing the upper limit of the principal component of two images for confirming the contribution of each principal component with the image.
  • the validity verification process of the texture aging prediction model data using this principal component analysis is executed.
  • control unit 21 executes a process for creating a prediction model that has been machine-learned with the data of the remaining [n ⁇ 1] people excluding the data j (step S13-1). Specifically, the second learning unit 212 of the control unit 21 executes the second learning process in steps S12-1 to S12-2 using [n-1] data, thereby aging the texture. Generate a conversion model of.
  • control unit 21 executes an aging process using the prediction model using the pre-aging data of the data j (step S13-2).
  • the second learning unit 212 of the control unit 21 uses the data j before aging as input data, applies the generated conversion model to texture aging, and applies the three-dimensional mesh points after aging. Is calculated.
  • control unit 21 executes a comparison process between the post-aging data (correct answer) of the data j and the aging result (step S13-3). Specifically, the second learning unit 212 of the control unit 21 performs post-aging change of the three-dimensional face data after aging generated in step S13-2 and the data j stored in the aging data storage unit 22. And the error of each point of the three-dimensional mesh is calculated. In this case, the calculated error was found to be about 60% or less.
  • texture aging processing using wavelet transform will be described with reference to FIG.
  • aging difference data is estimated.
  • aging using principal component analysis does not increase them. Therefore, in order to age using existing stains and wrinkles, aging change estimation using wavelet transform is performed.
  • the control unit 21 performs a calculation process of an increase rate (wavelet coefficient Ri) due to aging of the wavelet component (step S14-1).
  • the second learning unit 212 of the control unit 21 extracts the aging data stored in the aging data storage unit 22.
  • the second learning unit 212 calculates all the wavelet coefficients Ri of each two-dimensional image with the data number j for each wavelet coefficient number i (for each pixel).
  • the second learning unit 212 sums up the calculated wavelet coefficients Ri (values for each pixel) of the calculated image data before aging.
  • the second learning unit 212 sums up the calculated wavelet coefficients Ri (values for each pixel) of the image data after aging.
  • the wavelet coefficient Ri (value per pixel) in the summed image after aging by the wavelet coefficient Ri (value per pixel) in the summed image before aging. Calculate the rate of change. In this case, when the wavelet coefficient Ri is less than 1, the second learning unit 212 calculates the rate of change as “1”.
  • FIG. 22A shows an image displayed by visualizing the wavelet coefficient Ri.
  • black indicates a wavelet coefficient Ri having a minimum value of “1”, and the whiter the value, the larger the value.
  • the image shows the low frequency component as it goes to the upper left. Specifically, one-dimensional wavelet transform in the horizontal direction is performed for each row to separate the low-frequency component and the high-frequency component, and one-dimensional conversion in the vertical direction is performed on each column of the converted signal. The image which repeated is shown.
  • ⁇ 3D shape aging learning process a third learning process for aging the three-dimensional shape is executed using FIG.
  • a model for predicting a shape change due to aging in a three-dimensional face image is calculated by machine learning using the homology model generated in the above-described shape homology modeling process.
  • the maximum correlation coefficient Cmax between the selected variables is set to “0.15”.
  • the control unit 21 executes the calculation process of the principal component score of the three-dimensional mesh (step S15-1). Specifically, the third learning unit 213 of the control unit 21 extracts the aging data stored in the aging data storage unit 22. Here, 144 pieces of aging data are extracted. The third learning unit 213 uses the principal component vector of the three-dimensional mesh generated in the principal component analysis of the three-dimensional mesh point in step S10-1 described above, and uses the three-dimensional mesh principal component score for the extracted secular change data. Is calculated.
  • control unit 21 executes machine learning processing as in step S7-6 (step S15-2). Specifically, the machine learning unit 214b of the control unit 21 executes machine learning processing using the dependent variable and the explanatory variable.
  • the aging change difference vector of the three-dimensional mesh principal component score normalized per unit year is used as the dependent variable
  • the principal component score of the three-dimensional mesh before aging is used as the explanatory variable.
  • the third learning unit 213 records the shape aging model generated by the machine learning unit 214b in the model storage unit 25.
  • the maximum value of the correlation coefficient between the aging change difference vector calculated in this way and the principal component score has a correlation of about “0.3”, and there is a constant correlation between the aging change and the principal component score. Therefore, it is appropriate to use for regression analysis.
  • the number of selected variables is around 30, which is reasonable when compared with the number of secular change data used for calculation. Further, it is found that the F value is 10 or more, the t value is 2 or more, and the determination coefficient is almost 70% or more in any principal component, and it is understood that the accuracy of the model is high.
  • the control unit 21 executes feature point extraction processing (step S16-1). Specifically, the aging management unit 215 of the control unit 21 executes face feature point extraction processing from the processing target two-dimensional face image data in the same manner as in step S1-2.
  • control unit 21 executes face orientation extraction processing (step S16-2). Specifically, the first processing unit 216 of the control unit 21 uses the angle prediction model stored in the model storage unit 25 to specify the direction in which the face was photographed from the coordinates of the extracted face feature points, A two-dimensional face image is converted to the front direction.
  • control unit 21 executes a three-dimensional meshing process (step S16-3). Specifically, the aging management unit 215 of the control unit 21 generates a three-dimensional mesh for a front-facing two-dimensional face image using the three-dimensional prediction model stored in the model storage unit 25. .
  • the control unit 21 executes a process for generating a normalized cylindrical coordinate system image (step S16-4). Specifically, the aging management unit 215 of the control unit 21 uses the prediction model calculated in step S16-3 to create a two-dimensional mesh of the processing target two-dimensional face image. The aging management unit 215 creates an image in each polygon by affine transformation into a polygon in cylindrical coordinate system coordinates. Here, the image information in the polygon may be insufficient depending on the face orientation of the two-dimensional face image to be processed.
  • the aging management unit 215 assumes that the left and right sides of the face are symmetric, and uses the center line of the left and right sides of the face as a symmetric line, and converts the polygons with insufficient image information into the polygons on the opposite left and right sides. Complement using image information.
  • control unit 21 executes an aging process for the three-dimensional mesh (step S16-5). Specifically, the third processing unit 218 of the control unit 21 inputs the three-dimensional mesh generated in step S16-3 to the shape aging model stored in the model storage unit 25, and ages the three-dimensional mesh. To do. In this case, the third processing unit 218 performs aging only on regions other than the shape prediction mask region.
  • the white portion shown in FIG. 26A is used as the shape prediction mask region. These portions include cheeks and nose muscles, and are regions where there is little change in shape due to aging.
  • the control unit 21 executes a texture aging process (step S16-6). Details of this processing will be described later with reference to FIG.
  • control unit 21 executes an aging three-dimensional face image generation process (step S16-7). Specifically, the third processing unit 218 of the control unit 21 synthesizes the aged images generated in steps S16-5 and S16-6, and generates an image in which the shape and texture are aged.
  • FIG. 27 (a) displays 30-year-old face image data.
  • FIGS. 27B and 27C show images calculated by the control unit 21 using the face image data as input data and performing aging prediction after 10 years and 15 years later. Here, the conversion from the two-dimensional to the three-dimensional is not performed, but it can be seen that it is reasonably aged.
  • control unit 21 executes an aging two-dimensional face image generation process (step S16-8). Specifically, the aging management unit 215 of the control unit 21 rotates the generated aging three-dimensional face image so that the face direction specified in step S16-2 is oriented, and the two-dimensional face at that time Generate an image. The aging management unit 215 displays the generated two-dimensional face image on the display.
  • step S16-6 the texture aging process (step S16-6) described above will be described with reference to FIGS.
  • the second processing unit 217 of the control unit 21 executes a wavelet transform process (step S17-1). Specifically, the second processing unit 217 calculates a wavelet coefficient R1i obtained by wavelet transforming the post-age image Ii using principal component analysis.
  • the second processing unit 217 of the control unit 21 compares the absolute values of the two wavelet coefficients, and executes a magnitude relation determination process (step S17-2).
  • the absolute value of the wavelet coefficient R1i based on the principal component analysis is compared with the value obtained by multiplying the absolute value of the wavelet coefficient R2i calculated by the texture aging process using the wavelet transform by the weighting coefficient w. It is determined whether or not the absolute value of the wavelet coefficient R1i is larger than a value obtained by multiplying the absolute value of the wavelet coefficient R2i by the weighting coefficient w.
  • step S17-2 when the absolute value of the wavelet coefficient R1i is larger than the value obtained by multiplying the absolute value of the wavelet coefficient R2i by the weighting coefficient w (in the case of “YES” in step S17-2), the second processing unit 217 of the control unit 21. Executes a process of substituting the wavelet coefficient R1i into the wavelet coefficient R3i to be used (step S17-3).
  • step S17-4 when the absolute value of the wavelet coefficient R1i is equal to or smaller than the value obtained by multiplying the absolute value of the wavelet coefficient R2i by the weighting coefficient w (in the case of “NO” in step S17-2), the second processing unit 217 of the control unit 21 Then, a process of substituting the wavelet coefficient R2i into the wavelet coefficient R3i to be used is executed (step S17-4).
  • the above processing is repeated for the wavelet coefficient number i.
  • the second processing unit 217 of the control unit 21 executes wavelet inverse transformation processing of the wavelet coefficient R3 to be used (step S17-5).
  • the second processing unit 217 of the control unit 21 performs a mask area reflection process (step S17-6). Specifically, the second processing unit 217 performs aging change only on regions other than the texture prediction mask region.
  • the white portion shown in FIG. 26B is used as the texture prediction mask region.
  • These portions include the eyes, nose, mouth, and the like, and are regions where there is little change in texture due to aging.
  • the control unit 21 of the present embodiment executes an aging management unit 215 that generates a face image after aging, and a second processing unit that performs a process of predicting a change due to aging for the texture (texture) of the face image. 217, a third processing unit 218 that predicts changes due to aging of the face shape.
  • aging prediction is performed in consideration of aging of the face shape and changes due to aging of the texture of the face image, so that an aging face image can be generated more accurately.
  • the learning management unit 210 of the aging prediction system 20 of the present embodiment uses the data recorded in the secular change data storage unit 22 and the snapshot data storage unit 23 to extract facial feature points, facial features A point normalization process and a homology modeling process are executed (steps S1-2 to S1-4). This makes it possible to generate a texture aging model and a shape aging model by using a plurality of actual sample data and sharing them in terms of anatomy.
  • the control unit 21 of the present embodiment includes a first processing unit 216 that executes a process of generating three-dimensional face data from a processing target two-dimensional face image. Thereby, even if the orientation of the two-dimensional face image to be processed is not the front or the like, an aging face image can be generated with the face orientation at the specified angle.
  • the first processing unit 216 of the aging prediction system 20 executes a two-dimensional face image angle learning process using a three-dimensional mesh (homology model).
  • a three-dimensional prediction model for converting from two dimensions to three dimensions can be generated using actual sample data.
  • the second processing unit 217 of the aging prediction system 20 of the present embodiment uses a texture aging model using principal component analysis and a texture aging model using wavelet (WAVELET) transformation, Execute texture aging processing. This makes it possible to more accurately generate an aged face image by using wavelet transformation that is aging using existing stains and wrinkles.
  • WAVELET wavelet
  • the second processing unit 217 of the aging prediction system 20 of the present embodiment stores a weighting coefficient w that is a value that determines the importance of a model using principal component analysis and a model using wavelet transform. . Thereby, the weighting of the texture aging model by principal component analysis and the texture aging model by wavelet transform can be changed by changing the weighting coefficient w.
  • the control unit 21 of the above embodiment uses the aging data stored in the aging data storage unit 22 and the snapshot data stored in the snapshot data storage unit 23 to use the texture aging model and the shape aging model.
  • a model was generated.
  • a texture aging model and a shape aging model may be generated for each attribute (for example, sex or age group) of secular change data or snapshot data.
  • the control unit 21 uses the secular change data and the snapshot data having the same attribute to normalize the cylindrical coordinate system image D5, the normalized face feature point data D6, and the three-dimensional mesh data D7 ( Homology model).
  • the control unit 21 uses these, the control unit 21 generates a texture aging model and a shape aging model for each attribute, and stores them in the model storage unit 25 in association with each attribute information.
  • the control unit 21 acquires face attribute information included in this image together with the processing target two-dimensional face image data.
  • the control part 21 performs an aging prediction process using the texture aging model and the shape aging model of the attribute corresponding to the acquired attribute information. Accordingly, more accurate face image data can be generated in consideration of the influence of texture and shape according to attributes such as sex and age group.
  • the control unit 21 of the above embodiment uses the wavelet coefficient R1i obtained by wavelet transforming the post-age image Ii using the principal component analysis, and the wavelet coefficient R2i of the texture aging model based on the wavelet transform.
  • the texture aging process is not limited to the case where these two wavelet coefficients R1i and R2i are used alternatively, but the statistical values of these two wavelet coefficients R1i and R2i (for example, depending on the average value or attribute) (Combined value by ratio) may be used.
  • the control unit 21 of the above embodiment uses wavelet transform in the texture aging process.
  • the aging process is not limited to wavelet transform. It is possible to use a method of deepening (emphasizing) the stains and wrinkles on the face texture.
  • the face texture may be aged using a known sharpening filter or Fourier transform.
  • the control part 21 of the said embodiment produced
  • the analysis process used to generate the aging model is not limited to the principal component analysis process. Any analysis process can be used to identify variables that express individual differences.
  • an aging model can be generated using independent component analysis (ICA) or multidimensional scaling (MDS). You may do it.
  • ICA independent component analysis
  • MDS multidimensional scaling
  • control unit 21 converts the two-dimensional face image to the front direction using the angle prediction model in the face direction estimation process (step S16-2) of the aging prediction process.
  • the face direction estimation process (step S16-2) of the aging prediction process is not limited to the method using the angle prediction model.
  • the direction in which the face was photographed in the image may be specified using a known procrustes analysis.
  • the control unit 21 performs the first learning process for the conversion from the two-dimensional face image to the three-dimensional face data, the second learning process for the texture aging, and the third learning process for the three-dimensional shape aging.
  • a machine learning process was executed.
  • the machine learning unit 214b of the control unit 21 calculates an explanatory variable (a feature amount used at the time of prediction) using a dependent variable (a prediction target feature amount) by multiple regression analysis.
  • Machine learning processing executed by the machine learning unit 214b of the control unit 21 is not limited to learning processing using multiple regression analysis, and other analysis methods may be used. For example, a coupling learning process, a learning process based on PLS regression (Partial Last Squares Regression), a learning process based on Support Vector Regression (Support Vector Regression; SVR), or the like may be performed.
  • the machine learning unit 214b generates a single row vector (one-dimensional vector) by combining the “prediction target feature amount” and “feature amount used during prediction” of each sample data.
  • the principal component coefficient of “rotation angle ( ⁇ , ⁇ )” is used as the “prediction target feature amount” in the first learning process, and “two-dimensional feature point of all data is used as the“ feature amount used during prediction ”.
  • Principal component coefficients of “the principal component score divided by the standard deviation” are used.
  • the machine learning unit 214b generates a data matrix of combined patch vectors in which the generated one-dimensional vectors are arranged in the column direction for each sample data.
  • a principal component analysis of the data matrix is performed to generate a principal component vector matrix.
  • This principal component vector matrix is a matrix in which the row vectors are arranged in the order of principal components that change drastically between the “prediction target feature value” and the “feature value used for prediction”.
  • the machine learning unit 214b executes orthogonalization processing of the principal component vector matrix.
  • the “feature value used for prediction” matrix is orthogonalized by the Gram Schmid method.
  • the “prediction target feature amount” is converted by multiplying the orthogonalization coefficient in the “feature amount used during prediction”.
  • the machine learning unit 214b generates using the orthogonalized “features to be used for prediction” (matrix Di, j) and the “prediction target feature values” (matrix Ei, j) converted accordingly.
  • the predicted model is recorded in the model storage unit 25.
  • the control unit 21 calculates the coefficient bi representing the weight of the principal component by inner product of the input data Xj and the matrix Di, j stored in the model storage unit 25.
  • the control unit 21 reconstructs prediction data Yj, which is a prediction vector, using the coefficient bi and the matrix Ei, j. Thereby, the control part 21 can calculate the prediction data Yj based on the input data Xj.
  • This PLS regression uses the covariance w i of independent variables (features to be predicted) and explanatory variables (features used at the time of prediction), and adds multiple regression analysis to the variables in descending order of their correlation. By doing so, a regression coefficient matrix is calculated. Specifically, the following processes [1] to [4] are repeated until the intersection determination error is minimized.
  • the intersection determination error is an error between the prediction result and the prediction target when the sample data is divided into the prediction target and the input data, the prediction target is predicted using the input data.
  • machine learning unit 214b calculates an independent variable (prediction target features), the covariance matrix of explanatory variables (features to be used for prediction) (correlation matrix) W i.
  • the covariance matrix W i of the independent variable and explanatory variables is calculated by the following equation.
  • T means a transposed matrix.
  • the machine learning unit 214b projects the independent variable X i onto the space of the covariance w i and calculates the score matrix ti.
  • the machine learning unit 214b executes an independent variable update process. Specifically, similarly to the update processing of the explanatory variables, the machine learning unit 214b calculates a regression coefficient matrix that predicts the independent variable from the score matrix, deletes information used for the regression once, and the remaining independent variables Is calculated.
  • the machine learning unit 214b determines whether or not the intersection determination error is minimum. Specifically, first, the machine learning unit 214b assumes that a part of the learning data (for example, 1/4 of the entire learning data) is a prediction target, and uses the data excluding these prediction targets as input data. Then, using the explanatory variable Y i + 1 calculated in the process [4] and the independent variable X i + 1 calculated in the process [4], an error from the prediction target is calculated.
  • the control unit 21 determines that the intersection determination error is not the minimum.
  • the machine learning unit 214b uses the explanatory variable Y i + 1 as Y i and the independent variable X i + 1 as X i and repeatedly executes the processing [1] and subsequent steps.
  • the machine learning unit 214b performs the calculation calculated up to the previous process [1].
  • the covariance matrix W is generated by arranging the variances w i in the horizontal direction.
  • the control unit 21 generates a matrix C by arranging the regression coefficients c i calculated in the process [3] in the horizontal direction using the covariance matrix W, and uses the covariance matrix W in the process [4].
  • the calculated regression coefficient p i is arranged in the horizontal direction to generate a matrix P.
  • a prediction model generated using the regression coefficient matrix B is recorded in the model storage unit 25.
  • control unit 21 performs prediction using the input data Xj and the recorded regression coefficient matrix B.
  • This support vector regression analysis is a nonlinear analysis and calculates a regression curve. Specifically, in the support vector regression analysis, the regression curve is calculated so that the sample data falls within the range (tube) of the regression curve ⁇ predetermined distance w. The data that goes out of the tube is taken as penalty data ⁇ , and a curve that minimizes the following equation and a predetermined distance w are calculated.
  • the adjustment constant C is a parameter for adjusting the allowable range of outliers. If this adjustment constant C is large, the allowable range becomes small. “ ⁇ + i” is “0” if the data i is in the tube, and is a value into which the distance from the tube is substituted if it is above the tube. “ ⁇ ⁇ i” is “0” if the data i is in the tube, and if it is below the tube, the distance to the tube is substituted.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Geometry (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

L'invention concerne un système de prédiction de vieillissement, comprenant : une unité de stockage de modèle qui stocke des modèles âge-forme permettant de prédire des changements associés à l'âge de formes de visage, des modèles âge-texture permettant de prédire des changements associés à l'âge de textures de surface du visage, et des modèles de prédiction tridimensionnels permettant de prédire des données tridimensionnelles à partir d'images bidimensionnelles ; et une unité de commande qui prédit le vieillissement. L'unité de commande est configurée pour exécuter un processus de prédiction consistant à produire un modèle de visage de vieillissement, et fournir en sortie le modèle de visage de vieillissement produit à une unité de sortie.
PCT/JP2016/063227 2015-07-09 2016-04-27 Système de prédiction de vieillissement, procédé de prédiction de vieillissement, et programme de prédiction de vieillissement WO2017006615A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
KR1020177025018A KR101968437B1 (ko) 2015-07-09 2016-04-27 가령화 예측 시스템, 가령화 예측 방법 및 가령화 예측 프로그램
CN201680016809.0A CN107408290A (zh) 2015-07-09 2016-04-27 增龄化预测系统、增龄化预测方法以及增龄化预测程序

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2015137942A JP5950486B1 (ja) 2015-04-01 2015-07-09 加齢化予測システム、加齢化予測方法及び加齢化予測プログラム
JP2015-137942 2015-07-09

Publications (1)

Publication Number Publication Date
WO2017006615A1 true WO2017006615A1 (fr) 2017-01-12

Family

ID=57709415

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2016/063227 WO2017006615A1 (fr) 2015-07-09 2016-04-27 Système de prédiction de vieillissement, procédé de prédiction de vieillissement, et programme de prédiction de vieillissement

Country Status (3)

Country Link
KR (1) KR101968437B1 (fr)
CN (1) CN107408290A (fr)
WO (1) WO2017006615A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110399813A (zh) * 2019-07-10 2019-11-01 深兰科技（上海）有限公司 一种年龄识别方法、装置、电子设备及存储介质
CN112287852A (zh) * 2020-11-02 2021-01-29 腾讯科技（深圳）有限公司 人脸图像的处理方法、显示方法、装置及设备
US11354844B2 (en) 2018-10-26 2022-06-07 Soul Machines Limited Digital character blending and generation system and method

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108230431B (zh) * 2018-01-24 2022-07-12 深圳市云之梦科技有限公司 一种二维虚拟形象的人体动作动画生成方法及系统
CN109886248B (zh) * 2019-03-08 2023-06-23 南方科技大学 一种图像生成方法、装置、存储介质及电子设备
JP6751540B1 (ja) * 2019-05-31 2020-09-09 みずほ情報総研株式会社 形状予測システム、形状予測方法及び形状予測プログラム

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH04199474A (ja) * 1990-11-29 1992-07-20 Matsushita Electric Ind Co Ltd 顔画像合成装置
JP2007265396A (ja) * 2006-03-29 2007-10-11 Mitsubishi Electric Research Laboratories Inc 顔モデルを生成する方法およびシステム
JP2013089032A (ja) * 2011-10-18 2013-05-13 Kao Corp 顔印象判定チャート
JP2014137719A (ja) * 2013-01-17 2014-07-28 Ntt Communications Corp 特徴点出力装置、特徴点出力プログラム、特徴点出力方法、検索装置、検索プログラムおよび検索方法

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4341135B2 (ja) * 2000-03-10 2009-10-07 コニカミノルタホールディングス株式会社 物体認識装置
JP2001273495A (ja) * 2000-03-24 2001-10-05 Minolta Co Ltd 物体認識装置
KR20010044586A (ko) * 2001-03-09 2001-06-05 조양일 얼굴 근육 운동 예측 맵을 이용한 3차원 얼굴 애니메이션구현 시스템 및 방법
US8391639B2 (en) * 2007-07-23 2013-03-05 The Procter & Gamble Company Method and apparatus for realistic simulation of wrinkle aging and de-aging
JP4728432B2 (ja) * 2008-01-16 2011-07-20 旭化成株式会社 顔姿勢推定装置、顔姿勢推定方法、及び、顔姿勢推定プログラム
JP4518157B2 (ja) * 2008-01-31 2010-08-04 カシオ計算機株式会社 撮像装置及びそのプログラム
JP4569670B2 (ja) * 2008-06-11 2010-10-27 ソニー株式会社 画像処理装置、画像処理方法およびプログラム
JP5428886B2 (ja) * 2010-01-19 2014-02-26 ソニー株式会社 情報処理装置、情報処理方法、及びそのプログラム
JP2012048649A (ja) * 2010-08-30 2012-03-08 Citizen Holdings Co Ltd 体形変化予測装置
JP5795979B2 (ja) * 2012-03-15 2015-10-14 株式会社東芝 人物画像処理装置、及び人物画像処理方法
US20150006243A1 (en) * 2013-06-28 2015-01-01 AZAPA R&D Americas, Inc. Digital information gathering and analyzing method and apparatus
KR20140004604A (ko) * 2013-10-02 2014-01-13 인텔렉추얼디스커버리 주식회사 ３차원 얼굴 생성 장치 및 방법

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH04199474A (ja) * 1990-11-29 1992-07-20 Matsushita Electric Ind Co Ltd 顔画像合成装置
JP2007265396A (ja) * 2006-03-29 2007-10-11 Mitsubishi Electric Research Laboratories Inc 顔モデルを生成する方法およびシステム
JP2013089032A (ja) * 2011-10-18 2013-05-13 Kao Corp 顔印象判定チャート
JP2014137719A (ja) * 2013-01-17 2014-07-28 Ntt Communications Corp 特徴点出力装置、特徴点出力プログラム、特徴点出力方法、検索装置、検索プログラムおよび検索方法

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
K, SCHERBAUM ET AL.: "Prediction of Individual Non-linear Aging Trajectories of Faces", COMPUTER GRAPHICS FORUM, vol. 26, no. 3, 3 September 2007 (2007-09-03), pages 285 - 294, XP055345892 *
SHIGEO MORISHIMA: "Pattern Analysis and Synthesis of Human Face for Entertainment Applications", JOURNAL OF THE SOCIETY OF INSTRUMENT AND CONTROL ENGINEERS, vol. 53, no. 7, 10 July 2014 (2014-07-10), pages 593 - 598 *
TAKESHI NAGATA ET AL.: "Seibutsu·Iryo, Zairyo· Kikibun'ya Muke 'Kodo Gazo Shori Solution", IMAGE LAB, vol. 25, no. 5, 10 May 2014 (2014-05-10), pages 48 - 51 *
YUSUKE TAZOE ET AL.: "Aging Simulation of Personal Face Based on Conversion of 3D Geometry and Texture", IEICE TECHNICAL REPORT, vol. 109, no. 471, 8 March 2010 (2010-03-08), pages 151 - 156 *
YUSUKE TAZOE ET AL.: "Keijo Henkei to Patch Tiling ni Motozuku Texture Henkan ni yoru Nenrei Henka Kao Simulation", VISUAL COMPUTING/ GRAPHICS TO CAD GODO SYMPOSIUM 2012, 23 June 2012 (2012-06-23), pages 1 - 8 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11354844B2 (en) 2018-10-26 2022-06-07 Soul Machines Limited Digital character blending and generation system and method
CN110399813A (zh) * 2019-07-10 2019-11-01 深兰科技（上海）有限公司 一种年龄识别方法、装置、电子设备及存储介质
CN112287852A (zh) * 2020-11-02 2021-01-29 腾讯科技（深圳）有限公司 人脸图像的处理方法、显示方法、装置及设备
WO2022089166A1 (fr) * 2020-11-02 2022-05-05 腾讯科技（深圳）有限公司 Procédé et appareil de traitement d'images de visage, procédé et appareil d'affichage d'images de visage, et dispositif
CN112287852B (zh) * 2020-11-02 2023-11-21 腾讯科技（深圳）有限公司 人脸图像的处理方法、显示方法、装置及设备

Also Published As

Publication number Publication date
KR101968437B1 (ko) 2019-04-11
KR20170115591A (ko) 2017-10-17
CN107408290A (zh) 2017-11-28

Similar Documents

Publication Publication Date Title
JP5950486B1 (ja) 加齢化予測システム、加齢化予測方法及び加齢化予測プログラム
WO2017006615A1 (fr) Système de prédiction de vieillissement, procédé de prédiction de vieillissement, et programme de prédiction de vieillissement
Ichim et al. Phace: Physics-based face modeling and animation
Rohmer et al. Animation wrinkling: augmenting coarse cloth simulations with realistic-looking wrinkles
CN109002763B (zh) 基于同源连续性的模拟人脸老化的方法及装置
JP6207210B2 (ja) 情報処理装置およびその方法
KR102010161B1 (ko) 정보를 예측하기 위한 시스템, 방법 및 프로그램
KR20130003170A (ko) 표정 제어점에 기초한 강체 영역의 표현 방법 및 장치
JP2018503470A (ja) 統計的技術を用いて形成されたデジタル歯冠モデルに表面詳細を追加するシステム及び方法
CN111091624A (zh) 一种从单张图片生成高精度可驱动人脸三维模型的方法
JP4170096B2 (ja) 対象の３次元表面上にマップされた３次元メッシュモデルの適合性評価のための画像処理装置
Dupej et al. Statistical Mesh Shape Analysis with Nonlandmark Nonrigid Registration.
JP2012185545A (ja) 顔画像処理装置
Nguyen et al. Kinect-driven patient-specific head, skull, and muscle network modelling for facial palsy patients
Wang et al. Modeling of personalized anatomy using plastic strains
JP5639499B2 (ja) 顔画像処理装置
Hansen et al. Adaptive parametrization of multivariate B-splines for image registration
CN111275610A (zh) 一种人脸变老图像处理方法及系统
JP2017122993A (ja) 画像処理装置、画像処理方法及びプログラム
JP5650012B2 (ja) 顔画像処理方法、美容カウンセリング方法および顔画像処理装置
Danckaers et al. Adaptable digital human models from 3D body scans
JP5954846B2 (ja) 形状データ生成プログラム、形状データ生成方法及び形状データ生成装置
JP7100842B2 (ja) モデル解析装置、モデル解析方法、およびモデル解析プログラム
CN112329640A (zh) 基于眼部肌肉运动分析的面部神经麻痹疾病康复检测系统
JP6751540B1 (ja) 形状予測システム、形状予測方法及び形状予測プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16821086

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 20177025018

Country of ref document: KR

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16821086

Country of ref document: EP

Kind code of ref document: A1