JP2009075880A - Apparatus and program for deforming virtual face model - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a virtual face model deformed into a certain expression with efficiency and high accuracy. <P>SOLUTION: A virtual face model deformation apparatus for deforming the expression of a virtual face model to match the expression of a subject's face photographed has an input unit for inputting information on the positions of a plurality of preset feature points from the subject's face; a deformation estimating unit for estimating, using expression deformation data obtained from a plurality of previously stored face images for learning, deformation data corresponding to the information on the positions of the plurality of feature points input by the input unit; and a deforming unit for combining the deformation data estimated by the deformation estimating unit with a virtual face model selected from a set of a plurality of preset virtual face models, and deforming the expression of the virtual face model. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a virtual face model deforming apparatus and a virtual face model deforming program, and in particular, a virtual face model deforming apparatus and a virtual face model deforming for acquiring a virtual face model deformed into a predetermined expression efficiently and with high accuracy. Regarding the program.

  Conventionally, three-dimensional computer models are frequently used in industries such as computer graphics (CG) and computer animation. Many three-dimensional CG models (virtual three-dimensional models) describe the surface shape of an object (including a person) with a “mesh” (polygonal three-dimensional network). It is described by image information called “texture”.

  In addition, when a 3D model is rendered using conventional CG software, the appearance of an object viewed from an arbitrary angle under almost arbitrary lighting conditions can be obtained by pasting a “texture” on the visible portion of the “mesh”. It can be reproduced as a still image or a moving image (animation).

  However, the CG model has a problem that it is difficult to reproduce the face of a person who plays a facial expression so that it looks natural, and the movement (dynamic deformation) associated with the facial expression is particularly difficult to reproduce naturally.

  Therefore, when producing a facial expression of a CG character using a three-dimensional CG model for a still image or a moving image, the face shape mesh corresponding to the target facial expression is a linear mesh of several basic facial expressions (including no expression). An approach is used that approximates as a combination ("blend shape"). Also, when creating a video sequence that dynamically changes from an expressionless expression to a desired expression (full expression), a technique that approximates the blend shape of the same base expression for each frame is often used. Yes.

  Here, the blend shape is a technique for generating a facial expression by preparing a base facial expression and a plurality of facial expression objects different from the facial expression and mixing them with a predetermined degree to the base.

In addition, in order to improve the naturalness of facial expressions reproduced by CG software, the faces associated with facial expressions from still images or videos obtained by photographing the face of the performer (subject) who performed the facial expression with a camera or the like. A method of learning the deformation of the image, moving to the CG model, and reproducing the expression with the CG character has been studied (for example, see Non-Patent Documents 1 and 2).
Raousaiou et al., Parameterized Facial Expression Synthesis Based on MPEG-4, EURASIP Journal on Applied Signal Processing 10, 1021-1038, 2002. Zhang et al., Geometric-Driving Photorealistic Facial Expression Synthesis, IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 1, 48-60, 2006.

  By the way, as described above, in order to naturally reproduce the facial expression performed by a subject such as a performer with a three-dimensional CG model, the positions and movements of a large number of feature points on the face for each performer in advance. Must be extracted automatically. On the other hand, it is troublesome to manually specify the positions of many feature points of performers, and real-time operation becomes impossible.

  Here, in the method described in Non-Patent Document 1 described above, when a facial expression is reproduced by deforming a three-dimensional model of a face from the positions and movements of many feature points, parameters for deforming the model are set. The operator etc. adjusts manually. Further, the technique disclosed in Non-Patent Document 2 also discloses a system in which an operator or the like manually adjusts many feature points to generate a model expression. Therefore, in both cases, it takes time and effort for the operator, and the accuracy differs depending on the setting of the operator, so that a highly accurate result cannot be obtained.

  Further, as described above, when approximating dynamically changing facial expressions, a method of combining blend shapes of several fixed base expression meshes is used for each frame. As a result, the blend / shape combination coefficient varies from frame to frame, but the approximation in all frames results in a combination of the same base facial expressions, so that a highly accurate expression cannot be reproduced.

  The present invention has been made in view of the above-described problems, and provides a virtual face model deforming apparatus and a virtual face model deforming program for acquiring a virtual face model that has been transformed into a predetermined facial expression efficiently and with high accuracy. The purpose is to do.

  In order to solve the above problems, the present invention employs means for solving the problems having the following characteristics.

  According to the first aspect of the present invention, in the virtual face model deforming device that deforms the facial expression of the virtual face model in correspondence with the facial expression of the photographed subject, a plurality of feature points set in advance from the face of the subject Corresponding to the position information of the plurality of feature points input by the input unit using the input unit for inputting the position information and facial expression deformation data obtained from a plurality of learning face images accumulated in advance. A deformation estimation unit that estimates deformation data, and the deformation data estimated by the deformation estimation unit are combined with a virtual face model selected from a set of a plurality of virtual face models set in advance to deform the facial expression of the virtual face model It has a deformation part.

  According to the first aspect of the present invention, it is possible to acquire a virtual face model that has been transformed into a predetermined facial expression efficiently and with high accuracy.

  According to a second aspect of the present invention, the deformation estimation unit obtains deformation data that approximates the position information using a deformation database having base mesh deformation data corresponding to a plurality of facial expressions of a plurality of preset models. It is characterized by estimating.

  According to the second aspect of the present invention, deformation data can be estimated on the basis of the vertex of the mesh, and the expression of the virtual face model corresponding to the expression of the subject is deformed with high accuracy using the deformation data. be able to.

  According to a third aspect of the present invention, the deformation estimation unit calculates a difference between a position vector of the model without expression from a position vector of a feature point corresponding to the facial expression of the subject input by the input unit. The modification vector of the input feature point obtained in this way is projected onto a partial space in a certain image frame to obtain a projection coefficient, and the modification data is generated by the projection coefficient.

  According to the third aspect of the invention, by generating the deformation data using the projection coefficient, only the component belonging to the partial space remains, and the component orthogonal to the partial space can be removed. Therefore, deformation data necessary for the deformation of the virtual face model can be acquired efficiently and with high accuracy.

  The invention described in claim 4 is characterized in that the deformation unit deforms the expression of the virtual face model by combining the deformation data and the mesh data of the virtual face model so as to approximate the position information of the feature point. And

  According to the fourth aspect of the present invention, the virtual face model can be transformed into a more natural expression efficiently and with high accuracy.

  The invention described in claim 5 is characterized in that the deforming unit adjusts the number of times of use of time-series data of dynamic deformation accompanying the facial expression of the plurality of preset virtual face models.

  According to the invention described in claim 5, it is possible to adjust in detail according to the performance of the apparatus, the degree of quality of the output result, the desired processing speed, and the like. Thereby, highly accurate deformation data can be acquired.

  According to a sixth aspect of the present invention, in the virtual face model deformation program for deforming the expression of the virtual face model in correspondence with the expression of the face of the photographed subject, a plurality of preset presets from the face of the subject to the computer Position information of a plurality of feature points input by the input process using input processing for inputting the position information of the feature points and deformation data of facial expressions obtained from a plurality of learning face images accumulated in advance The facial expression of the virtual face model by combining the deformation estimation process for estimating the deformation data corresponding to, and the deformation data estimated by the deformation estimation process in combination with a virtual face model selected from a set of a plurality of preset virtual face models And a deformation process for deforming.

  According to the sixth aspect of the present invention, it is possible to acquire a virtual face model that has been transformed into a predetermined facial expression efficiently and with high accuracy. Further, by installing the execution program in the computer, a virtual face model that is easily transformed into a predetermined expression can be acquired.

  According to the present invention, it is possible to acquire a virtual face model transformed into a predetermined facial expression efficiently and with high accuracy.

<Outline of the present invention>
The present invention, for example, dynamically acquires position information of a small number of feature points in the face of an actual person (subject) such as a performer of a program by a face feature point tracking device, and based on the position information, As a blend shape of multiple learning 3D face models (basic facial expressions), it approximates the deformation accompanying the facial expression of the performer, deforms the facial expression of another virtual 3D model, and creates a still image or animation (animation) And so on.

  Here, in order to determine the value of the blend / shape combination coefficient, the error (approximation error) between the feature point extracted from the face image of the performer or the like and the mesh vertex of the 3D model at approximately the same position as each of them. Minimize. Furthermore, the present invention further reduces the approximation error by dynamically deforming not only the blend shape coefficient but also the base expression.

<Embodiment>
Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

<Virtual face model transformation system>
FIG. 1 is a diagram illustrating a configuration example of a virtual face model deformation system in the present embodiment. A virtual face model deformation system 10 shown in FIG. 1 is configured to include a camera 11 as an image capturing unit that captures the face of an actual person (subject), a face feature point tracking device 12, and a virtual face model deformation device 13. Has been. The virtual face model deformation device 13 according to the present invention includes an input unit 21, a deformation estimation unit 22, a deformation database 23, a deformation unit 24, a display virtual three-dimensional face model aggregation unit 25, and a display unit 26. And an output unit 27.

  The camera 11 shown in FIG. 1 acquires an image including the face of the subject 31 such as a program performer and acquires a moving image of the face. The camera 11 also outputs the captured moving image of the subject 31 to the face feature point tracking device 12.

  The face feature point tracking device 12 performs face detection, tracking, and recognition using the temporal continuity of moving images taken by the camera 11, and acquires face feature points. Specifically, template matching is performed on a captured moving image with a preset variable template using features such as Gabor wavelet coefficients. This matching process is performed on all moving images with the same data representation.

  Further, the face feature point tracking device 12 extracts a face portion from the moving image, and extracts a feature point for the extracted face. Here, FIG. 2 is a diagram illustrating an example of extracted facial feature points. In the example of the feature points illustrated in FIG. 2, nine points including the eyebrows, the eyes, the corners of the eyes, the nose, the mouth, and the mouth are extracted as feature points 32-1 to 32-9 from the face area of the photographed subject 31. The location and number of feature points are not particularly limited to this.

  Further, the face feature point tracking device 12 may have a multi-resolution matching function. In this case, the face feature point tracking device 12 performs matching from a low resolution to a high resolution and performs weighted estimation, thereby improving reliability. The impact of low data can be reduced. Further, the face feature point tracking device 12 performs robust estimation, integrates matching information for the number of frames by temporal / spatial filtering and Bayes estimation, and outputs the result of the feature point coordinates as position information.

  As for the acquisition of facial feature points in the facial feature point tracking device 12, for example, an existing facial feature point tracking method (for example, Simon Krippingdel et al., “A unified approach to face detection / tracking / recognition of moving images”). The Institute of Electronics, Information and Communication Engineers, IEICE Technical Report PRMU 98-200 (January 1999)) and the like can be used.

  The face feature point tracking device 12 described above outputs position information of several feature points in the face image of the subject being photographed by the camera 11 to the virtual face model deforming device 13.

  Note that the position information of the feature points input to the virtual face model deforming device 13 may be input in real time, for example, during shooting of the subject 31, or may be temporarily stored as a moving image in an external storage device or the like. You may input into the virtual face model deformation | transformation apparatus 13 at the predetermined timing which wants to perform the process in the virtual face model deformation | transformation apparatus 13. FIG. Further, the position information of the feature points obtained by a method other than the above using another device or the like may be input.

  In the virtual face model deformation device 13, when the input unit 21 inputs position information of each feature point of the face image, the input unit 21 outputs the position information (coordinate information) to the deformation estimation unit 22.

  The deformation estimation unit 22 uses the coordinates of each feature point of the face image obtained from the input unit 21 and deformation data (basic mesh deformation data, subsamples) obtained from a plurality of learning face images stored in advance in the deformation database 23. Deformation data) is used to estimate appropriate deformation data for approximating (including matching) the expression of a virtual three-dimensional face model, which will be described later, with respect to the expression of the captured face image. Details of the deformation database 23 will be described later. Further, the deformation estimation unit 22 outputs an estimation parameter (deformation data) for the deformation to the deformation unit 24.

  The deforming unit 24 is selected in advance by a user or the like from the display virtual three-dimensional face model set unit 25 having shape mesh information and texture information for each of a plurality of virtual face models (CG models) stored in advance. The expression data of the virtual three-dimensional face model is deformed by combining the shape data of the arbitrary virtual three-dimensional face model with the deformation data estimated by the deformation estimation unit 22 described above.

  Specifically, the deformation unit 24 deforms the expression of the virtual face model by combining the deformation data and the virtual face model so as to approximate the coordinates of each feature point. Thereby, the virtual face model can be transformed into a more natural expression efficiently and with high accuracy. In addition, the deformation | transformation part 24 may adjust the use number of the time series data of the dynamic deformation | transformation accompanying a facial expression with the several virtual face model set beforehand. Thereby, it is possible to adjust in detail according to the performance of the apparatus, the degree of quality of the output result, the desired processing speed, etc., and highly accurate deformation data can be acquired.

  The deforming unit 24 outputs the deformed virtual three-dimensional face model face image data together with the mesh and texture information to the display unit 26 and causes the display unit 28 to display the face image data. Can be stored in the storage unit 27, or face image data can be output to the output unit 28 and output from the output unit 28 as an image file.

  Details of specific processes in the deformation estimation unit 22 and the deformation unit 24 described above will be described later.

  The display unit 26 directly displays the face image of the deformed virtual three-dimensional face model obtained from the deforming unit 24 or renders it on a display or the like using mesh and texture information (an object or figure given as numerical data) The face image of the virtual three-dimensional face model obtained by imaging the information regarding the image is displayed. The storage unit 27 stores image data obtained from the deformation unit 24 and the like. The output unit 28 outputs the image data obtained from the deforming unit 24 as a file in a predetermined format.

  Thereby, the expression of the virtual three-dimensional face model for display set in advance corresponding to the expression of the face image of the subject 31 can be deformed and displayed. Therefore, it is possible to acquire a virtual face model that has been transformed into a predetermined facial expression efficiently and with high accuracy.

  The input unit 21 and the output unit 28 described above can perform data transmission / reception with an external device or the like through a communication interface such as socket communication, and the display unit 26 is implemented with a display connected to a computer or the like. can do.

<About the deformation database 23>
Next, the deformation database 23 described above will be specifically described. FIG. 3 is an example for explaining the flow of processing up to the construction of the modified database. In FIG. 3, learning mesh deformation data 41 is used as learning data. This is because, for example, a certain number (M) of performers (actual persons) have E (type) facial expressions (for example, E = 3 when “smile”, “sadness”, “surprise”). Is composed of a sequence of the same number of frames from the expressionless to the full expression (predetermined expression) of the three-dimensional face model (face shape mesh).

  In the example of FIG. 3, for example, the length of the sequence (number of frames) is F + 1 frame (0 frame to F frame), and the face appears in the first frame (0 frame) shown in FIG. It is assumed that the expression smoothly changes to full (full expression) until the last frame (F frame).

  In order to construct the deformation database 23, first, average information 42 of mesh vertex coordinates of all persons (1 to M) is acquired for each frame of each facial expression. Thus, E sequences p (1),..., P (E) are obtained. Then, by extracting (sampling) the coordinates of some vertices as subsamples 43 from the mesh information of the sequence p (1),..., P (E), the sequence q (1),. Ask. As the mesh vertex to be extracted, a vertex closest to the feature point (input feature point) input from the face feature point tracking device 12 is selected (ideally, the mesh vertex and the position of the feature point coincide with each other). Preferably).

  Accordingly, the final deformation database 23 includes mesh sequences (basic mesh deformation data) p (1),... P (E) and subsampled sequences (subsample deformation data) q ( 1),... Q (E).

<About the deformation estimation unit 22 and the deformation unit 24>
Next, the deformation estimation unit 22 and the deformation unit 24 described above will be specifically described. FIG. 4 is a diagram illustrating an example for explaining the flow of processing of the deformation estimation unit and the deformation unit in the present embodiment. The deformation estimation unit 22 uses the coordinates of each feature point from the input unit 21 and the deformation data group in the deformation database 23 to subtract a vector r [0] when there is no expression from the input feature point vector r to obtain a certain frame. Projecting to the subspace spanned by the subsample deformation data q (1) [j] -q [0],..., q (E) [j] -q [0] in j. Thereby, the projection coefficients α (1),..., Α (E) are obtained. Note that q [0] = q (1) [0] =... = Q (E) [0] indicates subsample deformation data when there is no expression.

  Next, using the projection coefficients α (1),..., Α (E) obtained by the above processing, the base mesh deformation data p (1) [j] −p [0],. Calculate the weighted linear combination deformation of p (E) [j] −p [0] ((α (1) × (p (1) [j] −p [0]) ×... × α (E) ( p (E) [j] −p [0]))), and outputs the calculation result to the deformation unit 24 as deformation data. Note that p [0] = p (1) [0] =... = P (E) [0] represents the base mesh deformation data when there is no expression.

  Next, the deformation unit 24 adds (combines) the deformation data obtained from the deformation estimation unit 22 to the mesh s [0] of the display virtual three-dimensional face model to deform, and as a result, the deformed display is displayed. Output mesh s. Note that the mesh s [0] indicates that there is no expression.

  Here, FIG. 5 is a diagram illustrating an example of setting the frame position of the deformation data used as the base expression. As shown in FIG. 5, the frame number j using the deformation data of the base mesh is set according to the magnitude of the deformation vector | r−r [0] | of the input feature points.

  When the deformation vector | r−r [0] | of the input feature point is small, the approximation of the output facial expression is a blend shape combination of the frame J of the base mesh deformation data, and | r−r [0] | If it is larger, it becomes a blend shape combination of frames J <= j <= F of the base mesh deformation data corresponding to the size.

  Here, the processing procedure of the above-described embodiment will be described more specifically. For example, in the process of fixing the base facial expression, first, the deformation data of the face shape (CG model mesh) that played E (type) basic facial expressions in advance is collected offline, and the same length (number of frames) is collected. ) And stored in the deformation database 23 as base mesh deformation data p (1),..., P (E). In each sequence, facial expressions appear larger as time passes with the face initially having no expression. At this time, only mesh vertices closest to the position of each feature point of the face image input from the face feature point tracking device 12 described above as a subsample are extracted from each frame of p (1),..., P (E). , Subsample deformation data q (1),..., Q (E) are stored in the deformation database 23.

The deformation estimation unit 22 obtains a deformation vector rr [0] of the input feature point obtained by subtracting a value r [0] when the performer's face is expressionless from the coordinate vector r of the input feature point. , Q (E) [j] -q [0] is projected onto a subspace spanned by subsample modified data q (1) [j] -q [0],. As a result, only the component belonging to this partial space remains with respect to the deformation vector rr [0] of the input feature point (the component orthogonal to the partial space is removed). As a result of projection, coefficients α (1),..., Α (E) are obtained. here,
Q [j] = [q (1) [j] −q [0]... Q (E) [j] −q [0]]
Then,
α = (Q [j] ^T Q [j]) ⁻¹ Q [j] ^T (r−r [0])
Can be obtained as Note that q (0) = q (1) [0] =... = Q (E) [0] indicates subsample deformation data when there is no expression.

  Further, using the projection coefficients α (1),..., Α (E) obtained by the above processing, the base mesh deformation data p (1) [j] −p [0],. E) A weighted linear combination of [j] -p [0] is calculated, and the calculation result is output to the deformation unit 24 as deformation data. Note that p [0] = p (1) [0] =... = P (E) [0] indicates the base mesh deformation data when there is no expression.

The deformation unit 24 adds (combines) the deformation data obtained from the deformation estimation unit 22 to the mesh s [0] of the display virtual three-dimensional face model, and outputs the resulting mesh s. Note that s [0] indicates that there is no expression. here,
P [j] = [p (1) [j] −p [0]... P (E) [j] −p [0]]
Then, the deformed mesh to be output is
s = s [0] + α (1) (p (1) [j] −p [0]) +... + α (E) (p (E) [j] −p [0]) = s [0] + P [J] α.
Can be obtained as

  Here, there are a plurality of possibilities for selecting the frame j to be used as the base expression in the learning data. For example, when j = F (the last frame indicating “full expression” of the learning data) is fixed, the output mesh deformation s−s [0] in each frame is p (1) [F] −p [0. ],..., P (E) [F] -p [0], which corresponds to a normal blend shape method. Note that this is the best choice when the amount of calculation is important.

  On the other hand, if the frame j is likely to change as the input feature point is transformed from an expressionless expression to a full expression, the resulting approximation error can be reduced. However, if the frame j is set small, the deformation vector p () [j] -p [0] or q () [j] -q [0] also becomes small, and the noise becomes relatively large. Therefore, in this embodiment, as shown in FIG. 5, the deformation data after the frame (frame J) in which the deformation vector becomes sufficiently large is used.

  As one method for selecting frame j> = J, for example, a frame j corresponding to the magnitude of the deformation vector rr [0] of the input feature point is set by the following algorithm.

j = (int) (k * | r−r [0] |;
if (j <J) j: = J;
Here, the constant k (gradient) depends on system parameters and the like, and is therefore set by experimentation or the like.

  Therefore, as described above, when the deformation vector | r−r [0] | of the input feature point is small, the approximation of the output facial expression becomes a blend shape combination of the frame J of the base mesh deformation data, and | r− When r [0] | is sufficiently large, a blend shape combination of frames J <= j <= F of the base mesh deformation data corresponding to the size is obtained.

  Depending on the capacity of the memory or the like, the amount of processing calculation, etc., other settings other than these two examples (an example using the frame J <= j <= F of the modified data and an example using only the frame J = F) There is also a method. For example, using only some of the basis vectors shown in FIG. 5 reduces the amount of calculation by using all of them.

  Therefore, the deformation unit 24 adjusts the number of times of use of time-series data of dynamic deformation associated with facial expressions for a plurality of preset virtual face models. Thereby, it is possible to adjust in detail according to the performance of the apparatus, the degree of quality of the output result, the desired processing speed, and the like. Thereby, highly accurate deformation data can be acquired.

  On the other hand, if the combination of the deformation data in all the frames of J <= j <= F is taken, the approximation error is minimized, but the calculation amount is heavy. Therefore, it can be arbitrarily set according to desired conditions.

  As described above, according to this embodiment, it is possible to acquire a virtual face model that has been transformed into a predetermined facial expression efficiently and with high accuracy.

<Virtual face model transformation program>
Here, the virtual face model deforming device 13 includes a volatile storage medium such as a CPU and a RAM, a non-volatile storage medium such as a ROM, an input device such as a mouse, a keyboard, and a pointing device, and a display unit that displays images and data. And a computer having an interface for communicating with the outside.

  Therefore, each function in the input unit 21, the deformation estimation unit 22, the deformation unit 24, the display unit 26, and the output unit 27 included in the virtual face model deformation device 13 causes the CPU to execute a program describing these functions. Can be realized respectively. These programs can also be stored and distributed in a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

  That is, an execution program (virtual face model deformation program) for causing a computer to execute the processing in each configuration described above is generated, and the program is installed in, for example, a general-purpose personal computer or a server, so that the virtual face model deformation is performed. Processing can be realized.

<Hardware configuration>
Here, an example of a hardware configuration of an executable computer in the present invention will be described with reference to the drawings. FIG. 6 is a diagram illustrating an example of a hardware configuration capable of realizing the virtual face model deformation process according to the present invention.

  6 includes an input device 51, an output device 52, a drive device 53, an auxiliary storage device 54, a memory device 55, a CPU (Central Processing Unit) 56 for performing various controls, and a network connection device. 57 are connected to each other by a system bus B.

  The input device 51 includes a keyboard and a pointing device such as a mouse operated by a user, a voice input device such as a microphone, and the like, and inputs various operation signals such as execution of a program from the user. The output device 52 includes a display for displaying various windows and data necessary for operating the computer main body for performing processing in the present invention, a speaker for outputting sound, and the like, and the program of the program is controlled by the control program of the CPU 56. Execution progress, results, etc. can be displayed or voice output.

  Here, in the present invention, the execution program installed in the computer main body is provided by the recording medium 58 such as an optical disk, for example. The recording medium 58 on which the program is recorded can be set in the drive device 53, and the execution program included in the recording medium 58 is installed in the auxiliary storage device 54 from the recording medium 58 via the drive device 53.

  The auxiliary storage device 54 is a storage means such as a hard disk, and can store an execution program according to the present invention, a control program provided in a computer, etc., and perform input / output as necessary.

  The memory device 55 stores an execution program or the like read from the auxiliary storage device 54 by the CPU 56. The memory device 55 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU 56 controls processing of the entire computer, such as various operations and data input / output with each hardware component, based on a control program such as an OS (Operating System) and an execution program stored in the memory device 55. Each processing can be realized. Further, the CPU 56 can acquire various types of information necessary during execution of the program from the auxiliary storage device 54, and the CPU 56 can also store processing results and the like.

  The network connection device 57 obtains an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or an execution result obtained by executing the program or an execution in the present invention The program itself can be provided to other terminals.

  With the hardware configuration as described above, it is possible to efficiently realize the virtual face model deformation process at a low cost without requiring a special device configuration. In addition, by installing the program, a virtual face model that is easily transformed into a predetermined facial expression can be acquired.

<Virtual face model deformation processing procedure>
Next, the virtual face model deformation processing procedure by the execution program (virtual face model deformation program) in the present invention will be described using a flowchart. FIG. 7 is a flowchart illustrating an example of a virtual face model deformation process procedure according to the present embodiment.

  In FIG. 7, as described above, coordinates as position information of each feature point of a face obtained by processing using a face feature point tracking device or the like from a face image of a subject such as a performer photographed by a camera or the like. A moving image including information is input (S01). Next, deformation data corresponding to the coordinates of each feature point of the face input using the above-described deformation database 23 is estimated from the coordinate information of each input feature point (S02).

  Next, a virtual face model for display is selected from a set of a plurality of virtual face models set in advance (S03), and how much time series data of dynamic deformation is used for the moving image as necessary. Or the number of uses is adjusted (S04). Further, the facial expression of the virtual face model is transformed by combining the virtual face model with the transformation data so as to approximate (including matching) the coordinates of the facial feature point input in the process of S01 (S05). The model is displayed on a display unit such as a display (S06).

  Instead of the above-described processing of S06, a file having a predetermined format may be generated from the deformed virtual face model and stored in the storage unit or output to an external device.

  As described above, according to the present invention, it is possible to acquire a virtual face model that has been transformed into a predetermined facial expression efficiently and with high accuracy. More specifically, in comparison with the conventional technique in which a facial CG model plays a facial expression, the present invention is based on the position of a feature point tracked in the face of a performer (subject) and a CG model (virtual face model). ), The appearance of the performer is approximated, so that it looks more natural and takes less time than directly adjusting the CG model in detail.

  In addition, it is possible to estimate deformation data based on mesh vertices by estimating deformation data using a deformation database having base mesh deformation data corresponding to a plurality of facial expressions of a plurality of preset models. It is possible to deform the expression of the virtual face model corresponding to the expression of the subject with high accuracy using the deformation data. Further, by generating deformation data using a projection coefficient or the like, only the components belonging to the partial space remain, and the components orthogonal to the partial space can be removed. Therefore, deformation data necessary for the deformation of the virtual face model can be acquired efficiently and with high accuracy.

  In addition, using the function of combining several fixed CG model shapes called “Blend Shape” that CG software often provides, the approximation error of facial expressions is reduced by combining time-series data of dynamic deformation accompanying facial expressions. be able to. Furthermore, the scale and calculation amount can be adjusted by adjusting how much the time series data of dynamic deformation is used.

  The embodiment described above can be applied to a three-dimensional model, but is not limited thereto, and can be applied to, for example, a two-dimensional model. The present invention can be widely applied to technical fields such as computer graphics (CG) and computer animation.

  Although the preferred embodiment of the present invention has been described in detail above, the present invention is not limited to the specific embodiment, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

It is a figure which shows the example of 1 structure of the virtual face model deformation | transformation system in this embodiment. It is a figure which shows an example of the feature point of the extracted face. It is a figure of an example for demonstrating the flow of a process until construction of a deformation | transformation database. It is a figure of an example for demonstrating the flow of a process of the deformation | transformation estimation part in this embodiment, and a deformation | transformation part. It is a figure of an example for demonstrating the setting of the frame position of the deformation data used as a base expression. It is a figure which shows an example of the hardware constitutions which can implement | achieve the virtual face model deformation | transformation process in this invention. It is a flowchart which shows an example of the virtual face model deformation | transformation process procedure in this embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Virtual face model deformation | transformation system 11 Camera 12 Face feature point tracking apparatus 13 Virtual face model deformation | transformation apparatus 21 Input part 22 Deformation estimation part 23 Deformation database 24 Deformation part 25 Display virtual three-dimensional face model collection part 26 Display part 27 Output part 31 Subject 41 Learning mesh deformation data 42 Average information of mesh vertex coordinates of all persons 43 Subsample 51 Input device 52 Output device 53 Drive device 54 Auxiliary storage device 55 Memory device 56 CPU
57 Network connection device 58 Recording medium

Claims (6)

In a virtual face model transformation device that transforms the facial expression of a virtual face model in accordance with the facial expression of a photographed subject,
An input unit for inputting position information of a plurality of feature points set in advance from the face of the subject;
A deformation estimation unit that estimates deformation data corresponding to position information of a plurality of feature points input by the input unit using deformation data of facial expressions obtained from a plurality of facial images for learning accumulated in advance;
A deformation unit that deforms the expression of the virtual face model by combining the deformation data estimated by the deformation estimation unit with a virtual face model selected from a set of a plurality of virtual face models set in advance. Virtual face model deformation device. The deformation estimation unit includes:
The virtual face model according to claim 1, wherein deformation data that approximates the position information is estimated using a deformation database having base mesh deformation data corresponding to a plurality of facial expressions of a plurality of models set in advance. Deformation device. The deformation estimation unit includes:
A deformation vector of an input feature point obtained by subtracting a position vector at the time of expressionlessness of the model from a position vector of a feature point corresponding to a facial expression of the subject input by the input unit in a certain image frame 3. The virtual face model deformation apparatus according to claim 1, wherein a projection coefficient is obtained by projecting onto a partial space, and deformation data is generated based on the projection coefficient. The deformation part is
The expression of the virtual face model is deformed by combining the deformation data and the mesh data of the virtual face model so as to approximate the position information of the feature point. Virtual face model deformation device. The deformation part is
5. The virtual face model deformation apparatus according to claim 1, wherein the number of uses of time-series data of dynamic deformation associated with a facial expression of the plurality of preset virtual face models is adjusted. . In the virtual face model transformation program that transforms the facial expression of the virtual face model according to the facial expression of the photographed subject,
On the computer,
An input process for inputting positional information of a plurality of feature points set in advance from the face of the subject;
Deformation estimation processing for estimating deformation data corresponding to position information of a plurality of feature points input by the input processing using deformation data of facial expressions obtained from a plurality of facial images for learning accumulated in advance;
Virtual face model deformation for executing deformation processing for deforming a facial expression of a virtual face model by combining deformation data estimated by the deformation estimation processing with a virtual face model selected from a set of a plurality of virtual face models set in advance program.