JP2003022451A - Method and program for image formation, computer- readable recording medium having program for image formation recorded thereon, and device for image formation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To restore an image to a shape having nearly the same precision as the original image. SOLUTION: An image composing method includes a step (S01) for receiving the original image, a step (S03) for generating the parameters of a reference model by extracting shape data showing the shape of an object and texture data representing a texture corresponding to the shape data from the inputted original image and projecting them on the reference model, a step (S04) for changing at least one element of the generated parameters, a step (S05) for reconstituting the shape data and texture data of the object corresponding to the changed parameters, a step (S06) for correcting the reconstituted texture data by using the texture data extracted from the original image, and a step (S08) for generating an image by using the reconstituted shape data and reconstituted texture data.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image generation method, an image generation program, a computer-readable recording medium recording the image generation program, and an image generation apparatus, and more particularly, it projects an input image onto an object model space. The present invention relates to an editable image generating method, an image generating program, a computer-readable recording medium recording the image generating program, and an image generating apparatus.

[0002]

2. Description of the Related Art Conventionally, a method is known in which a facial image including a human face is mapped on an eigenspace model to mainly describe features such as facial expressions in a lower dimensional expression. Japanese Unexamined Patent Application Publication No. 10-228544 discloses a technique of encoding an input image using an eigenface database. For encoding, the KL (Karhuuen-Loeve) extension known as principal component analysis is used. KL
By encoding the input image with extensions, a lower dimensional representation can be found to explain the change in features. In Japanese Patent Laid-Open No. 10-228544, the coding coefficient (eigenvalue) of the eigenface, which represents the input face image by points in a lower dimensional space using the eigenface database, is transmitted to the receiving side. Using a provided eigenface database, a face image is reconstructed from the received eigenface coding coefficients.

[0003]

However, in Japanese Patent Laid-Open No. 10-228544, the coding coefficient obtained by coding using the eigenface database is directly decoded using the eigenface database. It restores the image of the face. Therefore, the image of the face used for encoding on the transmitting side and the image of the face restored on the receiving side have the same facial expression. Therefore, the facial expression cannot be changed.

Further, when the eigenface image is decoded using the eigenface database, the coding coefficient is represented by a point in a space lower in dimension than the image, and therefore the eigenface database is used for decoding. As for the image, the original image is not restored as it is, and there are problems that the shape is different and the resolution is lowered.

The present invention has been made to solve the above-mentioned problems, and one of the objects of the present invention is to solve the problem even if the accuracy of the shape in the object model space is worse than the original image. An object of the present invention is to provide an image generation method, an image generation program, a computer-readable recording medium recording the image generation program, and an image generation apparatus capable of restoring the image with a shape having substantially the same accuracy as the image.

Another object of the present invention is to be able to restore an image with a texture having substantially the same accuracy as the original image even if the accuracy of the texture in the object model space is worse than that of the original image. An object is to provide an image generation method, an image generation program, a computer-readable recording medium recording the image generation program, and an image generation device.

[0007]

According to one aspect of the present invention for achieving the above object, an image generation method includes a step of receiving an input of an original image, and a method of determining an object shape from the input original image. Extracting the shape data representing the shape data and the texture data representing the texture corresponding to the shape data, and extracting the shape data extracted into the object model space obtained by statistically analyzing a plurality of object images having the same attribute. Projecting the texture data to generate parameters in the object model space, changing at least one element of the generated parameters, and reconstructing shape data and texture data of the object corresponding to the changed parameters. And the texture extracted from the original image. Data using, comprises the steps of correcting the texture data reconstructed, and generating an image using the texture data reconstructed and the reconstructed shape data.

According to the present invention, the reconstructed texture data is corrected using the texture data extracted from the original image before the reconstructed shape data and the reconstructed texture data are combined. It For this reason,
Even when the accuracy of the texture in the object model space is poorer than that of the original image, it is possible to provide an image generation method capable of restoring the image with the texture of about the same accuracy as the original image.

Preferably, the method further comprises the step of correcting the reconstructed shape data using the shape data extracted from the original image.

According to the present invention, the reconstructed shape data is corrected using the shape data extracted from the original image before the reconstructed shape data and the reconstructed texture data are combined. It Therefore, even if the accuracy of the shape in the object model space is poorer than that of the original image, it is possible to provide an image generation method capable of restoring an image with a shape having almost the same accuracy as the original image. it can.

Preferably, the step of correcting the reconstructed texture data includes the step of correcting the reconstructed texture data using the texture data extracted from the original image and the shape data extracted from the original image. Including further.

According to the present invention, the reconstructed texture data is corrected using the texture data extracted from the original image and the shape data extracted from the original image. Can be corrected dynamically.

According to another aspect of the present invention, an image generation method includes a step of receiving an input of an original image, shape data representing a shape of an object from the input original image, and a texture representing a texture corresponding to the shape data. Generate the parameters in the object model space by projecting the extracted shape data and texture data onto the object model space obtained by statistically analyzing the step of extracting data and multiple object images having the same attributes. , Changing at least one element of the generated parameter, reconstructing the shape data and texture data of the object corresponding to the changed parameter, and the shape data extracted from the original image. To correct the reconstructed shape data using Comprising a step, and generating an image using the shape data reconstructed reconstructed texture data.

According to the present invention, the reconstructed shape data is corrected by using the shape data extracted from the original image before the reconstructed shape data and the reconstructed texture data are combined. Therefore, even if the accuracy of the shape in the object model space is lower than that of the original image, it is possible to provide an image generation method capable of restoring an image with a shape having substantially the same accuracy as the original image. it can.

Preferably, the step of correcting the reconstructed texture data includes the step of correcting the reconstructed texture data using the texture data extracted from the original image and the shape data extracted from the original image. Including further.

According to the present invention, the reconstructed texture data is corrected using the texture data extracted from the original image and the shape data extracted from the original image. Can be corrected dynamically.

Preferably, the object model space obtained by statistically analyzing a plurality of object images having the same attribute is an eigenspace obtained by principal component analysis.

According to the present invention, since the object model space is an eigenspace obtained by principal component analysis, the object model space can be easily obtained.

Preferably, the object is a human face. According to the present invention, it is possible to synthesize an image in which the shape and texture of the human face are changed.

Preferably, the step of changing the element of the parameter includes the step of changing the parameter belonging to the facial expression category.

According to the present invention, since the parameters belonging to the facial expression category are changed, it is possible to synthesize an image of a face with a changed expression.

Preferably, the step of changing the element of the parameter includes the step of changing the parameter belonging to the age category.

According to the present invention, since the parameters belonging to the age category are changed, it is possible to synthesize face images of different ages.

According to still another aspect of the present invention, the image generation program represents a step of receiving an input of an original image, shape data representing the shape of an object from the input original image, and a texture corresponding to the shape data. The step of extracting the texture data and projecting the extracted shape data and texture data onto the object model space obtained by statistically analyzing a plurality of object images having the same attributes to set the parameters in the object model space. A step of generating, a step of changing at least one element of the generated parameter, a step of reconstructing shape data and texture data of an object corresponding to the changed parameter, and a texture data extracted from the original image. Was reconstructed using And correcting the box Cha data, and a step of generating an image using the texture data reconstructed and the reconstructed shape data to the computer.

According to the present invention, the reconstructed texture data is corrected using the texture data extracted from the original image before the reconstructed shape data and the reconstructed texture data are combined. It For this reason,
Even if the accuracy of the texture in the object model space is poorer than that of the original image, an image generation program capable of restoring the image with a texture having substantially the same accuracy as the original image and a computer recording the image generation program A readable recording medium can be provided.

According to still another aspect of the present invention, the image generation program represents a step of receiving an input of an original image, shape data representing the shape of an object from the input original image, and a texture corresponding to the shape data. The step of extracting the texture data and projecting the extracted shape data and texture data onto the object model space obtained by statistically analyzing a plurality of object images having the same attributes to set the parameters in the object model space. A step of generating, a step of changing at least one element of the generated parameter, a step of reconstructing shape data and texture data of the object corresponding to the changed parameter, and a shape data extracted from the original image Using,
A computer is made to perform the step of correcting the reconstructed shape data and the step of generating an image using the reconstructed shape data and the reconstructed texture data.

According to the present invention, the reconstructed shape data is corrected using the shape data extracted from the original image before the reconstructed shape data and the reconstructed texture data are combined. It Therefore, even if the accuracy of the shape in the object model space is poorer than that of the original image, an image generation program and an image generation program capable of restoring the image with the shape of almost the same accuracy as the original image are provided. It is possible to provide a computer-readable recording medium in which the data is recorded.

According to still another aspect of the present invention, the image generating apparatus receives the input of the original image, the receiving means, the shape data representing the shape of the object from the input original image, and the texture corresponding to the shape data. In the object model space, the extraction means for extracting the represented texture data and the extracted shape data and texture data are projected onto the object model space obtained by statistically analyzing a plurality of object images having the same attribute. Generating means for generating a parameter, changing means for changing at least one element of the generated parameter, reconstructing means for reconstructing shape data and texture data of an object corresponding to the changed parameter, and an original image Using the texture data extracted from
Correction means for correcting the reconstructed texture data,
And a synthesizing unit for generating an image using the reconstructed shape data and the reconstructed texture data.

According to the present invention, the reconstructed texture data is corrected using the texture data extracted from the original image before the reconstructed shape data and the reconstructed texture data are combined. Therefore, even if the accuracy of the texture in the object model space is lower than that of the original image, it is possible to provide an image generation device capable of restoring the image with the texture of almost the same accuracy as the original image. it can.

According to still another aspect of the present invention, the image generating apparatus receives the input of the original image, the reception means, the shape data representing the shape of the object from the input original image, and the texture corresponding to the shape data. In the object model space, the extraction means for extracting the represented texture data and the extracted shape data and texture data are projected onto the object model space obtained by statistically analyzing a plurality of object images having the same attribute. Generating means for generating a parameter, changing means for changing at least one element of the generated parameter, reconstructing means for reconstructing shape data and texture data of an object corresponding to the changed parameter, and an original image Shape data reconstructed using the shape data extracted from Comprising a correction means for correcting, and combining means for generating an image by using the reconstructed shape data and the reconstructed texture data.

According to the present invention, the reconstructed shape data is corrected using the shape data extracted from the original image before the reconstructed shape data and the reconstructed texture data are combined. Therefore, even if the accuracy of the shape in the object model space is poorer than that of the original image, it is possible to provide an image generation apparatus capable of restoring an image with a shape having substantially the same accuracy as the original image. it can.

[0032]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding members, and the duplicated description will not be repeated.

[First Embodiment] FIG. 1 is a block diagram showing a schematic configuration of an image generation system according to a first embodiment of the present invention. Referring to FIG. 1, the image generation system 1 includes an image input device 20 for inputting an image of a human face.
And an image generation apparatus 10 for generating an image in which the facial expression included in the image received from the image input apparatus 20 is changed.
An image display device 30 for displaying an image generated by the image generation device 10, and a control device 40 connected to the image input device 20, the image generation device 10 and the image display device 30 for controlling them. , And an external storage device 50 connected to the control device 40.

The image input device 20 is an image scanner and has a line type CCD (Charge Coupled Device) sensor. A photograph of a face is read, and a two-dimensional face image is output to the image generation device 10. It is also possible to use a digital camera, a digital video camera, or the like capable of photographing an actual person and outputting a two-dimensional face image as digital data. Further, the image input device 20 may be an input terminal for connecting to an external image scanner, digital camera, or the like.

The image generating apparatus 10 projects the face image received from the image input apparatus 20 onto an eigenspace described later, changes facial features such as facial expressions on the eigenspace model, and then changes the facial expression. Is decoded using the eigenspace model and output to the image display device 30.

The image generating apparatus 10 can be constituted by a personal computer or the like, and has a central processing unit (CPU).
And a read-only memory (ROM) for recording the program executed by the CPU, and a random access memory (RAM) used when the program is executed by the CPU
And a fixed recording device such as a magnetic recording device. CPU
Is connected to an interface for connecting to the image input device 20 and the image display device 30.

The image display device 30 is a display which is composed of a liquid crystal display device, a cathode ray tube or the like and which displays the image generated by the image generation device 10. A printer may be used instead of the display.

The control device 40 is connected to the image input device 20, the image generation device 10 and the image display device 30 and controls their operations. Specifically, the image input device 20 reads an image, transmits the image to the image generation device 10, inputs a change parameter for changing the facial expression of the image generation device 10, and generates an image for the image generation device 10. Instructions and transmission of the generated image to the image display device 30 are performed.

The external storage device 50 reads the program and data recorded in the computer-readable recording medium 51 and sends them to the image generation device 10 via the control device 40. In addition, according to an instruction from the image generating apparatus 10, necessary data is written in the computer-readable recording medium 51.

The computer-readable recording medium 51 is a tape system such as a magnetic tape or a cassette tape, a magnetic disk (flexible disk, hard disk, etc.) or an optical disk (CD-ROM / MO / MD / DVD).
Etc.) disk systems, IC cards (including memory cards), optical cards, etc., or mask RO
It is a medium such as a semiconductor memory such as an M, an EPROM, a flash memory, or the like that fixedly carries a program. Also,
The recording medium 51 may be a recording medium that fluidly carries the program so that the program may be downloaded from a network.

The program is a concept including not only a program directly executable by the central processing unit, but also a program in a source program format, a compressed program, an encrypted program and the like.

FIG. 2 is a diagram showing an example of an operation panel of the control device 40 of the image generation system according to the first embodiment. 2, the control device 40 includes a facial expression operation panel 41, an age operation panel 42, an OK button 47,
An NG button 48 and a reset button 49 are included. The image display device 30 is provided adjacent to the operation panel of the control device 40.
Are arranged.

The facial expression operation panel 41 displays a slide switch 41a for changing a change parameter for expressing a joyful expression, a slide switch 41b for changing a change parameter for expressing an angry expression, and a sad expression. A slide switch 41c for changing the change parameter and a slide switch 41d for changing the change parameter expressing a surprising expression are included. The age operation panel 42 includes a slide switch 42a for changing a change parameter representing age.

Each switch 41a to 41d, 42 of the facial expression operation panel 41 or the age operation panel 42 of the control device 40
The user can set a desired change parameter with a. Then, the slide switches 41a to 41d,
Each change parameter set by 42a is displayed on the OK button 4
It is transmitted to the image generation apparatus 10 by pressing 7.

The display portion of the image display device 30 is arranged near the operation panel of the control device 40. Image input device 20
When the image is further read, the read image is transmitted to the image generation device 10 and the image display device 30. The image display device 30 displays the face image received from the image input device 20 on the display unit.

In this state, the user operates the slide switches 41a to 41d of the facial expression operation panel 41 of the control device 40 and the slide switch 42a of the age operation panel 42 to set respective change parameters, and then the OK button 47 is pressed. When pressed, the set change parameter is transmitted to the image generating apparatus 10. In the image generation device 10,
A face image whose expression is changed according to the received change parameter is generated and output to the image display device 30. In the image display device 30, the face image after the facial expression is changed is displayed on the display unit.

The NG button 48 is a button for inputting an instruction to further change the facial expression of the face image generated by the image generating apparatus 10.

The reset button 49 is used for the facial expression operation panel 4
1 and a switch for returning the change parameter set on the age operation panel 42 to the default state. When the reset button 49 is pressed, the slide switches 41a ...
41d and 42a are automatically returned to the default positions.

The operation panel of the control device 40 is displayed on the display of the image display device 30, and the slide switch 41a is moved by using a pointing device such as a mouse.
˜41d, 42a may be configured. In this case, the control device 40 and the image display device 30 are integrated.

FIG. 3 is a block diagram showing a schematic function of the image generating apparatus of the image generating system according to the first embodiment. With reference to FIG. 3, the image generation device 10 extracts the shape of the human face included in the two-dimensional image received from the image input device 20 and generates a shape vector.
2, a texture extraction unit 103 that extracts a texture corresponding to the shape vector generated by the shape extraction unit 102 from the two-dimensional image received from the image input device 20, and generates a texture vector g, a reference model 111, and a feature model. A storage unit 110 for storing 112, and an encoding unit 10 for projecting a shape vector s and a texture vector g onto a reference model 111 to generate a parameter vector c.
4 and the parameter vector c generated by the encoding unit 104
Is converted using the feature model 112.
5 and the decoding unit 106 that generates the shape vector s _t and the texture vector g _t by decoding the parameter vector c _t converted by the feature conversion unit 105 in the reference model 111, and the decoding unit 106. shape determination unit 1 corrected using the generated the generated shape vector s _t in shape extraction section 102 shape vector s
07, and the texture determination unit 10 that corrects the texture vector g _t decoded by the decoding unit 106 using the texture vector g generated by the texture extraction unit 103.
8 and the shape vector s ′ corrected by the shape determination unit 107.
An image restoration unit 1 that restores a human face image using the texture vector g ′ corrected by the texture determination unit 108.
09 and.

The image input device 20 outputs an image including a human face, and the image is output to the image generation device 10.

The shape extracting unit 102 extracts the shape of the face from the image input from the image input device 20 by hand using a pointing device or the like. The shape of the face means, for example, the contour of the face, the contour of each part such as eyes, eyebrows, nose, mouth and the like. When a feature point is designated by a pointing device or the like to identify each contour, the coordinate value of the designated feature point is generated as a shape vector s. The shape vector s is a vector in which the coordinates of the feature points are arranged in a predetermined order. Therefore, it is possible to specify the contour of the human face, the contours of the eyes, the nose, the eyebrows, the mouth, and the like from the shape vector s.

It should be noted that the feature extraction can be performed automatically by performing edge extraction processing or the like on the image instead of the manual operation using a pointing device.

The generated shape vector s is used as a texture extraction unit 103, an encoding unit 104 and a shape determination unit 107.
Is output to.

The texture extraction unit 103 extracts the texture of each part of the human face using the image input from the image input device 20 and the shape vector s input from the shape extraction unit 102. More specifically, the pixel value of the feature point specified by the shape vector s is extracted, and the texture vector g is generated. The texture vector g is a vector in which the pixel values of the feature points specified by the shape vector s are arranged corresponding to each element of the shape vector s. Therefore, the shape vector s and the texture vector g define the contour and texture of the human face.

The texture vector g generated by the texture extraction unit 103 is output to the coding unit 104 and the texture determination unit 108.

In the encoding unit 104, the shape vector s and the texture extraction unit 10 received from the shape extraction unit 102.
The texture vector s received from No. 3 is normalized using appropriate normalization functions f _s and f _g . Then, the normalized shape vector f _s (s) and the normalized texture vector f _g (g) are projected onto the reference model 111 to generate a parameter vector c. The generated parameter vector c is output to the feature conversion unit 105.
Further, the normalization function f _s used in the normalization process of the shape vector s is output to the shape determination unit 107, and the normalization function f _g used in the normalization process of the texture vector g is input to the texture determination unit 108. Is output.

The normalization function f _s is used to normalize the shape vector s. The normalization here includes, for example, affine transformation such as rotation, enlargement, reduction, or parallel movement of a shape, or linear correction.

The normalization function f _g is used to normalize the texture vector g. The normalization here includes, for example, resolution conversion, density histogram conversion,
Color correction conversion etc. are included.

Reference model 11 stored in storage unit 110
1 has a lower dimension than that created by performing statistical analysis by, for example, the principal component analysis method using the shape vector and texture vector of the face obtained from each of a plurality of images including human faces. It is a statistical model. This statistical model is an eigenmodel space.

The coding unit 104 projects the shape vector s and the texture vector g on the reference model 111 created in advance. Thereby, the parameter vector c in the reference model is derived. Parameter vector c
Is a parameter vector determined from the shape vector s and the texture vector g in the reference model 111. The parameter vector c is a multidimensional vector in which elements are arranged in descending order of importance in representing a face image.

The characteristic conversion unit 105 uses the characteristic model 112 for the parameter vector c received from the encoding unit 104 to generate a parameter vector c _t obtained by converting the parameter vector c.

The characteristic model 112 will be described below. Here, as an example, a case where the facial expression component of a human face is converted will be described. In this case, the feature model 112
Is a feature model space for facial expressions. Such a feature model can be created, for example, by the following method.

(1) Prepare one image containing faces of various persons with various expressions and one image containing faceless faces.

(2) For each of these images,
The shape extracting unit 102, the texture extracting unit 103, and the encoding unit 104 calculate the parameter vector c.

(3) Extract facial expression components. The expression component is extracted by a difference vector obtained by taking the difference between the parameter vector c _e generated based on the image containing the face of the same person having the facial expression and the parameter vector c _n generated based on the image containing the expressionless face. Calculate x.

X = c _e −c _{n Since the} difference vector x is calculated from the parameter vectors c _e and c _n of the same person, only the component relating to the facial expression excluding individuality is extracted from the difference vector x. Will be. The difference vector x calculated in this manner is calculated for each of a plurality of persons and for each facial expression.

(4) For the difference vector x, for example, principal component analysis is performed for each facial expression to obtain an eigenspace for each facial expression. That is, the eigenspace for each facial expression is a difference average vector bar x obtained by averaging a difference vector from a parameter vector c representing an image including an expressionless face, and eigenvectors e ₁ , e ₂ , e ₃ , ... Around this point. Can be defined as

On the other hand, the conversion of the parameter vector c relating to the facial expression using this feature model can be performed in the following procedure.

(1) In the parameter vector c output from the encoding unit 104, the difference average vector bar x corresponding to the facial expression to be changed, and the eigenvectors e ₁ and e ₂ defining the eigenspace of the feature model 112. , E ₃ , ...
The coefficients h, k ₁ , k ₂ , k ₃ , ... Are appropriately linearly added and converted into a parameter vector c _t to which an arbitrary facial expression component is added.

This conversion equation is expressed by equation (1). The parameter vector c _t converted by the feature conversion unit 105 is output to the decoding unit 106. The decoding unit 106 uses the parameter vector c _t received from the feature conversion unit 105 and the reference model 111 to generate a shape vector s _t and a texture vector g _t . The shape vector s _t and the texture vector g _t are vectors respectively representing the shape and texture of the result to which the facial expression is added.

The correction section 121 includes a shape determining section 107 and a texture determining section 108. Shape determining unit 107
Is subjected to a conversion process using the equation (2) using the shape vector s received from the shape extraction unit 102 and the shape vector s _t representing the shape to which the facial expression is added, received from the decoding unit 106, and output Generate a shape vector s'.

Here, the transformation function f _S ^-1 is obtained by the shape extraction unit 1
This is an inverse function of the normalization function f _S used in the normalization process in 02. The output shape vector s ′ generated by the shape determination unit 107 is output to the image restoration unit 109.

The texture determining unit 108 uses the texture vector g received from the texture extracting unit 103 and the texture vector g _t representing the texture of the image added with the expression received from the decoding unit 106,
The conversion process according to the equation (3) is performed to determine the output texture vector g '.

Here, the function f _g ^-1 is the texture extraction unit 1
It is an inverse function of the normalization function f _g used in the normalization process in 03. Since the function f _g or the inverse function f _g ^-1 includes resolution conversion by thinning out pixels, the composite function f _g * f _g ^-1 that combines these functions is not necessarily an identity conversion.

By blending the change from the original image with the texture vector g _t obtained from the reference model 111 into the texture vector g obtained from the original image, the texture vector g _t obtained from the reference model alone can be restored. Since the component that cannot be corrected is interpolated by the texture vector g obtained from the original image, it is possible to output an image with more realistic image quality without unnaturalness.

The output texture vector g ′ generated by the texture determining section 108 is output to the image restoring section 109.

In the image restoration unit 109, the shape determination unit 107
Using the output shape vector s ′ thus received and the output texture vector g ′ received from the texture determining unit 108, warping processing is performed to transform the output texture vector g ′ into the shape represented by the output shape vector s ′. Then, the output image I ′ (s ′, g ′) is generated by the warping process. The generated output image I (s ′, g ′) is output to the image display device 30.

FIG. 4 is a flow chart showing the flow of image generation processing executed by the image generation apparatus 10 according to the first embodiment. With reference to FIG. 4, the image generation device 10
Then, the original image is input from the image input device 20 (step S01). The original image input from the image input device 20 includes a human face.

Then, the input original image is output to the image display device 30 (step S02). As a result, the image read by the image input device 20 is displayed on the image display device 30.

Next, the shape extraction unit 102 and the texture extraction unit 1 are applied to the original image input from the image input device.
In each of 03, a shape vector s and a texture vector g are generated, and the shape vector s and the texture vector g generated by the encoding unit 104 are projected on the reference model 111 to be encoded. Then, the parameter vector c generated as a result of the encoding is output to the feature conversion unit 105 (step S03).

A feature conversion instruction is received from control device 40 (step S04). The feature conversion instruction here is shown in FIG.
Expression operation panel 41 or age operation panel 42 shown in FIG.
The change parameter input by is received from the control device 40.

Then, based on the feature conversion instruction received from the control device 40, the difference average vector bar x corresponding to the facial expression according to the change parameter received from the feature model 112 and the eigenvector e defining the feature model 112.
_1, e _2, e _3, ... by using the convert the parameter vector c _t representing the face of the parameter vector c expression is added using the equation (1). This conversion processing is performed in the characteristic conversion unit 105.

Next, the parameters converted in step S04.
Data vector c_tWith the reference model 111.
Cutle s_tAnd texture vector g_tDecrypt to and (
Step S05). This makes it possible to add facial expressions with facial expressions.
Shape vector s that defines the image _tAnd texture vector g_t
And will be generated.

Then, the generated shape vector s _t is corrected by the shape determination unit 107 using the shape vector s generated by the shape extraction unit 102, and also generated by the texture determination unit 108 by the decoding unit 106. The texture vector g _t is corrected using the texture vector g generated by the texture extraction unit 103 (step S0
6). The shape vector s _t is corrected based on the equation (2), and the texture vector g _t is corrected based on the equation (3).

Then, the image restoration unit 109 uses the corrected shape vector s ′ and the corrected texture vector g ′ to perform warping processing for transforming the texture vector g ′ into the shape represented by the shape vector s ′. As a result, the output image is generated (step S07).

Then, the generated output image is output to the image display device 30 (step S08). As a result, an image in which the facial expression is converted into the facial expression input from the operation panel of the control device 40 is output to the image display device 30.

Next, in step S09, it is determined whether or not an OK instruction has been input from the control device 40. If the OK instruction is input, the process proceeds to step S01, and if not, the process proceeds to step S04. The OK instruction is
When the OK button 47 of the control device 40 is pressed, the image is output to the image generation device 10. When the NG button 48 is pressed, an NG instruction is output to the image generation device 10. When an NG instruction is input from control device 40, the processes of steps S04 to S08 are repeated. As a result, the user of the image generation system 1 operates the slide switch 4 on the operation panel of the control device 40.
It is possible to repeat 1a to 41d, 42a until a face image having a desired expression is output to the image display device 30.

In step S10, the output image is stored in the storage section 110 of the image generating apparatus 10. In the next step S11, it is determined whether or not another image is input from the image input device 20. If another image is input, the process proceeds to step S01, and if not, the process ends.

As described above, in the image generation system according to the first embodiment, the texture vector g obtained from the interpolated original image and the texture vector g _t obtained from the reference model 111 are blended, Texture vector g _t obtained from the reference model
Since the components that cannot be restored only by the interpolation are interpolated by the texture vector g obtained from the original image, it is possible to output an image with more realistic image quality without unnaturalness.

Next, a modification of the texture determining unit 108 and the shape determining unit 107 of the image generating system according to the first embodiment will be described.

<First Modification of Texture Determining Unit 108> In the texture determining unit 108 described above, a texture vector representing the texture of the image to which the facial expression generated by the decoding unit 106 is added based on the equation (3). g _t is the texture vector g generated by the texture extraction unit 103
Was corrected using. The deformed texture determination unit 108 determines the blending ratio by the equation (4) using the coefficient k _g . As a result, it becomes possible to adjust the blending ratio of the texture vector g generated from the input image and the texture vector g _t generated by the decoding unit 106 by adding the facial expression with the coefficient k _g . Therefore, whether the emphasis is placed on the resolution of the image to be restored or the change in facial expression is determined by the coefficient k _g.
Can be determined at. _If the value of coefficient k _g is large,
The texture vector g representing the texture of the image to which the facial expression is added is emphasized, but when the resolution of the reference model 111 is small, the resolution is also small.

As described above, by blending the texture vector g _t restored from the reference model 111 and the texture vector g extracted from the original image at an appropriate ratio, a more realistic image can be restored.

<Second Modification of Texture Determining Unit 108> Next, a second modification of the texture determining unit 108 will be described. Texture determining unit 1 in the second modified example
08 uses the texture vector g generated by the texture extraction unit 103 and the texture vector g _t representing the image to which the facial expression is added by the decoding unit 106, using the equation (5) with an appropriate coefficient k _g . The texture vector g'is determined. Here, the function f _g ^-1 is an inverse function of the conversion function f _g used in the shape normalization process. The output texture vector g ′ is output to the image restoration unit 109.

In the equation (5), the inverse normalization conversion is applied to the change amount of the texture vector g _t for the image whose expression is changed by the feature conversion unit 105, and this is adjusted by the coefficient k _g which is the mixture ratio. Add to the texture vector g extracted from the original image. Therefore, the reference model 111
Even if the resolution of the texture is lower than the resolution of the original image, it is possible to restore an image having the same resolution as the original image. Further, the ratio of blending the change in texture can be determined by the coefficient k _g . As a result, it is possible to reduce the influence of the coding distortion generated in the coding process performed by the coding unit 104 and the decoding process performed by the decoding unit 130 on the image quality.

<Third Modification of Texture Determining Section 108> The texture determining section 108 in the third modification is
Using the expression (6), the texture vector g _t, which is generated by the decoding unit 106 and represents the texture of the image to which the facial expression is added, is corrected using the texture vector g generated by the texture extraction unit 103. In the equation (6), the function K _g outputs a value controlled by the shape vector s _t which is generated by the shape determining unit 107 and represents the shape of the image to which the facial expression is added. As a result, the value output by the function K _g can be changed depending on the face shape portion. For example, in the mouth and eyes, K _g = 1.0
And then, other portions is the value to be output as K _g = 0.0 it is possible to be controlled by the shape vector s _t. That is, the mouth and eyes are texture vectors g _t
Can be reflected 100%, and the texture vector g _t can be omitted for other face portions. As a result, the facial expression can be partially changed while maintaining the resolution of the texture vector g generated by the texture extraction unit 103.

<First Modification of Shape Determining Unit 107> In the first modification of the shape determining unit 107, the image added with the facial expression generated by the decoding unit 106 using the equation (7). shape vector s _t a shape extraction unit representing the shapes 102
The correction is performed using the shape vector s generated in (1). The shape vector s and the shape vector s to which the facial expression is added
_The proportion at which _t and is compounded is defined by the coefficient k _s . Therefore, the shape vector s generated from the input image
If, it is possible to adjust the mixing ratio of the generated shape vector s _t in the decoding unit 106 is added expression by a factor k _g. Therefore, whether the emphasis is placed on the resolution of the image to be restored or the change in facial expression is determined by the coefficient k.
It can be determined by _g . _If the value of coefficient k _g is large,
The shape vector s representing the shape of the image to which the facial expression is added is emphasized, but when the resolution of the reference model 111 is small, the resolution is also small.

<Second Modification of Shape Determining Unit 107> Second
In the shape determining unit 107 in the modified example of (8)
Using Equation, shape vector s _t that generated expression representing the shape of the added image decoding unit 106 is corrected using the shape vector s produced by the shape extracting unit 102. (8) The expression contains a function K _s value to be output by the shape vector s _t representing the shape of the image expression is added is controlled. Thus, the value of the function K _s is output, a portion of the shape of the face, for example mouth and eye portion, the output of the function K _s is "1.0", the function K _s in other parts Output Can be like "0.0". That is, mouth and eyes of the parts using a shape vector s _t representing the shape of the image expression is added, with respect to portions of the other face may be corrected to use the shape vector s.

It is also possible to combine the above-described shape determining unit 107 or its modification with the texture determining unit 108 or its modification. For example, the second modification of the shape determining unit 107 and the third modification of the texture determining unit 108.
The reference model 11
Shape vector s restored by changing the facial expression using 1
Whether _t and the texture vector g _t are used or whether the shape vector s and the texture vector g obtained from the original image are used can be selectively weighted by the face shape portion. For example, for the mouth, eyes, nose, eyebrows, contour, etc. of the face, it is possible to change which data is used and which data is used with emphasis. For this reason, for example, even if the original image is changed from an image containing a face with closed eyes or mouth to an image with open eyes or mouth, the shape of the eyes or mouth is not changed. Textures (in this case, eyes, teeth, etc.) can be restored.

[0100]

[Equation 1]

[Second Embodiment] Next, an image generation system according to a second embodiment will be described. Second
The image generation system in the embodiment is a modification of the image generation device 10 of the image generation system in the first embodiment. Since the other points are the same as those of the image generation system according to the first embodiment, only different points will be described.

FIG. 5 is a block diagram showing a schematic function of the image generating apparatus 60 according to the second embodiment. With reference to FIG. 5, the image generation device 60 according to the second embodiment.
Is a modification of the decoding unit 106 and the correction unit 121 in the first embodiment. The other points are the same as those of the image generating apparatus 10 in the first embodiment.

Image generation device 6 in the second embodiment
Decoding unit 130 of 0, in addition to generating the shape vector s _t and texture vector g _t using the reference model 111 from the parameter vector c _t receive from the feature transform unit 105, the encoding unit 104 outputs The shape vector s from the parameter vector c using the reference model 111
Generate _O and the texture vector g _O. Decoding section 1
30 resulting shape vector s _t, s _O is output together to the shape determination unit 132. The texture vectors g _t and g _O generated by the decoding unit 130 are both output to the texture determination unit 133.

[0104] In the shape determining unit 132, by using the equation (9), the shape vector s _t is corrected. This correction, shape decoded from parameter vector c vector s _O
And the shape vector s generated by the shape extraction unit 102 are added. The output shape vector s ′ generated using the equation (9) is output to the image restoration unit 109.

By adding the difference between the shape vector s _t after changing the facial expression in the reference model 111 and the shape vector s ₀ before changing the facial expression to the shape vector s obtained from the original image, the reference model The component that cannot be restored only by the shape vector s _t obtained from the above can be interpolated by the shape vector s obtained from the original image.

The texture determining unit 133 corrects the texture vector g _t representing the texture of the image added with the facial expression, which is generated by the decoding unit 130, using the expression (10). In this correction, the texture vector g _O decoded from the parameter vector c and the texture vector g generated by the texture extraction unit 103 are taken into consideration. The output texture vector g ′ generated using the equation (10) is output to the image restoration unit 109.

By adding the difference between the texture vector g _t after the facial expression is changed by the reference model 111 and the texture vector g ₀ before the facial expression is changed to the texture vector g obtained from the original image, The component that cannot be restored only by the texture vector g _t obtained from the above can be interpolated by the texture vector g obtained from the original image.

As described above, in the image generation system according to the second embodiment, the reference model 111
Only the component obtained by performing the feature conversion between the shape vector s _t and the texture vector g _t obtained by performing the feature conversion above is
The shape vector s and the texture vector g obtained from the original image are added respectively. Therefore, even if the face included in the image to be processed does not fit the reference model 111 well, and the image restored by using the reference model 111 has insufficient image quality, the image quality deteriorates. Can be prevented. That is, the reference model 11
Even if the resolution of the texture 1 is lower than the resolution of the original image, it is possible to restore the image having the same resolution as the original image.

Further, it is possible to reduce the influence of the coding distortion caused by the coding process performed by the coding unit 104 and the decoding process performed by the decoding unit 130 on the image quality.

<First Modification of Texture Determining Section 133> In the first modification of texture determining section 133,
Using the equation (11), the texture vector g _{t, which} is generated by the decoding unit 130 and represents the texture of the image to which the facial expression is added, is corrected. For this correction, the degree of correction is determined by an appropriate coefficient k _g .

The corrected texture vector g'is the texture vector g generated from the input image,
A mixture ratio of the texture vector g _t generated by the decoding unit 106 to which the facial expression is added and the change amount on the reference model 111 is adjusted by a coefficient k _g . Therefore, it is possible to determine the proportion in which the variation is mixed by the coefficient _kg .

<Second Modification of Texture Determining Section 133> In the texture determining section 133 in the second modification,
The texture vector g _t representing the texture of the image added with the facial expression generated by the decoding unit 130 is corrected using the expression (12).

For this correction, the texture vector g _O before the expression added by the decoding unit 130 and the texture vector g before the decoding are used. (1
In the expression (2), the function K _g is a function whose output is controlled by the shape vector s _t which is generated by the decoding unit 130 and represents the shape of the image to which the facial expression is added. For example, the output of the function K _g is set to “1.0” for a part of the shape of the face, for example, the mouth and eyes, and the other part is changed to “0.0” for the output of the function K _g . It is possible. Therefore, the change on the reference model 111 can be partially blended.

As a result, it is possible to reduce the influence of the coding distortion caused by the coding and decoding of the shape vector and the texture vector on the image quality, and to change only a part of the face.

<First Modification of Shape Determining Unit 132> In the first modification of the shape determining unit 132, the image added with the facial expression generated by the decoding unit 130 using the equation (13). shape vector s _t representing the shape is corrected. The feature conversion unit 105 performs inverse normalization conversion on the change amount of the texture obtained from the texture vector s ₀ before the expression of the texture vector s _{t to which} the expression is added, and the texture extracted from the original image. When blending with respect to the vector g, the blending ratio is determined by the coefficient k _s .

Therefore, the shape vector s generated from the input image and the facial expression are added to the decoding unit 106.
In the the mixing ratio of the change in the on reference model 111 of the generated shape vector s _t can be adjusted by a factor k _g. Therefore, even when there is a difference in resolution between the input image and the restored image, it is possible to appropriately deal with it.

<Second Modification of Shape Determining Unit 132> Second
The shape determining unit 132 in the modified example uses the equation (14).
Then, the shape vector is corrected. The shape vector s'is
Shape vector s output by the decoding unit 130 _t, S_Oand
Using the shape vector s output by the shape extraction unit 102,
It is determined. Further, the function K in the equation (14)_sIs the
Shape vector s output by the encoding unit 130_tDepending on the value of
Is a function whose output changes. For example, the function K_sOut of
Force in the case of part of the shape of the face, for example the mouth and eyes
Is "1.0", and other parts are "0.0"
So that the output can be made different depending on the difference of the parts.
Is a function that As a result, on the reference model 111
The change in the shape can be partially blended.

The shape determining unit 132 or the shape determining unit 132 described above is used.
The modification and the texture determination unit 133 or its modification are
It is also possible to combine. For example, the shape determining unit 1
32 second modification and texture determination unit 133 second
By combining with the modification, the reference model 111
Shape vector s restored by changing the facial expression using _t
And texture vector g_tOn the reference model 111 of
Shape vector obtained from the original image
s and texture vector g
Can be selectively weighted. Was
For example, face, mouth, eye, nose, eyebrows, outline, etc.
Data, and which data to use
Can be changed. Therefore, for example,
If the image contains a face with closed eyes or mouth,
When the facial expression is changed to an open facial expression image,
Even the shape and texture of the eyes and mouth (in this case the eyes
It becomes possible to restore spheres, teeth, etc.).

[0119]

[Equation 2]

The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of an image generation system in one of the first embodiments.

FIG. 2 is a diagram illustrating an example of an operation panel of a control device of the image generation system according to the first embodiment.

FIG. 3 is a block diagram showing a schematic function of an image generation device of the image generation system according to the first embodiment.

FIG. 4 is a flowchart showing a flow of image generation processing executed by the image generation apparatus according to the first embodiment.

FIG. 5 is a block diagram showing a schematic function of an image generating apparatus according to a second embodiment.

[Explanation of symbols]

1 image generation system, 10,60 image generation device, 2
0 image input device, 30 image display device, 40 control device, 41 facial expression operation panel, 42 age operation panel, 4
1a, 41b, 41c, 41d, 42d slide switch, 50 external storage device, 51 recording medium, 102
Shape extraction unit, 103 texture extraction unit, 104 encoding unit, 105 feature conversion unit, 106 decoding unit, 10
7, 132 shape determination unit, 108, 133 texture determination unit, 109 image restoration unit, 110 storage unit, 111
Reference model, 112 feature model, 121 correction unit, 1
30 Decoding unit.

Continued front page    F-term (reference) 5B050 BA06 CA07 DA01 EA04 EA28                       FA02                 5B057 BA11 CA16 CB18 CG07 CH08                       CH11                 5B080 BA07 GA22

Claims (11)

[Claims] 1. A step of receiving an input of an original image; a step of extracting shape data representing a shape of an object and texture data representing a texture corresponding to the shape data from the input original image; Generating a parameter in the object model space by projecting the extracted shape data and texture data in an object model space obtained by statistically analyzing a plurality of object images having; Changing at least one element of the parameter, reconstructing shape data and texture data of the object corresponding to the changed parameter, and using texture data extracted from the original image,
Compensating the reconstructed texture data, and generating an image using the reconstructed shape data and the compensated texture data,
Image generation method. 2. The image generation method according to claim 1, further comprising a step of correcting the reconstructed shape data by using shape data extracted from the original image. 3. The step of correcting the reconstructed texture data corrects the reconstructed texture data using texture data extracted from the original image and shape data extracted from the original image. The image generation method according to claim 1, further comprising: 4. A step of receiving an input of an original image; a step of extracting shape data representing a shape of an object and texture data representing a texture corresponding to the shape data from the input original image; Generating a parameter in the object model space by projecting the extracted shape data and texture data in an object model space obtained by statistically analyzing a plurality of object images having; Changing at least one element of the parameter; reconstructing shape data and texture data of the object corresponding to the changed parameter; and using the shape data extracted from the original image, A screen for correcting the configured shape data. Tsu including a flop, and generating an image using said corrected shape data and texture data the reconstructed,
Image generation method. 5. The image according to claim 4, further comprising the step of correcting the reconstructed texture data using texture data extracted from the original image and shape data extracted from the original image. Generation method. 6. The object according to claim 1, wherein the object is a human face, and the step of changing an element of the parameter includes a step of changing a parameter belonging to a facial expression category or an age category. The image generation method according to claim 1. 7. A step of receiving an input of an original image; a step of extracting shape data representing a shape of an object and texture data representing a texture corresponding to the shape data from the input original image; Generating a parameter in the object model space by projecting the extracted shape data and texture data in an object model space obtained by statistically analyzing a plurality of object images having; Changing at least one element of the parameter, reconstructing shape data and texture data of the object corresponding to the changed parameter, and using texture data extracted from the original image,
An image generation program that causes a computer to execute a step of correcting the reconstructed texture data and a step of generating an image using the reconstructed shape data and the corrected texture data. 8. A step of receiving an input of an original image; a step of extracting shape data representing a shape of an object and texture data representing a texture corresponding to the shape data from the input original image; Generating a parameter in the object model space by projecting the extracted shape data and texture data in an object model space obtained by statistically analyzing a plurality of object images having; Changing at least one element of the parameter; reconstructing shape data and texture data of the object corresponding to the changed parameter; and using the shape data extracted from the original image, A screen for correcting the configured shape data. And-up, and a step of generating an image using said corrected shape data and texture data the reconstructed on the computer, the image generating program. 9. A computer-readable recording medium in which the image generation program according to claim 7 or 8 is recorded. 10. An accepting means for accepting an input of an original image, and an extracting means for extracting shape data representing an object shape and texture data representing a texture corresponding to the shape data from the input original image, the same. Parameter generation means for generating parameters in the object model space by projecting the extracted shape data and texture data in the object model space obtained by statistically analyzing a plurality of object images having attributes, Changing means for changing at least one element of the generated parameter; reconstructing means for reconstructing the shape data and texture data of the object corresponding to the changed parameter; and the texture extracted from the original image. Using the data,
An image generation apparatus comprising: a correction unit that corrects the reconstructed texture data; and an image generation unit that generates an image using the reconstructed shape data and the texture data corrected by the correction unit. . 11. Receiving means for receiving an input of an original image, and extraction means for extracting shape data representing a shape of an object and texture data representing a texture corresponding to the shape data from the input original image Parameter generation means for generating parameters in the object model space by projecting the extracted shape data and texture data in the object model space obtained by statistically analyzing a plurality of object images having attributes. Changing means for changing at least one element of the generated parameter; reconstructing means for reconstructing the shape data and texture data of the object corresponding to the changed parameter; and the shape extracted from the original image. Using the data, the reconstructed shape data And correcting means for correcting, with an image generating means for generating an image using the corrected shape data and texture data the reconstructed by said correction means, the image generating apparatus.