WO2006129791A1 - 画像処理システム、3次元形状推定システム、物体位置姿勢推定システム及び画像生成システム - Google Patents
画像処理システム、3次元形状推定システム、物体位置姿勢推定システム及び画像生成システム Download PDFInfo
- Publication number
- WO2006129791A1 WO2006129791A1 PCT/JP2006/311087 JP2006311087W WO2006129791A1 WO 2006129791 A1 WO2006129791 A1 WO 2006129791A1 JP 2006311087 W JP2006311087 W JP 2006311087W WO 2006129791 A1 WO2006129791 A1 WO 2006129791A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- individual
- illumination
- image
- general
- parameter
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/10—Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/60—Extraction of image or video features relating to illumination properties, e.g. using a reflectance or lighting model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/169—Holistic features and representations, i.e. based on the facial image taken as a whole
Definitions
- Image processing system 3D shape estimation system, object position and orientation estimation system, and image generation system
- the present invention relates to an image processing system, an image processing method, and an image processing program for processing an input image.
- an input image whose position and orientation and lighting conditions are unknown, a three-dimensional shape of an object that is reflected in the input image and whose shape and texture are not divided, or a position Z and posture of the object are determined.
- the present invention relates to a 3D shape estimation system, an object position / orientation estimation system, a 3D shape estimation method, and an object position / orientation estimation method.
- the present invention also relates to an image generation system and an image generation method for generating a reproduction image that reproduces an input image.
- a method for preparing a predetermined deformation model in advance and generating a 3D shape and texture has been developed.
- a general-purpose deformation model is prepared by measuring a certain number of objects in advance for a set of objects similar to various human faces. Then, by giving appropriate numerical parameters to the prepared general-purpose deformation model, 3D shapes and textures of individual objects that have not been measured are generated.
- an example of a technique for estimating a 3D shape and a texture using a predetermined deformation model Is described in Non-Patent Document 2.
- the 3D shape and texture of a large number of human faces are measured in advance using a three-dimensional shape measuring device, and measured from the measured data group.
- a general-purpose 3D face model that can generate the 3D shape and texture of the face is generated.
- This general-purpose face model stores the average values of the 3D shapes and textures of a large number of human faces and the basis vector group representing the amount of change from these average values.
- the 3D shape and texture of an arbitrary person's face are approximated by adding the product multiplied by to the average value. In other words, if the parameters for shape and texture are specified, 3D shape data and texture of individual human faces can be obtained.
- Non-Patent Document 2 When the general-purpose face model described in Non-Patent Document 2 and the CG technique described in Non-Patent Document 1 are combined, a desired image can be reproduced.
- the desired image can be reproduced by designating each 3D shape, texture, position and orientation, and lighting condition parameters for an individual object.
- FIG. 11 is a block diagram showing an example of the configuration of a 3D shape estimation system that estimates the 3D shape of an object based on a general-purpose face model.
- the conventional three-dimensional shape estimation system based on the general-purpose face model includes general-purpose face model generation means 900, illumination condition initial value input means 901, position and orientation initial value input means 902, face model generation means. 903, perspective transformation means 904, shadow shadow generation means 905, and parameter update means 906 are included.
- the conventional 3D shape estimation system having such a configuration operates as follows.
- General-purpose face model generation means 900 generates a general-purpose face model (also referred to as a general-purpose 3D face model) in advance. Further, the general-purpose face model generation means 900 outputs the generated general-purpose 3D face model to the face model generation means 903.
- a general-purpose face model also referred to as a general-purpose 3D face model
- Illumination condition initial value input means 901 provides an approximate value of an illumination parameter indicating an illumination state when an input image is captured of an input image in which an object whose 3D shape is to be estimated is captured. Enter as initial value.
- the illumination condition initial value input means 901 has, as an example, the direction of a point light source located at one infinity as (1 0, 1 ⁇ ), and the three-dimensional point light source The direction vector 1, intensity L, and the value a indicating the intensity of diffused light are input as illumination parameters.
- an input image and a three-dimensional image generated by the CG technique described in Non-Patent Document 1 are used as a computer terminal. Display them side by side. Then, while watching the screen of the computer terminal, it is possible for the user (for example, an operator) to input lighting parameters by manually adjusting so that the shadow and shadow as close as possible to the input image are displayed. it can.
- Position / orientation initial value input means 902 inputs an approximate value of a parameter (referred to as a position / orientation parameter) indicating the position and orientation of an object shown in the input image as an initial value.
- a position / orientation parameter indicating the position and orientation of an object shown in the input image as an initial value.
- the position / orientation initial value input means 902 for example, when the position of the object is a three-dimensional coordinate vector o and the direction of the object is represented by rotation angles ⁇ , 0, and ⁇ around the x, y, and z axes. Then, the matrix R indicating the rotation of these rotation angles is input as a position and orientation parameter.
- a 3D image generated by the CG technique described in Non-Patent Document 1 is displayed along with the input image on the computer terminal. . Then, while viewing the screen of the computer terminal, the user can input the position and orientation parameters by manually adjusting the object so that the object is displayed as close as possible to the input image.
- the user selects the position of a characteristic part of the object and inputs the position of the selected part on the image. Then, the three-dimensional coordinate data of each part is stored in advance, and the position and orientation parameters of the object are calculated using the three-dimensional coordinate data of the part corresponding to the input position among the three-dimensional coordinate data stored in advance. It is also possible.
- the face model generation unit 903 stores the general-purpose 3D face model calculated in advance by the general-purpose face model generation unit 900. Further, the face model generation means 903 inputs the 3D shape parameter and the texture parameter, and uses the general-purpose 3D face model to obtain the individual 3D shape unique to the person described in the input parameter and the individual texture data. Calculate and output.
- the general-purpose 3D face model is an average value ⁇ bar S calculated from the three-dimensional shape ⁇ S ' ⁇ and texture data ⁇ ' ⁇ of a large number of people (for example, about 200 people) measured in advance. , Bar ⁇ and the basis vector group ⁇ S, ⁇ indicating the deviation from the average value.
- the means 900 creates a matrix (covariance matrix) S in which three-dimensional shape vectors obtained by subtracting the average value are arranged using Equation (1).
- the general-purpose face model generation means 900 obtains eigenvalues and eigenvalues of the covariance matrix S, and corresponds to ⁇ (for example, about 100) from the obtained eigenvalue ⁇ 2 having a large value.
- the basis vector group ⁇ S ⁇ to be stored is stored. Then the row with n eigenvalues in the diagonal component
- General-purpose face model generation means 9 where ⁇ is a column and V is a matrix of basis vectors ⁇ S ⁇ .
- the general-purpose face model generation unit 900 calculates a total of about 100 eigenvalues ⁇ 2 and basis vectors ⁇ by calculating ⁇ and ij by calculating the texture in exactly the same way as the 3D shape data. And remember.
- the basis vector, the eigenvalue data, and the force general-purpose 3D face model obtained by the general-purpose face model generation unit 900 are obtained.
- the three-dimensional shape data of a face of an individual person is obtained from three-dimensional coordinate values (X, y.
- Equation (3) ⁇ s ⁇ is a shape parameter, and ⁇ t ⁇ is a texture parameter. That is, the face model generation means 103 receives the shape parameter ⁇ s ⁇ and the texture parameter ⁇ t ⁇ , and calculates the individual 3D shape S and the individual texture T using Equation (3). Output.
- the 3D shape estimation system shown in FIG. 11 repeatedly performs update processing of each parameter, and outputs each parameter value when a predetermined convergence condition is satisfied as a 3D shape estimation result. .
- the face model generation means 903 calculates the individual 3D shape S and the individual texture T using the value 0 as the initial values of the shape parameter and the texture parameter at the first processing of the iterative processing.
- the perspective transformation means 904 receives the individual 3D shape S, the individual texture T, and the position / orientation meter of the object as input, and determines which pixel in the reproduced image to generate each data point of the 3D shape. The correspondence table shown is calculated. If the focal length of the camera that captured the input image is f and the center of the image is (c, c), the perspective transformation means 904 calculates the image coordinates (u, V) of the point i of the 3D face model as ) To calculate.
- the perspective transformation means 904 divides the entire surface of the three-dimensional shape into triangular polygons, calculates (u,) for each vertex of the polygon, fills the points inside the polygon, and uses the value of Z Perform hidden surface removal.
- This hidden surface removal process is a standard technique of CG technology and can be performed using various graphics library programs.
- the perspective conversion means 904 can obtain a correspondence table indicating the correspondence relationship of the polygon j corresponding to each pixel i of the reproduced image.
- the shadow shadow generation means 905 receives the individual 3D shape, the correspondence table, and the individual texture, and calculates the luminance value of each pixel of the reproduced image.
- the shadow generation unit 905 can perform a shadow generation process using a standard method of CG technology described in Non-Patent Document 1, for example.
- the shadow shadow generating means 905 calculates a normal vector n for each data point i of the individual three-dimensional shape S.
- the other two vertex data points of the triangular polygon to which point i belongs on the object surface are set to (j, k) counterclockwise and point (i, j, p, p, a 3D vector representing the 3D coordinates of k)
- the shadow shadow generating means 905 calculates the luminance value I of the pixel i by the equation (6) using the Phong reflection model (Non-patent Reference 1).
- ⁇ ⁇ ) is the texture luminance value of polygon j corresponding to pixel i indicated by the correspondence table.
- K is the specular reflection constant (fixed value)
- V is the specular reflection characteristic (fixed value).
- C is a value indicating whether or not the polygon j is covered with a shadow with respect to the light source direction 1, and takes a value of 0 or 1.
- the shadow shadow generation means 905 can determine the cast shadow by calculating using a standard method of CG technology such as a ray tracing method.
- the shadow shadow generation unit 905 calculates a luminance value for each pixel of the reproduced image, and generates a reproduced image based on the calculation result.
- I (I, I,..., I) is a vector in which the luminance values of the pixels where the corresponding shape data points exist are arranged.
- the parameter update unit 906 compares the input image with the reproduction image generated by the shadow shadow generation unit 905, and the position and orientation, illumination, three-dimensional shape, and texture of the reproduction image are close to the input image. Change each parameter.
- the three-dimensional shape estimation system repeats the process using the face model generation unit 903, the perspective conversion unit 904, the shadow shadow generation unit 905, and the meter update unit 906 described above, thereby obtaining an optimum parameter value. Explore.
- the parameter updating means 906 first calculates the cost function E related to the similarity of images by the equation (7).
- the parameter updating unit 906 obtains a cost function E for the shape parameter and the texture parameter according to the model prior probability distribution using Equation (8).
- the meter updating means 906 searches for a parameter that minimizes the cost function E by the stochastic descent method. Therefore, the parameter update means 906 calculates the differential city of E for each parameter of shape, texture, illumination, and position and orientation using equation (9).
- Meter updating means 906 updates the parameter to a new value a * using equation (10).
- the meter update unit 906 determines that the iterative process has converged and ends the process. Further, for example, the parameter updating unit 906 has a smaller error between the reproduced image generated using the updated parameter and the input image than the error between the reproduced image generated using the parameter before the update and the input image. If not, it is determined that the repeated process has converged, and the process ends.
- the 3D shape estimation system is the closest to the input image during the iterative process! ⁇ The individual 3D shape when the image can be generated is output as the 3D shape estimation result.
- Non-patent literature l Manson Woo, Jackie Neider, Tom Davis, "OpenGL Programming Guide (Original 2nd Edition)", Addison 'Wesley'Publicishers' Japan, p. 16
- Non-Patent Document 2 Volker Blanz, Thomas Vetter, "Face Recognition Based on Fitting a 3D Morphable Model", PAMI, 2003, vol. 25, No. 9, pp.1063— 1074
- the first problem is that the initial value of the illumination condition meter is required and the input burden of the initial value of the illumination parameter is large.
- the illumination parameter must be a physical parameter that represents the intensity of the diffuse light source and the direction and intensity of the point light source, and should describe the actual illumination environment in which the input image was taken.
- the position and orientation parameters of an object are easy to understand sensuously, so it is not difficult to input initial values of position and orientation parameters.
- a user for example, an operator manually determines lighting parameter values that approximate this shooting environment. Is a difficult and laborious task.
- the second problem is that the relationship between the brightness value of the image to be generated and the illumination parameter is non-linear, and the processing load is large. In other words, many iterations are required to search for the optimum parameters for the nonlinear model, so the processing time is large and the processing burden is large. Also, since many iterations are required for a nonlinear model, it falls into a local solution and it is difficult to accurately estimate parameters immediately.
- the present invention examines the initial values of the illumination condition parameters in advance, Image processing system that can process images without input by moving, 3D shape estimation system, object position and orientation estimation system, image generation system, image processing method, 3D shape estimation method, object position and orientation estimation method
- An object of the present invention is to provide an image generation method and an image processing program.
- the present invention speeds up the process by reducing the number of iterations, and also enables an image processing system, a three-dimensional shape estimation system, and an object position / orientation estimation that can avoid a local solution and estimate an accurate parameter.
- An object is to provide a system, an image generation system, an image processing method, a three-dimensional shape estimation method, an object position and orientation estimation method, an image generation method, and an image processing program.
- the present invention provides an image processing system, a three-dimensional shape estimation system, an object position / posture estimation system, an image generation system, an image processing method, a three-dimensional shape, which can speed up the calculation processing of shadows and shadows in image generation.
- An object is to provide an estimation method, an object position / posture estimation method, an image generation method, and an image processing program.
- the present invention provides an image processing system, a three-dimensional shape estimation system, an object position / posture estimation system, an image generation system, which can perform high-speed calculation of an illumination base required when using a linear illumination model,
- An object is to provide an image processing method, a three-dimensional shape estimation method, an object position and orientation estimation method, an image generation method, and an image processing program.
- the image processing system uses a general-purpose 3D object model obtained based on 3D shape data and texture data of a plurality of similar objects, to generate 3D shape parameters and texture parameters.
- the individual data generation means for example, realized by the face model generation means 103 that generates individual three-dimensional shape data indicating the three-dimensional shape of the individual object and individual texture data indicating the texture of the individual object.
- the first illumination basis calculation means (for example, realized by the illumination basis calculation means 201) and the first illumination basis calculation means Using the individual illumination basis group, a reproduced image which is an image reproducing the same illumination condition as the input image 3D shape parameters given to the general-purpose 3D object model so that the reproduced image generated by the reproduced image generating means (for example, realized by the illumination correction means 203) and the reproduced image generated by the reproduced image generating means are close to the input image.
- the “3D shape parameter” is a parameter that indicates what deformation is applied to the standard shape data and texture data to obtain the 3D shape of the object.
- the “texture parameter” is a parameter that indicates what kind of variation is given to the standard shape data and the texture data to form the texture of the object.
- the first illumination basis calculation means obtains an individual illumination base using a spherical harmonic function based on the individual three-dimensional shape data generated by the individual data generation means.
- the first illumination basis calculation means includes: illumination condition setting means for setting a plurality of different illumination conditions (for example, realized by a CPU of an information processing device that operates according to a program); Luminance calculation means (for example, realized by a CPU of an information processing device that operates according to a program) that calculates the luminance value of each part of the object surface under each illumination condition set by the illumination condition setting means, and the luminance calculation means It may also include a calculation means (for example, realized by a CPU of an information processing device that operates according to a program) for obtaining an individual illumination base group based on the measured luminance value.
- the image processing system uses a general-purpose 3D object model obtained based on 3D shape data and texture data of a plurality of similar objects, and based on the 3D shape parameter and the texture parameter.
- Individual data generation means for generating individual 3D shape data indicating the three-dimensional shape of individual objects and individual texture data indicating the texture of individual objects, and individual objects corresponding to general-purpose three-dimensional object models
- a general-purpose illumination base model generator for generating a general-purpose illumination base model for generating an illumination base, which is data indicating basic fluctuations in the degree of reflection due to differences in the direction and intensity of the illumination applied to 3D surfaces of individual objects in different lighting conditions using a stage (for example, realized by the general lighting base model generation means 300) and a general lighting base model generated by the general lighting base model generation means
- second illumination basis calculation means for example, realized by the illumination basis model calculation means 301 for obtaining individual illumination bases each indicating a variation in the luminance value of each part.
- the image processing system includes an output unit that outputs the individual 3D shape data generated by the individual data generation unit and the individual illumination basis generated by the second illumination basis calculation unit. May be.
- the image processing system uses the individual illumination basis group obtained by the second illumination basis calculation means to generate a reproduction image generation means for generating a reproduction image that is an image that reproduces the same illumination condition as the input image.
- Parameter updating means for repeatedly updating the three-dimensional shape parameter and texture parameter given to the general-purpose three-dimensional object model until a predetermined convergence condition is satisfied so that the reproduced image generated by the reproduced image generating means is close to the input image. May be provided.
- the second illumination basis calculation means includes an individual illumination base corresponding to individual 3D shape data generated by the general illumination basis model generation means using the general-purpose 3D object model, It may be obtained using the same parameters as the 3D shape parameters given to the general 3D object shape model.
- the general-purpose illumination base model generation means is realized by a parameter setting means (for example, a sample parameter setting means 3001) for setting a random three-dimensional shape parameter to be given to the general-purpose three-dimensional object model. And the illumination base group corresponding to each parameter is obtained based on the sample 3D shape group generated by applying the parameter group set by the parameter setting means to the general-purpose 3D object model.
- 3 illumination basis calculation means for example, realized by the illumination basis calculation means 3002), the illumination basis group obtained by the third illumination basis calculation means, the basis vectors and eigenvalues of the general-purpose 3D object model, Is used to calculate the basis vector of the general illumination basis model (for example, realized by the basis model calculation means 3003). That) and even those, including,.
- the third illumination basis calculation means may obtain an individual illumination basis using a spherical harmonic function based on the individual three-dimensional shape data.
- the third illumination basis calculation means includes: illumination condition setting means for setting a plurality of different illumination conditions (for example, realized by a CPU of an information processing device that operates according to a program); Luminance calculation means (for example, realized by a CPU of an information processing device that operates according to a program) that calculates the luminance value of each part of the object surface under each illumination condition set by the illumination condition setting means, and the luminance calculation means Calculation means (for example, realized by a CPU of an information processing device that operates according to a program) that obtains an individual illumination basis group by principal component analysis based on the calculated luminance value may be included.
- illumination condition setting means for setting a plurality of different illumination conditions
- Luminance calculation means for example, realized by a CPU of an information processing device that operates according to a program
- the luminance calculation means Calculation means for example, realized by a CPU of an information processing device that operates according to a program
- the three-dimensional shape estimation system is a three-dimensional shape estimation system that estimates the three-dimensional shape of an object using an image processing system, and when a predetermined convergence condition is satisfied, Based on 3D shape parameters and texture parameters, equipped with shape output means (for example, output means provided in an information processing apparatus for realizing an image processing system) that outputs a 3D shape generated by a general-purpose 3D object model It is characterized by that.
- shape output means for example, output means provided in an information processing apparatus for realizing an image processing system
- An object position / orientation estimation system is an object position / orientation estimation system that uses an image processing system to estimate the position and orientation of an object, and when a predetermined convergence condition is satisfied, A position / orientation output unit that outputs a position / orientation parameter indicating the position and orientation of an object (for example, an output unit included in an information processing apparatus that implements an image processing system) is provided.
- An image generation system is an image generation system that uses an image processing system to generate a reproduced image that is an image that reproduces the same illumination conditions as an input image, and when a predetermined convergence condition is satisfied. And an image output means for outputting a reproduced image at the time of convergence (for example, an output means included in an information processing apparatus for realizing an image processing system).
- An image processing method is a method for processing three-dimensional shape data and text of a plurality of similar objects.
- the general-purpose 3D object model obtained based on the imager data based on the 3D shape parameter and the texture parameter, the individual 3D shape data indicating the 3D shape of the individual object and the texture of the individual object Based on the generated individual 3D shape data and the individual texture data, and the brightness value of each part of the 3D surface of the individual object under a plurality of different illumination conditions.
- a reproduction image generation that generates a reproduction image that is an image that reproduces the same illumination conditions as the input image Steps and 3D shape parameters and text given to the general-purpose 3D object model so that the generated reproduced image is close to the input image.
- the parameters characterized in that it comprises a parameter update step of repeatedly updated until a predetermined convergence condition is satisfied.
- the image processing method is to obtain an individual illumination basis using a spherical harmonic function based on the individual three-dimensional shape data generated in the individual data generation step in the first illumination base calculation step.
- the first illumination basis calculation step a plurality of different illumination conditions are set, the brightness values of the respective parts of the object surface under each set illumination condition are obtained, and the obtained brightness values are obtained. Based on this, the individual illumination base group may be obtained.
- the image processing method uses a general-purpose 3D object model obtained based on 3D shape data and texture data of a plurality of similar objects, and converts a 3D shape parameter and a texture parameter. Based on the individual data generation step for generating individual 3D shape data indicating the three-dimensional shape of the individual object and individual texture data indicating the texture of the individual object, and corresponding to the general-purpose three-dimensional object model.
- the second may include an illumination basis calculation step of obtaining is.
- the image processing method includes individual 3D shape data generated in the individual data generation step. And an output step for outputting the individual illumination basis generated in the second illumination basis calculation step.
- the image processing method uses the individual illumination basis group obtained in the second illumination basis calculation step to generate a reproduction image generation step that generates a reproduction image that is an image that reproduces the same illumination condition as the input image.
- a parameter updating step that repeatedly updates the 3D shape parameters and texture parameters given to the general-purpose 3D object model until a predetermined convergence condition is satisfied so that the generated reproduced image is close to the input image.
- the individual illumination base corresponding to the individual 3D shape data generated using the general-purpose 3D object model in the general-purpose illumination base model generation step in the second illumination base calculation step It may be obtained using the same parameters as the 3D shape parameters given to the general 3D object shape model.
- the image processing method is generated by setting a random three-dimensional shape parameter to be given to the general-purpose three-dimensional object model, and giving the set parameter group to the general-purpose three-dimensional object model.
- a third illumination basis calculation step for obtaining an illumination basis group corresponding to each parameter based on the sample 3D shape group to be obtained, the obtained illumination basis group, the basis vectors and eigenvalues of the general-purpose 3D object model, And a basis calculation step for obtaining a basis vector of the general illumination basis model.
- the image processing method may obtain the individual illumination base using a spherical harmonic function based on the individual 3D shape data in the third illumination basis calculation step.
- the third illumination basis calculation step a plurality of different illumination conditions are set, the brightness values of the respective parts of the object surface under each set illumination condition are obtained, and the obtained brightness values are obtained.
- the individual illumination basis group may be obtained by principal component analysis.
- the three-dimensional shape estimation method according to the present invention is a three-dimensional shape estimation method that uses an image processing method to estimate the three-dimensional shape of an object.
- the three-dimensional shape estimation method 3 General purpose cubic based on dimensional shape parameters and texture parameters It includes a shape output step for outputting a 3D shape generated by the original object model.
- the object position / orientation estimation method is an object position / orientation estimation method for estimating the position and orientation of an object using an image processing method, and when a predetermined convergence condition is satisfied, And a position / orientation output step of outputting a position / orientation parameter indicating the position and orientation of the object.
- An image generation method is an image generation method that uses an image processing method to generate a reproduced image that is an image that reproduces the same illumination conditions as an input image, and when a predetermined convergence condition is satisfied. And an image output step of outputting a reproduced image at the time of convergence.
- An image processing program uses a general-purpose three-dimensional object model obtained based on three-dimensional shape data and texture data of a plurality of similar objects on a computer, and uses a three-dimensional shape parameter and a texture parameter.
- a first illumination base calculation process for obtaining individual illumination bases each indicating a variation in luminance value of each part of the three-dimensional surface of an individual object under a plurality of different illumination conditions, and the obtained individual illumination base
- a reproduction image generation process that generates a reproduction image that reproduces the same illumination conditions as the input image, and the generated reproduction image are input. It is characterized by executing a parameter update process that repeatedly updates the 3D shape parameters and texture parameters given to the general-purpose 3D object model until a predetermined convergence condition is satisfied, so that it is close to a force image. To do.
- the image processing program uses a general-purpose 3D object model obtained based on 3D shape data and texture data of a plurality of similar objects on a computer, and uses a 3D shape parameter and a texture parameter. Based on the above, individual data generation processing for generating individual 3D shape data indicating the 3D shape of individual objects and individual texture data indicating the texture of individual objects, and general-purpose 3D object models are supported. Demonstrate basic variations in the degree of reflection due to differences in the direction and intensity of illumination applied to individual objects.
- a second illumination base calculation process for obtaining individual illumination bases each indicating a change in luminance value of each part of the surface may be executed.
- the luminance value of the reproduced image is calculated using the illumination base. Since processing is performed using a linear illumination model that shows the change in image luminance value due to illumination as a linear sum of the illumination bases, the initial value of the illumination parameter is not required, and the luminance value of the image under any lighting condition can be calculated. Can be reproduced. Therefore, it is possible to process the image without checking the initial values of the illumination condition parameters in advance or manually inputting them by the user.
- the illumination parameters that approximate the shooting environment as compared with the case where the position and orientation parameter values are input. It is difficult for a user (for example, an operator) to input the value of.
- the input image can be processed without inputting the initial value of the illumination norm, the operation burden on the user can be reduced.
- the linear illumination model by using the linear illumination model, it is not necessary to search for the illumination parameter by the probabilistic descent method. This eliminates the need to perform nonlinear optimization processing that repeats image generation for illumination parameters. Accordingly, it is possible to reduce the number of iterations for estimating the three-dimensional shape parameter and the texture parameter, to speed up the processing, and to avoid the local solution and to estimate the accurate parameter.
- the general-purpose illumination base model is used to obtain the illumination base by performing only a simple linear operation from the three-dimensional shape parameters. be able to. Therefore, by calculating the individual illumination basis using the general-purpose illumination basis model, it is possible to eliminate the generation of learning vectors necessary for normal vector calculation and individual illumination basis calculation including shadows. it can. Therefore, it is possible to speed up the calculation processing of shadows and shadows in image generation, and it is possible to perform the calculation of the illumination base necessary when using the linear illumination model at high speed.
- t 1] is a block diagram showing an example of the configuration of an image processing system according to the present invention.
- FIG. 2 is a block diagram showing an example of the configuration of illumination basis calculation means.
- FIG. 3 is a flowchart showing an example of processing in which the image processing system processes an input image and obtains each parameter.
- FIG. 4 is a block diagram showing another configuration example of the image processing system.
- FIG. 5 is a block diagram showing an example of a configuration of general-purpose illumination base model generation means.
- FIG. 6 is a flowchart showing an example of processing in which the image processing system generates a general-purpose illumination base model in advance.
- FIG. 7 is a flowchart showing another example of processing in which the image processing system processes an input image and obtains each parameter.
- FIG. 8 is a block diagram showing still another configuration example of the image processing system.
- FIG. 9 is an explanatory diagram showing a specific example of an image processing system.
- FIG. 10 is an explanatory diagram showing an example of an input image and a reproduced image generated by the CG technique.
- FIG. 11 is a block diagram illustrating a configuration example of a conventional three-dimensional shape estimation system.
- FIG. 1 is a block diagram showing an example of the configuration of an image processing system according to the present invention.
- the image processing system includes a position / orientation initial value input means 102, a face model generation means 103, an illumination basis calculation means 201, a perspective transformation means 202, an illumination correction means 203, a parameter update. Means 204.
- the image processing system is realized by an information processing apparatus such as a workstation or a personal computer.
- the image processing system is used for, for example, a 3D shape estimation system that estimates the 3D shape and texture of an object.
- the image processing system is used for an object position / orientation estimation system that estimates the position and orientation of an object, for example.
- the image processing system is used for, for example, an application of an image generation system that generates a predetermined reproduction image (for example, a CG image) that reproduces an input image.
- the position / posture initial value input means 102 is specifically realized by a CPU of an information processing device that operates according to a program and an input device such as a mouse or a keyboard.
- the position / orientation initial value input means 102 has a function of inputting an initial value of a position / orientation parameter, which is a parameter indicating the position and orientation of an object in the input image.
- a position / orientation parameter which is a parameter indicating the position and orientation of an object in the input image.
- the position of the object is a three-dimensional coordinate vector o and the direction of the object is represented by rotation angles ⁇ , ⁇ , and ⁇ around the x, y, and z axes.
- the position / orientation initial value input means 102 inputs a matrix R indicating the rotation of the rotation angles ⁇ , ⁇ , and ⁇ as a position / orientation parameter.
- an initial position / orientation value is input.
- the means 102 displays a three-dimensional image (also referred to as a CG screen) generated using CG technology along with the input image on a display device such as a display device.
- a user for example, an operator
- operates an input device such as a keyboard or a mouse while looking at the display screen to change the orientation of the object in the CG screen so that it is close to the orientation of the object in the input image.
- the initial value of the position and orientation parameter is instructed to input.
- the position / orientation initial value input means 102 inputs the initial value of the position / orientation parameter according to the input instruction of the user.
- face model generation means 103 is realized by a CPU of an information processing device that operates according to a program.
- the face model generation means 103 receives a 3D shape parameter and a texture parameter as input, and uses a general-purpose 3D face model to indicate an individual 3D shape indicating the 3D shape of an individual object (person) and an individual object. It has a function to calculate an individual texture indicating the texture of and.
- the "three-dimensional shape parameter” is a parameter indicating what deformation is applied to the standard shape data and texture data to obtain the three-dimensional shape of the object.
- the “3D shape parameter” is used to determine what kind of deformation is applied to the average shape shown in the general-purpose 3D face model to obtain the 3D shape of the person's face. It is a numerical parameter shown.
- the “3D shape parameter” is a parameter for describing the 3D shape of an individual's face using a general-purpose 3D face model.
- the "texture parameter” is a parameter indicating what kind of variation is given to the standard shape data and texture data to make the texture of the object.
- the “texture parameter” is a numerical parameter that indicates what kind of variation the average texture shown in the general-purpose 3D face model gives to the texture of the person's face. is there.
- the “texture parameter” is a parameter for describing the texture of the surface of an individual's face using a general-purpose 3D face model.
- the “general-purpose 3D face model (general-purpose 3D face model)” is a model in which the 3D shape of various human faces and the texture of the face surface are described by parameters.
- the “general-purpose 3D model” consists of basic 3D shape of human face and texture (reflectance or color) of the face surface, and data indicating the deformation of these 3D shape and texture. This embodiment Then, the “general-purpose 3D model” indicates the average value calculated from the 3D shape and texture data of multiple persons (for example, about 200 people) that have been measured in advance, and the deviation from the average value. It consists of basis vectors.
- the 3D shape parameter and the texture parameter for an individual person are known, the 3D shape parameter and the texture data are applied to the general-purpose 3D face model, so that the individual person's A face model can be obtained.
- the 3D shape parameters and texture data are applied to the general-purpose 3D face model to obtain individual human face models. Also expressed.
- the image processing system includes general-purpose face model generation means (not shown) that generates a general-purpose three-dimensional face model in advance.
- the general-purpose face model generation means generates a general-purpose three-dimensional face model in advance using the equations (1) and (2) according to the same process as the general-purpose face model generation means 900 shown in FIG.
- the image processing system includes, for example, face model storage means (not shown) that stores a general-purpose three-dimensional face model generated by a general-purpose face model generation unit.
- the general-purpose face model generation means is specifically realized by a CPU of an information processing device that operates according to a program.
- the face model storage means is specifically realized by a database device such as a magnetic disk device or an optical disk device.
- the face model generation unit 103 uses a general-purpose 3D face model stored in advance by the face model storage unit, and uses an individual person's face model (specifically, an individual 3D shape). And individual texture). In this case, the face model generation unit 103 calculates an individual three-dimensional shape and an individual texture using Expression (3) according to the same processing as the face model generation unit 903 shown in FIG.
- Illumination base calculation means 201 is specifically realized by a CPU of an information processing device that operates according to a program.
- the illumination basis calculation means 201 receives the individual 3D shape and the individual texture obtained by the face model generation means 103 as input, and describes the variation of the luminance value of each position (polygon) on the surface of the individual object under various illumination conditions. It has a function to calculate the individual illumination base.
- the “illumination base” is data that describes the basic variation in the degree of reflection (luminance value) due to the difference in the direction and intensity of illumination applied to the face.
- the illumination basis calculation means 201 calculates an individual 3D shape data force normal vector and further uses the individual texture to determine the illumination base. Calculate analytically.
- the calculation method of the above illumination base is, for example, “Ronen Basri, David W. Jacobs, Lambertian Reflectance and Linear bubspaces, IEEE Trans. PAMI, 2003, Vol. 25, No. 2, pp.218-233 ( Reference A) ”.
- the illumination basis calculation means 201 uses the illumination basis calculation technique described in these documents A and B for an image processing system (for example, a three-dimensional shape estimation system).
- the individual illumination base is calculated from the individual 3D shape data and individual texture generated by giving the 3D shape parameters and texture parameters to the general-purpose face model.
- FIG. 2 is a block diagram showing an example of the configuration of the illumination base calculation means 201. As shown in FIG. As shown in FIG. 2, the illumination basis calculation means 201 includes a normal vector calculation means 2011 and a harmonic function calculation means 2012.
- the normal vector calculation means 2011 has a function for obtaining a normal vector based on the individual three-dimensional shape obtained by the face model generation means 103.
- the normal vector calculation means 2011 calculates a normal vector n for each polygon i having an individual three-dimensional shape using equation (11).
- the harmonic function calculation means 2012 has a function of obtaining an individual illumination base using a predetermined spherical harmonic function based on an individual three-dimensional shape and an individual texture.
- the harmonic function calculation means 2012 uses a predetermined spherical harmonic function based on the normal vector obtained by the normal vector calculation means 2011 and the individual texture obtained by the face model generation means 103. Find the individual illumination basis.
- the polygon i component of the jth individual illumination base in the j-th dimension is b
- the texture brightness is t (i) .
- the harmonic function calculation means 2012 can calculate the illumination base using equation (12).
- the harmonic function calculation means 2012 may obtain the illumination base by reducing the number of dimensions for high-speed calculation.
- the harmonic function calculation means 2012 can obtain the illumination base by increasing the number of dimensions in order to improve accuracy.
- the illumination base calculation unit 201 first sets various illumination conditions when calculating the individual illumination base. Also, the illumination base calculation means 201 obtains the luminance value of each part of the object surface under each set illumination condition. Then, the illumination basis calculation means 201 obtains an individual illumination basis group based on the obtained luminance value.
- the perspective conversion means 202 is realized by a CPU of an information processing apparatus that operates according to a program.
- the perspective transformation means 202 uses the three-dimensional coordinates of the three shape data, the position Z posture of the face, and the parameters of the camera (input image capturing means) to determine the position on the three-dimensional surface and the coordinates on the image.
- the function of determining the correspondence relationship is provided.
- the perspective conversion means 202 obtains a correspondence table indicating which pixel in the reproduced image each data point of the three-dimensional shape can be seen.
- the “reproduced image” is an image reproduced by a computer by a calculation process using the estimated 3D face shape, texture, lighting conditions, and position and orientation so as to be the same as the input image.
- the “perspective conversion unit 202” further has a function of rearranging the elements of the illumination base vector b.
- Illumination correction means 203 is specifically realized by a CPU of an information processing apparatus that operates according to a program.
- the illumination correction means 203 has a function of correcting the coefficient of the linear sum so that it is the same as the shadow of the input image when calculating the luminance value of the face surface in the image by the linear sum of the illumination bases.
- the illumination correction unit 203 includes an illumination basis calculation unit 20. Using the individual illumination basis group obtained in 1 above, it has a function to generate a reproduction image that is an image that reproduces the same illumination conditions as the input image.
- the illumination correction means 203 receives the individual illumination basis group ⁇ b ⁇ and the input image I as input, and calculates the luminance value of the reproduced image I by the equation (13).
- ⁇ is a coefficient indicating illumination
- the illumination correction unit 203 uniquely obtains the coefficient ⁇ straightforward by the linear least square method without requiring an iterative search. be able to.
- the meter updating means 204 is realized by a CPU of an information processing device that operates according to a program.
- the parameter update unit 204 has a function of updating each parameter (three-dimensional shape parameter, texture parameter, and position / orientation parameter) based on the input image and the reproduced image generated by the illumination correction unit 203.
- the parameter updating unit 204 compares the reproduced image I with the input image I to obtain a reproduced image.
- the position and orientation, 3D shape, and texture parameters are changed so that the image is close to the input image. Then, the image processing system repeats the processing executed by each unit after the face model generation unit 103 until a predetermined convergence condition is satisfied, and searches for an optimal parameter value.
- the meter updating means 204 calculates a cost function E related to the similarity of images using equation (7) according to the same processing as the parameter updating means 906 shown in FIG. In addition
- the “cost function” is a function indicating the cost of error between the reproduced image and the input image.
- the “cost function” indicates how much the estimated three-dimensional face shape, texture, illumination condition, and position / posture match with the input image.
- the parameter updating means 204 searches for a parameter that minimizes the cost function E by the stochastic descent method. Therefore, the parameter updating means 204 calculates the differential value of E for each parameter of the three-dimensional shape, the texture, and the position / orientation using the equation (14).
- the meter updating unit 204 determines whether or not the update amount of each of the obtained parameters (three-dimensional shape parameter, texture meter, and position / orientation parameter) is larger than a predetermined threshold value. If the calculated parameter update amount is smaller than a predetermined threshold, the parameter update unit 204 determines that the iterative process has converged and ends the process. Further, for example, the meter updating means 204 has a smaller error between the reproduced image generated using the updated parameter and the input image than the error between the reproduced image generated using the parameter before the updating and the input image. If not, it is determined that the iterative process has converged, and the process ends.
- the image processing system When the image processing system is applied to a 3D shape estimation system, the image processing system is closest to the input image in the process of iterative processing! When an image can be generated (when processing is converged) ) Individual 3D shape and texture are output as 3D shape estimation results. In this case, when determining that the iterative process has converged, the meter updating unit 204 outputs the updated three-dimensional shape parameter and texture parameter as a three-dimensional shape estimation result.
- the image processing system When the image processing system is applied to the object position and orientation estimation system, the image processing system The position Z posture when the image was generated is output as the position and posture estimation result. In this case, when the parameter updating unit 204 determines that the iterative process has converged, the updated position / orientation parameter is output as a position / orientation estimation result.
- the image processing system When the image processing system is applied to an image generation system, the image processing system is closest to the input image in the process of iterative processing! ⁇ Outputs a reproducible image when an image can be generated. In this case, when the meter updating unit 204 determines that the repeated processing has converged, the illumination correcting unit 203 outputs the generated reproduced image.
- an information processing apparatus that implements an image processing system is an output device that outputs a 3D shape estimation result, a position / orientation estimation result, and a reproduced image to a storage medium such as a CD-ROM (operates according to a program). Realized by CPU and input / output unit of information processing equipment Provided).
- An information processing apparatus that implements an image processing system may display a 3D shape estimation result, a position / orientation estimation result, and a reproduced image on a display device such as a display device, for example.
- the storage device (not shown) of the information processing apparatus that implements the image processing system stores various programs for processing the input image.
- a storage device of an information processing device uses a general-purpose 3D object model obtained based on 3D shape data and texture data of a plurality of similar objects to a computer, and uses 3D shape parameters and texture parameters.
- individual data processing that generates individual 3D shape data that indicates the 3D shape of individual objects and individual texture data that indicates the texture of individual objects, and the generated individual 3D shape data and individual texture data
- the first illumination basis calculation processing for obtaining individual illumination bases each indicating the variation of the luminance value of each part of the three-dimensional surface of an individual object under various illumination conditions, and using the obtained individual illumination basis group
- the reproduction image generation process that generates a reproduction image that reproduces the same illumination conditions as the input image, and the generated reproduction image is close to the input image.
- an image processing program for executing a parameter update process for repeatedly updating a 3D shape parameter and a texture parameter given to a general-purpose 3D face model until a predetermined convergence condition is satisfied is stored. .
- FIG. 3 is a flowchart showing an example of processing in which the image processing system processes the input image and obtains each parameter.
- the image processing system includes imaging means such as a camera, for example. Then, for example, when an input image is captured using an imaging unit in accordance with a user's imaging operation, the image processing system starts processing the input image and obtains the value of each parameter.
- the position / posture initial value input means 102 of the image processing system inputs initial values of the position / posture parameters ⁇ p, y, ⁇ , ⁇ in accordance with a user input instruction (step SI 1).
- the face model generation means 103 uses the current face value of the three-dimensional shape parameter ⁇ and the texture parameter ⁇ t ⁇ , and uses the general-purpose face model shown in Equation (3), An individual texture is calculated (step S12).
- the face model generation means 103 when performing the first process of the repetition process, uses the value 0 (the initial value of each parameter ⁇ s ⁇ and ⁇ t ⁇ Zero) to obtain individual 3D shape and individual texture.
- the illumination base calculation unit 201 uses the individual three-dimensional shape and the individual texture obtained by the face model generation unit 103 to obtain a normal vector using equation (11). Further, the illumination basis calculation means 201 calculates the individual illumination basis group using the equation (12) (step S13).
- the perspective transformation means 202 uses the individual 3D shape obtained by the face model generation means 103 and the current position / orientation parameter, based on the relationship shown in Expression (4), and A correspondence relationship with the polygon having a three-dimensional shape is obtained (step S14).
- the perspective transformation means 202 uses the initial values of the position / orientation parameters input by the position / orientation initial value input means 102 when performing the first process of the iterative process, and the reproduced image pixels and individual 3D shapes. Find the relationship with the polygon.
- the perspective conversion unit 202 obtains a vector group ⁇ b ⁇ in which the values of the individual illumination bases of the polygons corresponding to each pixel of the reproduced image are arranged, and generates them as an individual illumination base image group.
- the illumination correction means 203 includes the input image I and the individual illumination generated by the illumination basis calculation means 201.
- the meter update means 204 is a reproduction image generated by the input image I and the illumination correction means 203.
- the parameter updating unit 204 updates each parameter of the three-dimensional shape, texture, and position / orientation using the stochastic descent method so as to minimize the cost function (step S16).
- the parameter update means 204 determines whether or not the parameter search by the probabilistic descent method has converged (step S17). If the difference between the new updated parameter calculated in step S16 and the original parameter before the update is smaller than a predetermined threshold, the parameter update unit 204 determines that the iterative process has converged and ends the process. If the cost function E is not smaller than the value before the parameter update, the parameter update unit 204 determines that the iterative process has converged and ends the process. If it does not converge, the process returns to step S12, and the image processing system repeats the processes in and after step S12 using the updated parameter ⁇ *.
- the image processing system determines each estimation result.
- the updated parameters (3D shape parameters, texture parameters, position and orientation parameters) and reproduced images are output (step S18).
- the image processing system outputs each updated parameter and reproduced image to a storage medium such as a CD-ROM. Further, for example, the image processing system displays the updated parameters and reproduced images on the display device.
- the image processing system includes the illumination base calculation unit 201 and the illumination correction unit 203, and calculates the luminance value of the reproduced image using the illumination base. Since the brightness value of the reproduced image can be calculated without using the illumination parameter, the user does not need to input the initial value of the illumination parameter. Therefore, the image can be processed without checking the initial values of the illumination condition parameters in advance or manually inputting them by the user.
- FIG. 4 is a block diagram showing another configuration example of the image processing system.
- the image processing system includes a general-purpose illumination base model generation unit 300, a position / orientation initial value input unit 102, a face model generation unit 103, and an illumination base model calculation unit 301. And a perspective conversion means 202, an illumination correction means 203, and a meter update means 204.
- the present embodiment includes an illumination base model generation means 300 in addition to the components shown in the first embodiment of the image processing system capability. Further, the present embodiment is different from the first embodiment in that the image processing system includes an illumination base model calculation unit 301 instead of the illumination basis calculation unit 201.
- the general-purpose illumination base model generation means 300 is realized by a CPU of an information processing device that operates according to a program.
- the illumination base model generation means 300 is a preliminary step before generating each parameter and reproduction image using the image processing system. Therefore, it has a function to generate a general-purpose illumination base model using a general-purpose 3D face model.
- the "illumination base model” is a model that describes individual lighting bases of individuals using predetermined parameters.
- the “illumination basis model” includes an average individual illumination basis and data indicating how the individual illumination basis of each individual's face varies depending on the shape of the face from the average individual illumination basis. Including.
- the general-purpose illumination base model corresponds to the general-purpose three-dimensional object shape model, and indicates the basic variation in the degree of reflection due to differences in the direction and intensity of illumination applied to individual objects. Used to generate
- the general-purpose illumination base model generation means 300 generates a general-purpose illumination base model in advance using the general-purpose three-dimensional model stored by the face model storage means included in the image processing system.
- the image processing system includes an illumination base model storage unit (not shown) that stores the general illumination base model generated by the general illumination base model generation unit 300.
- the illumination basis model storage means is specifically realized by a database device such as a magnetic disk device or an optical disk device.
- FIG. 5 is a block diagram showing an example of the configuration of the general-purpose illumination base model generation means 300. As shown in FIG.
- the illumination base model generation means 300 includes sample parameter setting means 3001, illumination base calculation means 3002, and base model calculation means 3003.
- the sample parameter setting means 3001 has a function of setting a random three-dimensional shape parameter to be given to the general-purpose three-dimensional object shape model.
- the sample parameter setting means 3001 sets various shape parameters for the general-purpose 3D face model, and generates a set of many sample 3D shapes and sample textures.
- the sample parameter setting means 3001 can determine these parameters using various determination methods.
- the sample parameter setting unit 300 1 assumes that the three-dimensional shape parameter ⁇ s ⁇ is a random number according to a Gaussian distribution with an average of zero and a variance of eigenvalues ⁇ and ⁇ . decide. Also sample ti
- n is determined, and n samples of 3D shape data are generated using equation (4).
- the illumination basis calculation means 3002 corresponds to each parameter based on the sample 3D shape group generated by applying the parameter group set by the sample parameter setting means 3001 to the general-purpose 3D object model. A function for obtaining the illumination base group is provided.
- the illumination basis calculation means 3002 obtains an individual illumination base using a predetermined spherical harmonic function based on the individual three-dimensional shape.
- the illumination basis calculation unit 3002 calculates the sample illumination basis group by using t (i) as 1 in equation (12) without using texture data. This is different from the harmonic function calculation means 2012 shown in the embodiment. That is, the illumination basis calculation means 3002 has only a component that changes depending on the three-dimensional shape among the brightness variations of the image.
- the illumination base calculation means 3002 first sets various illumination conditions when calculating the individual illumination base. Also, the illumination base calculation means 3002 obtains the luminance value of each part of the object surface under each set illumination condition. Then, the illumination basis calculation means 3002 obtains an individual illumination basis group based on the obtained luminance value.
- the basis model calculation means 3003 has a function for obtaining the basis vector of the general illumination basis model using the illumination basis group obtained by the illumination basis calculation means 3002, the basis vector and the eigenvalue of the general purpose 3D object model. .
- the basis model calculation unit 3003 calculates a basis vector that represents the sample illumination basis group ⁇ b ⁇ generated by the basis model calculation unit 3003 by a linear model.
- B be the vector in which the elements of the illumination basis group ⁇ b ⁇ of sample 3D shape data k are arranged, and let B be its average vector.
- a predetermined matrix B is defined by equation (15).
- Basis model calculation means 3003 uses a predetermined matrix U, ⁇ (for general-purpose 3D face model calculation). Use the value of matrix U, ⁇ ) shown in equation (2) to use and calculate matrix C using equation (16)
- the basis model calculation means 3003 stores the column vectors of the matrix C in the illumination basis model storage means as basis vector groups ⁇ c ⁇ of the general-purpose illumination basis model.
- the illumination basis model calculation means 301 has the individual illumination basis group ⁇ b of the object whose 3D shape parameter is ⁇ s ⁇ and the individual texture card T ⁇ .
- (k) indicates the value of the k-th element.
- Illumination base model calculation means 301 is specifically realized by a CPU of an information processing device that operates according to a program.
- the illumination base model calculation means 301 receives the current three-dimensional shape parameter ⁇ s ⁇ and the individual texture ⁇ T ⁇ obtained by the face model generation means 103 as inputs, and the general-purpose illumination base model generation means 300 generates the general-purpose A function to calculate the illumination base using the illumination basis model is provided.
- the illumination basis model calculation means 301 obtains an individual illumination basis group using Equation (17).
- the illumination basis model calculation means 301 calculates the equation (17) that does not require the calculation of an individual three-dimensional shape force normal vector.
- the individual illumination basis group ⁇ b ⁇ can be obtained only by the simple linear operation used.
- the illumination base model calculation means 301 uses a general-purpose 3D object corresponding to the individual 3D shape generated by the general-purpose illumination base model generation means 300 using the general-purpose 3D object shape model. Use the same parameters as the 3D shape parameters given to the shape model.
- the functions of the position / posture initial value input means 102, the face model generation means 103, the perspective conversion means 104, the illumination correction means 203, and the parameter update means 204 are the same as those shown in the first embodiment. It is.
- FIG. 6 is a flowchart showing an example of processing in which the image processing system generates a general-purpose illumination base model in advance.
- the general-purpose illumination base model generation means 300 determines an appropriate number of sets of random three-dimensional shape parameters for the general-purpose three-dimensional face model (step S21). In addition, the general-purpose illumination base model generation means 300 calculates an individual 3D shape from each set of the determined 3D shape parameters using a general-purpose 3D face model, and obtains a normal vector of the obtained individual 3D shape. Calculate Further, the general-purpose illumination basis model generation means 300 calculates an illumination basis vector group on the assumption that all the luminances of the texture are 1 (step S22).
- the general-purpose illumination base model generation means 300 calculates a base vector group of the general-purpose illumination base model for the obtained illumination base vector group using the equation (16) (step S23). Further, the general-purpose illumination base model generation means 300 stores the obtained base solid group of the general-purpose illumination base model in the illumination base model storage means.
- a general-purpose illumination base model is generated in advance and stored in the illumination base model storage unit.
- the image processing system performs the following processing after performing the illumination base model generation processing described above.
- FIG. 7 is a flowchart showing another example of processing in which the image processing system processes an input image and obtains each parameter.
- the image processing system starts processing the input image and obtains the value of each parameter when the input image is captured using the imaging means, for example, according to the user's photographing operation.
- the position / orientation initial value input means 102 of the image processing system inputs initial values of the position / orientation parameters ⁇ , ⁇ , ⁇ , ⁇ from the input image in accordance with a user input instruction (step S31).
- the face model generation means 103 uses the current three-dimensional shape parameter ⁇ s ⁇ and the value of the texture parameter ⁇ t ⁇ and uses the general-purpose face model shown in Equation (3) to generate the individual three-dimensional shape. as well as An individual texture is calculated (step S32). In this case, the face model generation unit 103 uses the value 0 (zero) as the initial value of each parameter ⁇ s ⁇ and ⁇ t ⁇ when performing the first process of the iterative process, Find the texture.
- the illumination basis model calculation means 301 is based on the three-dimensional shape parameter ⁇ s ⁇ obtained by the face model generation means 103 and the general illumination basis model accumulated by the illumination basis model accumulation means. ) To calculate the individual illumination basis group (step S33).
- the perspective conversion means 202 uses the individual 3D shape obtained by the face model generation means 103 and the current position / orientation parameter, based on the relationship shown in Equation (4), and A correspondence relationship with the polygon having a three-dimensional shape is obtained (step S34).
- the perspective transformation means 202 uses the initial values of the position / orientation parameters input by the position / orientation initial value input means 102 when performing the first process of the iterative process, and the reproduced image pixels and individual 3D shapes. Find the relationship with the polygon. Further, the perspective conversion unit 202 obtains a vector group ⁇ b ⁇ in which the values of the individual illumination bases of the polygons corresponding to each pixel of the reproduced image are arranged, and generates them as an individual illumination base image group.
- the illumination correction means 203 is generated by the input image I and the illumination base model calculation means 301.
- the luminance value I of the reproduced image is calculated using Equation (13) (
- the meter update means 204 is a reproduction image generated by the input image I and the illumination correction means 203.
- the parameter updating unit 204 updates each parameter of the three-dimensional shape, texture, and position / orientation using the stochastic descent method so as to minimize the cost function (step S36).
- the meter updating means 204 determines whether or not the parameter search by the stochastic descent method has converged (step S37).
- the parameter updating means 204 determines that the iterative process has converged when the difference between the new updated parameter calculated in step S36 and the original parameter before the update is smaller than a predetermined threshold, and ends the process. To do. If the cost function E is not smaller than the value before the parameter update, the parameter update unit 204 determines that the iterative process has converged and ends the process. If it does not converge, the process returns to step S32, and the image processing system repeats the processes after step S32 using the updated parameter *. Do it back.
- the image processing system When it is determined that the iterative processing has converged, the image processing system outputs each updated parameter (three-dimensional shape parameter, texture parameter, position / orientation parameter) and reproduced image as each estimation result. (Step S38). For example, the image processing system outputs each updated parameter and reproduced image to a storage medium such as a CD-ROM. Further, for example, the image processing system displays the updated parameters and reproduced images on the display device.
- each updated parameter three-dimensional shape parameter, texture parameter, position / orientation parameter
- the image processing system includes the general-purpose illumination base model generation unit 300, and calculates the general-purpose illumination base model in advance.
- An illumination base can be obtained by performing only a simple linear operation from a three-dimensional shape parameter using a general illumination basis model.
- 3D shape data can be processed at high speed without the need to calculate normal vectors. Therefore, in addition to the effects shown in the first embodiment, it is possible to speed up the calculation processing of shadows and shadows in image generation, and to calculate the lighting base required when using the linear lighting model at high speed. .
- FIG. 8 is a block diagram showing still another configuration example of the image processing system.
- the image processing system includes a face model generation unit 103, a general-purpose illumination base model generation unit 300, an illumination among the components shown in the second embodiment.
- the image processing system receives 3D shape parameters and texture parameters of a person's face, and outputs individual 3D shapes and individual illumination base groups.
- general-purpose illumination base model generation means 300 generates a general-purpose illumination base model corresponding to the general-purpose 3D face model in advance according to the same processing as in the second embodiment. Further, according to the same processing as in the second embodiment, the face model generation unit 103 receives the 3D shape parameter and texture parameter of the individual face to be processed, and uses a general-purpose 3D face model, Individual 3D shape and texture are calculated. In addition, the illumination base model calculation means 301 follows the same process as in the second embodiment, The general illumination basis model is input and the individual illumination basis group is calculated.
- the image processing system outputs output means (in accordance with a program) that outputs the individual three-dimensional shape generated by the face model generation means 103 and the individual illumination basis group generated by the illumination basis model calculation means 301. It is implemented by the CPU and input / output unit of the information processing device that operates.
- the individual 3D shape and the individual illumination base are calculated based on the 3D shape parameter and the texture parameter. Therefore, a 3D face model (a set of 3D shape data and illumination base group) that can generate face images in various positions, orientations, and Z lighting conditions can be calculated at high speed.
- the individual 3D shape and the individual illumination basis group are required for processing in the face matching system described in Document B, for example.
- the image processing system has a function as a 3D face model generation system that generates and provides 3D shape data of an individual's face and an illumination base at high speed with each parameter force.
- the face matching system described in Document B individual 3D shape data and individual illumination bases measured by various persons are stored in a database, and each image is input to the input image. A reproduction image is generated using human data. Then, the person when the reproduction image closest to the input image can be generated is determined as the person.
- the image processing system shown in the present embodiment can be used in place of the database of the face matching system described in Document B. That is, in the face matching system, the 3D shape parameters and texture parameters of each person are stored in the database instead of the individual 3D shape data and the individual illumination base. Then, by using the image processing system shown in the present embodiment, if the individual 3D shape data and the individual illumination base of each individual are calculated, the face matching system described in Document B can be operated.
- FIG. 9 is an explanatory diagram showing a specific example of the image processing system.
- the image processing system includes an image input device 401 that captures and inputs an input image, a computer 402 that executes various processes, a storage device 403 that stores various data, an estimation And a storage medium 404 for recording and outputting the result data.
- the computer 402 includes the position / orientation initial value input means 102, face model generation means 103, illumination base calculation means 201, perspective transformation means 202, illumination correction means 203, and parameter update means 204 shown in the first embodiment. Is provided.
- the storage device 403 stores a general-purpose 3D face model in advance.
- the storage device 403 stores 100 shape basis vectors, 100 texture basis vectors, and eigenvalues corresponding to the shape basis vectors and the texture basis vectors, respectively, as general-purpose 3D face model data.
- the image processing system may improve estimation accuracy by using more than 100 bases.
- the image processing system can also improve the processing speed by reducing the base number from 100.
- the image input device 401 captures a human face image in accordance with a user input instruction, and outputs an input image 411 to the computer 402. Then, the computer 402 inputs the input image 411 from the image input device 401.
- the computer 402 displays an input image 411 in which one light hits the face at an angle of 60 degrees to the right and 80 degrees to the right, and the brightness intensity of the entire room is about half that of the light. Suppose you enter it. Of course, at the stage when the input image 411 is input, the exact orientation of the face and the arrangement of the illumination are unknown to the image processing system.
- the position / orientation initial value input means 102 of the computer 402 inputs initial values of the position / orientation parameters ⁇ , ⁇ , ⁇ , ⁇ of the input image 411 in accordance with an input instruction from the user of the image processing system.
- the position and orientation initial value input means 102 is displayed on the screen of the display device of the computer 402.
- an image generated by CG technology using the average 3D face shape and average texture is displayed.
- the position / orientation initial value input means 102 displays the input image 411 and the CG image 412 side by side on the display screen of the display device.
- the user changes the values of the position and orientation parameters ⁇ p, ⁇ , ⁇ , ⁇ while comparing the CG image 412 with the input image 411, and manually inputs a value close to the input image 411.
- the position / orientation initial value input means 102 inputs initial values of the position / orientation parameters ⁇ , ,, ⁇ , ⁇ in accordance with the input instruction of the user.
- the face model generation means 103 uses the general-purpose 3D face model shown in Equation (3) using the current 3D shape parameter ⁇ s ⁇ and the value of the texture parameter ⁇ t ⁇ . Calculate individual 3D shapes and textures.
- the face model generation means 103 reads the general-purpose 3D face model stored in advance in the storage device 403 from the storage device 403 and uses it to calculate the individual 3D shape and the individual texture. Further, the face model generation unit 103 obtains an individual three-dimensional shape and an individual texture using the value 0 (zero) as the initial value of each parameter ⁇ s ⁇ and ⁇ t ⁇ when performing the first process of the iterative process. .
- the illumination base calculation means 201 uses the individual 3D shape and the individual texture obtained by the face model generation means 103 to obtain a normal vector using equation (11). Further, the illumination basis calculation means 201 calculates the individual illumination basis group using Expression (12). In the present embodiment, the illumination base calculation means 201 obtains an individual illumination base group with nine illumination bases. Note that the number of illumination bases obtained by the illumination base calculation means 201 can be changed to various values, and the number of illumination bases to be obtained may be reduced in order to improve the processing speed. In addition, in order to improve the parameter estimation accuracy, the number of illumination bases to be obtained can be increased.
- the perspective conversion means 202 uses the individual 3D shape and the position / orientation parameter to obtain the correspondence between the pixel of the reproduced image and the polygon of the individual 3D shape based on the relationship shown in Equation (4). . Further, the perspective conversion means 202 generates a vector group ⁇ b ⁇ in which individual illumination base values of polygons corresponding to each pixel of the reproduced image are arranged as an individual illumination base image group. As an example, if the number of pixels of the input image corresponding to the individual 3D shape is 100, the perspective change The conversion unit 202 generates a vector group ⁇ b ⁇ including 100-dimensional vectors.
- the illumination correction means 203 includes the input image I and the individual illumination generated by the illumination basis calculation means 201.
- the luminance value I of the reproduced image is calculated using Equation (13).
- the meter update means 204 is a reproduction image generated by the input image I and the illumination correction means 203.
- the cost function ⁇ is calculated using the image I and the eigenvalues ⁇ and ⁇ of the general-purpose face model.
- si ti The cost function ⁇ is calculated using the image I and the eigenvalues ⁇ and ⁇ of the general-purpose face model.
- the parameter updating unit 204 updates each parameter of the three-dimensional shape, texture, and position / orientation using the stochastic descent method so as to minimize the cost function.
- the parameter updating means 204 determines whether or not the parameter search by the probabilistic descent method has converged. When the difference between the calculated new updated parameter and the original parameter before update is smaller than a predetermined threshold value, the meter update unit 204 determines that the iterative process has converged and ends the process. If the cost function E is not smaller than the value before the parameter update, the parameter update unit 204 determines that the iterative process has converged and ends the process. As an example, the meter updating means 204 uses, for example, the threshold “lcm” of the element P and the thresholds “1 °” of the face angles ⁇ , 0, and ⁇ as the threshold for the parameter update amount. Then, it is determined whether or not the parameter search has converged.
- the parameter updating means 204 determines that the parameter search has converged when the amount of change is smaller than these threshold values. If the image does not converge, the image processing system uses the updated parameters to execute each of the face model generation unit 103, the illumination base calculation unit 201, the perspective conversion unit 202, the illumination correction unit 203, and the parameter update unit 204. Repeat the process.
- the image processing system stores the individual 3D face shape and the individual texture generated by the face model generation unit 103 from the 3D shape parameters and the texture parameters at the time of convergence. Record and output.
- the image processing system may generate a CG image using the individual three-dimensional shape and the individual texture, and output the generated CG image to the storage medium 404.
- the image processing system may output the position and posture parameters when converged! /.
- image processing A description will be given of a case where the physical system is applied to an object position / orientation estimation system that estimates the position and orientation of a person's face in the face image based on the face image. Note that the configuration of the image processing system shown in the present embodiment corresponds to the configuration shown in the second embodiment of the present invention (that is, the configuration shown in FIG. 4).
- the computer 402 includes the general-purpose illumination base model generation means 300, the position / orientation initial value input means 102, the face model generation means 103, and the illumination base model calculation means shown in the second embodiment. 301, perspective conversion means 202, illumination correction means 203, and parameter update means 204.
- the storage device 403 stores a general-purpose three-dimensional face model in advance.
- the force using only the basis vectors and eigenvalues of the general-purpose face model.
- the matrix US that is, the matrix shown in Equation (2)
- the value of US is also stored in the storage means 403.
- the general-purpose illumination base model generation means 300 of the computer 402 performs in advance processing for calculating a general-purpose illumination base model as a preparation. .
- the sample parameter setting means 3001 determines an appropriate number of sets of random 3D shape parameters for the general-purpose 3D face model.
- the sample parameter setting means 3001 sets 100 sets of parameters at random. Note that the number of parameters can be variously changed.
- the illumination basis calculation means 3002 calculates the individual 3D shape from each set of 3D shape parameters determined by the sample parameter setting means 3001 using a general-purpose 3D face model, and obtains the obtained individual 3D Calculate the normal vector of the shape. Furthermore, the illumination basis calculation means 3002 calculates the illumination basis vectors, assuming that the texture brightness is all 1.
- the basis vector calculation means 3003 calculates the basis vector group of the general-purpose illumination basis model using the equation (16) for the illumination basis beta group obtained by the illumination basis calculation means 3002. Further, the basis model calculation means 3003 stores the basis vector group of the obtained general-purpose illumination basis model in the storage device 403. In this example, the number of 3D shape parameters is 100. Therefore, the basis model calculation means 3003 calculates a basis vector group including 100 basis vectors.
- the image processing system performs the following processing after performing the processing described above in advance.
- the position / orientation initial value input means 102 of the computer 402 inputs initial values of the position / orientation parameters ⁇ p, y, ⁇ , ⁇ from the input image in accordance with a user input instruction.
- the user manually inputs while viewing the CG image.
- the positions of both eyes and the edge of the mouth with respect to the input image 411 Explain the case where the user gives an input instruction while looking at the screen.
- the corresponding point position is also given to the average three-dimensional shape data in advance (for example, the position information of the corresponding point is stored in advance in the storage device 403).
- the position / orientation initial value input unit 102 calculates the face / orientation using the three-dimensional coordinates stored in advance and the position data of the points input according to the user's operation.
- the face model generation means 103 uses the values of the current shape parameter ⁇ s ⁇ and the texture parameter ⁇ t ⁇ and uses the general-purpose face model shown in Equation (3) to generate an individual 3D shape. And calculate the individual texture. In this case, the face model generation unit 103 obtains an individual three-dimensional shape and an individual texture by using the value 0 (zero) as an initial value of each parameter ⁇ s ⁇ and ⁇ t ⁇ in the first process of the iterative process. .
- the illumination base model calculation means 301 uses equation (17) based on the three-dimensional shape parameter ⁇ s ⁇ obtained by the face model generation means 103 and the general-purpose illumination base model stored in the storage device 403. To calculate an individual illumination basis group.
- the perspective conversion means 202 uses the individual 3D shape obtained by the face model generation means 103 and the current position / orientation parameter, based on the relationship shown in Equation (4), and Find the correspondence with a 3D polygon. Further, the perspective conversion means 202 obtains a vector group ⁇ b ⁇ in which the values of the individual illumination bases of the polygons corresponding to each pixel of the reproduced image are arranged, and generates them as an individual illumination base image group.
- the illumination correction means 203 is generated by the input image I and the illumination base model calculation means 301.
- the luminance value I of the reproduced image is calculated using Equation (13).
- the parameter update means 204 is the input image I and the regenerated image generated by the illumination correction means 203.
- the cost function ⁇ is calculated using the current image I and the eigenvalues ⁇ and ⁇ of the general-purpose face model model si ti
- the parameter updating unit 204 updates each parameter of the three-dimensional shape, texture, and position / orientation using the stochastic descent method so as to minimize the cost function.
- the parameter updating means 204 determines whether or not the parameter search by the probabilistic descent method has converged.
- the meter updating means 204 determines that the iterative process has converged when the difference between the new updated parameter calculated in step S36 shown in FIG. 7 and the original parameter before updating is smaller than a predetermined threshold value. The process ends. If the cost function E is not smaller than the value before the parameter update, the parameter update means 204 determines that the iterative process has converged and ends the process. If not converged, the processes executed by the face model generation unit 103, the illumination base model calculation unit 301, the perspective transformation unit 202, the illumination correction unit 203, and the parameter update unit 204 are repeatedly performed using the updated parameters.
- the image processing system If it is determined that the iterative process has converged, the image processing system outputs the position / orientation parameter value at the time of convergence as an estimated value of the position / orientation of the face in the input image. As an example, the image processing system records and outputs data (position and orientation parameters) indicating the position and orientation on the storage medium 404, for example.
- the program code to be used and the recording medium for recording the program code are within the scope of the present invention.
- the program code in this case includes those program codes when the above functions are realized in cooperation with the operating system and other application software.
- a recording medium for example, a disc type (optical disk, magneto-optical disk, magnetic disk, etc.), a tape type (magnetic tape, etc.), a card type (memory card, etc.), etc. can be used. .
- the present invention can be applied to the use of a three-dimensional shape estimation system that estimates a three-dimensional shape of a human face from one or a plurality of face images.
- the present invention can also be applied to the use of an object position / posture estimation system that estimates the orientation of a person's face by outputting estimated posture parameters.
- the present invention can be applied to the use of an image generation system that generates a reproduction image (for example, a CG image) that reproduces an input image.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Geometry (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Computer Graphics (AREA)
- Image Generation (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
Claims
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP06747117A EP1887511A1 (en) | 2005-06-03 | 2006-06-02 | Image processing system, 3-dimensional shape estimation system, object position posture estimation system, and image generation system |
US11/921,315 US8320660B2 (en) | 2005-06-03 | 2006-06-02 | Image processing system, 3-dimensional shape estimation system, object position/posture estimation system and image generation system |
JP2007519086A JP4793698B2 (ja) | 2005-06-03 | 2006-06-02 | 画像処理システム、3次元形状推定システム、物体位置姿勢推定システム及び画像生成システム |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2005164475 | 2005-06-03 | ||
JP2005-164475 | 2005-06-03 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2006129791A1 true WO2006129791A1 (ja) | 2006-12-07 |
Family
ID=37481718
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2006/311087 WO2006129791A1 (ja) | 2005-06-03 | 2006-06-02 | 画像処理システム、3次元形状推定システム、物体位置姿勢推定システム及び画像生成システム |
Country Status (5)
Country | Link |
---|---|
US (1) | US8320660B2 (ja) |
EP (1) | EP1887511A1 (ja) |
JP (1) | JP4793698B2 (ja) |
CN (1) | CN100492409C (ja) |
WO (1) | WO2006129791A1 (ja) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2014502741A (ja) * | 2010-12-09 | 2014-02-03 | サムスン エレクトロニクス カンパニー リミテッド | 映像処理装置、照明処理装置及びその方法 |
Families Citing this family (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8494286B2 (en) | 2008-02-05 | 2013-07-23 | DigitalOptics Corporation Europe Limited | Face detection in mid-shot digital images |
US7440593B1 (en) | 2003-06-26 | 2008-10-21 | Fotonation Vision Limited | Method of improving orientation and color balance of digital images using face detection information |
US7574016B2 (en) | 2003-06-26 | 2009-08-11 | Fotonation Vision Limited | Digital image processing using face detection information |
US7844076B2 (en) | 2003-06-26 | 2010-11-30 | Fotonation Vision Limited | Digital image processing using face detection and skin tone information |
US7792970B2 (en) | 2005-06-17 | 2010-09-07 | Fotonation Vision Limited | Method for establishing a paired connection between media devices |
US7269292B2 (en) | 2003-06-26 | 2007-09-11 | Fotonation Vision Limited | Digital image adjustable compression and resolution using face detection information |
US8498452B2 (en) | 2003-06-26 | 2013-07-30 | DigitalOptics Corporation Europe Limited | Digital image processing using face detection information |
US7471846B2 (en) | 2003-06-26 | 2008-12-30 | Fotonation Vision Limited | Perfecting the effect of flash within an image acquisition devices using face detection |
US8989453B2 (en) | 2003-06-26 | 2015-03-24 | Fotonation Limited | Digital image processing using face detection information |
US8948468B2 (en) | 2003-06-26 | 2015-02-03 | Fotonation Limited | Modification of viewing parameters for digital images using face detection information |
US7565030B2 (en) | 2003-06-26 | 2009-07-21 | Fotonation Vision Limited | Detecting orientation of digital images using face detection information |
US8503800B2 (en) | 2007-03-05 | 2013-08-06 | DigitalOptics Corporation Europe Limited | Illumination detection using classifier chains |
DE602007012246D1 (de) | 2006-06-12 | 2011-03-10 | Tessera Tech Ireland Ltd | Fortschritte bei der erweiterung der aam-techniken aus grauskalen- zu farbbildern |
US8055067B2 (en) | 2007-01-18 | 2011-11-08 | DigitalOptics Corporation Europe Limited | Color segmentation |
DE602008001607D1 (de) | 2007-02-28 | 2010-08-05 | Fotonation Vision Ltd | Trennung der direktionalen beleuchtungsvariabilität in der statistischen gesichtsmodellierung auf basis von texturraumzerlegungen |
WO2008107002A1 (en) | 2007-03-05 | 2008-09-12 | Fotonation Vision Limited | Face searching and detection in a digital image acquisition device |
US7916971B2 (en) | 2007-05-24 | 2011-03-29 | Tessera Technologies Ireland Limited | Image processing method and apparatus |
US7855737B2 (en) | 2008-03-26 | 2010-12-21 | Fotonation Ireland Limited | Method of making a digital camera image of a scene including the camera user |
WO2010012448A2 (en) | 2008-07-30 | 2010-02-04 | Fotonation Ireland Limited | Automatic face and skin beautification using face detection |
CN101499132B (zh) * | 2009-03-12 | 2013-05-01 | 广东药学院 | 一种人脸图像中特征点提取的三维变换搜索方法 |
US8379917B2 (en) | 2009-10-02 | 2013-02-19 | DigitalOptics Corporation Europe Limited | Face recognition performance using additional image features |
EP2590135A1 (en) * | 2010-06-30 | 2013-05-08 | Nec Corporation | Color image processing method, color image processing device, and color image processing program |
CN102680012B (zh) * | 2011-03-16 | 2015-07-08 | 鸿富锦精密工业(深圳)有限公司 | 量测信息自动输出系统及方法 |
KR101608253B1 (ko) * | 2011-08-09 | 2016-04-01 | 인텔 코포레이션 | 이미지 기반 멀티 뷰 3d 얼굴 생성 |
CN103092576A (zh) * | 2011-10-28 | 2013-05-08 | 鸿富锦精密工业(深圳)有限公司 | 量测程序输出系统及方法 |
US9123144B2 (en) * | 2011-11-11 | 2015-09-01 | Microsoft Technology Licensing, Llc | Computing 3D shape parameters for face animation |
JP5894426B2 (ja) * | 2011-12-13 | 2016-03-30 | パナソニック インテレクチュアル プロパティ コーポレーション オブアメリカPanasonic Intellectual Property Corporation of America | 計測対象抽出装置、顔形状推定装置、計測対象抽出方法および顔形状推定方法 |
CN103735268B (zh) * | 2013-09-29 | 2015-11-25 | 沈阳东软医疗系统有限公司 | 一种体位检测方法及系统 |
US9953111B2 (en) * | 2014-06-06 | 2018-04-24 | Matterport, Inc. | Semantic understanding of 3D data |
CN105426872B (zh) * | 2015-12-17 | 2019-06-21 | 电子科技大学 | 一种基于相关高斯过程回归的面部年龄估计方法 |
US11157985B2 (en) * | 2017-11-29 | 2021-10-26 | Ditto Technologies, Inc. | Recommendation system, method and computer program product based on a user's physical features |
US11521460B2 (en) | 2018-07-25 | 2022-12-06 | Konami Gaming, Inc. | Casino management system with a patron facial recognition system and methods of operating same |
US10878657B2 (en) | 2018-07-25 | 2020-12-29 | Konami Gaming, Inc. | Casino management system with a patron facial recognition system and methods of operating same |
CN110763150B (zh) * | 2018-07-26 | 2021-05-25 | 王东 | 一种面部轮廓测量方法 |
US10834413B2 (en) * | 2018-08-24 | 2020-11-10 | Disney Enterprises, Inc. | Fast and accurate block matching for computer generated content |
US11475608B2 (en) * | 2019-09-26 | 2022-10-18 | Apple Inc. | Face image generation with pose and expression control |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004054947A (ja) * | 2002-07-16 | 2004-02-19 | Nec Corp | 物体照合システム、物体照合方法および物体照合プログラム |
JP2004086929A (ja) * | 2003-12-08 | 2004-03-18 | Nec Corp | 画像照合装置 |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1994023390A1 (en) * | 1993-03-29 | 1994-10-13 | Matsushita Electric Industrial Co., Ltd. | Apparatus for identifying person |
JP3514945B2 (ja) * | 1997-05-26 | 2004-04-05 | 株式会社ソニー・コンピュータエンタテインメント | 画像作成方法および画像作成装置 |
US6002782A (en) * | 1997-11-12 | 1999-12-14 | Unisys Corporation | System and method for recognizing a 3-D object by generating a 2-D image of the object from a transformed 3-D model |
US6956569B1 (en) * | 2000-03-30 | 2005-10-18 | Nec Corporation | Method for matching a two dimensional image to one of a plurality of three dimensional candidate models contained in a database |
JP2001283216A (ja) * | 2000-04-03 | 2001-10-12 | Nec Corp | 画像照合装置、画像照合方法、及びそのプログラムを記録した記録媒体 |
JP4167390B2 (ja) * | 2000-11-20 | 2008-10-15 | 日本電気株式会社 | 物体照合方法,物体照合装置,およびそのプログラムを記録した記録媒体 |
US7212664B2 (en) * | 2003-08-07 | 2007-05-01 | Mitsubishi Electric Research Laboratories, Inc. | Constructing heads from 3D models and 2D silhouettes |
CN102298778B (zh) | 2003-10-30 | 2015-06-17 | 日本电气株式会社 | 用于估计对象状态的估计系统、估计方法和估计程序 |
-
2006
- 2006-06-02 EP EP06747117A patent/EP1887511A1/en not_active Withdrawn
- 2006-06-02 WO PCT/JP2006/311087 patent/WO2006129791A1/ja active Application Filing
- 2006-06-02 US US11/921,315 patent/US8320660B2/en active Active
- 2006-06-02 JP JP2007519086A patent/JP4793698B2/ja active Active
- 2006-06-02 CN CNB200680019642XA patent/CN100492409C/zh active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004054947A (ja) * | 2002-07-16 | 2004-02-19 | Nec Corp | 物体照合システム、物体照合方法および物体照合プログラム |
JP2004086929A (ja) * | 2003-12-08 | 2004-03-18 | Nec Corp | 画像照合装置 |
Non-Patent Citations (7)
Title |
---|
HAMANAKA M. ET AL.: "3 Jigen Deta o Key to shita Shisei. Shomei Hendo ni Gankyo na Kao Gazo Kensaku", IEICE TECHNICAL REPORT PRMU2004-59, vol. 104, no. 290, 3 September 2004 (2004-09-03), pages 41 - 48, XP003006122 * |
HAMANAKA M. ET AL.: "Nin'i Shisei ni Taio shita Hyojun Sokuchi Omomi o Mochiita Kao Shogo", IEICE TECHNICAL REPORT PRMU2003-105, vol. 103, no. 296, 2 September 2003 (2003-09-02), pages 61 - 66, XP002996559 * |
ISHIYAMA R. ET AL.: "3D Buttai Hyomenjo deno Mie Model Kochiku ni yoru Shisei to Shomei no Hendo ni Gankyu na Kao Shogo", INFORMATION PROCESSING SOCIETY OF JAPAN KENKYU HOKOKU 2003-CVIM-139, vol. 2003, no. 66, 3 July 2003 (2003-07-03), pages 61 - 68, XP003006121 * |
MANSON WOO; JACKIE NEIDER; TOM DAVIS: "Open GL Programming Guide", ADDISON-WESLEY PUBLISHERS JAPAN LTD, pages: 169 - 195 |
RONEN BASRI; DAVID W. JACOBS: "Lambertian Reflectance and Linear Subspaces", IEEE TRANS. PAMI, vol. 25, no. 2, 2003, pages 218 - 233 |
RUI ISHIYAMA; SHIZUO SAKAMOTO: "Geodesic Illumination Basis: Compensating for Illumination Variations in any Pose for Face Recognition", PROC. OF ICPR, vol. 4, 2002, pages 297 - 301 |
VOLKER BLANZ; THOMAS VETTER: "Face Recognition Based on Fitting a 3-dimensional Morphable Model", PAMI, vol. 25, no. 9, 2003, pages 1063 - 1074 |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2014502741A (ja) * | 2010-12-09 | 2014-02-03 | サムスン エレクトロニクス カンパニー リミテッド | 映像処理装置、照明処理装置及びその方法 |
Also Published As
Publication number | Publication date |
---|---|
EP1887511A1 (en) | 2008-02-13 |
JPWO2006129791A1 (ja) | 2009-01-08 |
US20090129665A1 (en) | 2009-05-21 |
CN100492409C (zh) | 2009-05-27 |
US8320660B2 (en) | 2012-11-27 |
CN101189637A (zh) | 2008-05-28 |
JP4793698B2 (ja) | 2011-10-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP4793698B2 (ja) | 画像処理システム、3次元形状推定システム、物体位置姿勢推定システム及び画像生成システム | |
Aldrian et al. | Inverse rendering of faces with a 3D morphable model | |
Lensch et al. | Image-based reconstruction of spatially varying materials | |
US8406556B2 (en) | Light transport reconstruction from sparsely captured images | |
US6956569B1 (en) | Method for matching a two dimensional image to one of a plurality of three dimensional candidate models contained in a database | |
US8009880B2 (en) | Recovering parameters from a sub-optimal image | |
Qin et al. | Aura 3D textures | |
US20070269111A1 (en) | Shape detection using coherent appearance modeling | |
JPWO2010026983A1 (ja) | 画像処理装置、画像処理方法、及び画像処理プログラム | |
US20200334862A1 (en) | Moving image generation apparatus, moving image generation method, and non-transitory recording medium | |
Pintus et al. | Objective and subjective evaluation of virtual relighting from reflectance transformation imaging data | |
Guo et al. | Brdf analysis with directional statistics and its applications | |
Kumar et al. | Non-Lambertian reflectance modeling and shape recovery of faces using tensor splines | |
Kartashova et al. | Light shapes: Perception-based visualizations of the global light transport | |
JP4849460B2 (ja) | 画像中の顔を認識する方法 | |
Rohe | An Optical Test Simulator Based on the Open-Source Blender Software. | |
CN116486018A (zh) | 三维重建方法、设备和存储介质 | |
JP2004054947A (ja) | 物体照合システム、物体照合方法および物体照合プログラム | |
Ohayon et al. | The perception-robustness tradeoff in deterministic image restoration | |
Ma et al. | A lighting robust fitting approach of 3D morphable model for face reconstruction | |
JP3910811B2 (ja) | テクスチャマッピング方法、テクスチャマッピング処理プログラム及びそのプログラムを記録したコンピュータ読取り可能な記録媒体 | |
Ngan | Acquisition and modeling of material appearance | |
Shim | Probabilistic approach to realistic face synthesis with a single uncalibrated image | |
Ma et al. | A lighting robust fitting approach of 3D morphable model using spherical harmonic illumination | |
CN111651623B (zh) | 高精度人脸表情库的构建方法、装置、设备及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 200680019642.X Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWE | Wipo information: entry into national phase |
Ref document number: 2007519086 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 11921315 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2006747117 Country of ref document: EP |
|
WWP | Wipo information: published in national office |
Ref document number: 2006747117 Country of ref document: EP |