JP2002032783A - Method and device for generating three-dimensional model - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily generate a three-dimensional model from a two-dimensional image without using three-dimensional data. SOLUTION: In this method for generating the three-dimensional model by deforming a three-dimensional standard model DS, the standard model DS is projected onto a two-dimensional picture FT obtained by photographing an object with the camera by using a camera parameter when the object is photographed, the model DS is fitted to the picture FT, also rough alignment of the standard model and the picture FT is performed at that time, a scale of a z direction is obtained on the basis of the scale information of x and y directions obtained in rough alignment, and the standard model in a depth direction is deformed on the basis of the obtained scale of the z direction.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a method and apparatus for generating a three-dimensional model, and is used, for example, for generating a three-dimensional model in the field of computer graphics.

[0002]

2. Description of the Related Art In recent years, three-dimensional CG (three-dimensional computer graphics) technology is often used for movies, games, and the like. In three-dimensional CG, virtual 3
Since a three-dimensional model or a light is arranged and moved in a three-dimensional space, the degree of freedom of expression is high.

Conventionally, a non-contact type three-dimensional measuring device using a light-section method or the like has been put into practical use, and three-dimensional data of an object can be created by performing measurement using the device. However, if the three-dimensional data obtained by the measurement is used as it is for the three-dimensional CG, there are various problems, such as complicated processing for reducing the data amount by thinning the obtained data.

To cope with this problem, there has been proposed a method in which a standard model of an object is prepared, and the standard model is deformed in accordance with the measured three-dimensional data (Japanese Patent Laid-Open No. 5-81377).

In this conventional method, three-dimensional shape information of three-dimensional data obtained by measurement, that is, a three-dimensional point group is used as a fitting target, and the surface of a standard model is fitted to the three-dimensional point group. Let it.

[0006]

However, obtaining three-dimensional data on an object is not as easy as obtaining a two-dimensional image. Further, since the amount of three-dimensional data is large, there is also a problem that data processing takes time, and the above-described data reduction processing is troublesome.

Further, when the surface of the standard model is deformed in accordance with the measured three-dimensional point group, the overall shape almost matches, but it is difficult to match local features. there were.

That is, for example, when a three-dimensional model of a human head is created by a conventional method, the completed 3D model is obtained.
In the dimensional model, the overall shape of the head matches the head of the person to be measured.However, for local parts that greatly affect the facial expression, such as the eyes or mouth, there is sufficient It does not lead to a match. Therefore, it has been difficult to express a person's detailed expression.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and has as its object to provide a method and apparatus for easily generating a three-dimensional model from a two-dimensional image without using three-dimensional data.

[0010]

The method according to the present invention comprises three steps.
A method for generating a three-dimensional model by deforming a three-dimensional standard model, comprising projecting the standard model onto a two-dimensional image obtained by photographing an object using a camera at the time of photographing. Fitting the standard model to the two-dimensional image, and at that time, roughly aligning the standard model with the two-dimensional image and obtaining scale information in the x and y directions obtained in the rough alignment , A scale in the z direction is obtained based on the obtained scale, and the standard model is deformed in the depth direction based on the obtained scale in the z direction.

Preferably, a contour or / and /
And feature points, and fitting is performed such that the contours and / or feature points approach corresponding contours and / or feature points on the two-dimensional image.

An apparatus according to the present invention stores a two-dimensional image obtained by photographing an object with a camera, and projects the standard model onto the two-dimensional image using camera parameters at the time of photographing. Means for obtaining a scale in the z-direction based on scale information in the x-direction and the y-direction obtained in the rough alignment when performing the rough alignment between the standard model and the two-dimensional image; means for deforming the standard model in the depth direction based on the scale in the z direction.

In the present invention, the term “fitting” refers to a process of transforming a standard model or a series of processes including the process.

[0014]

FIG. 1 is a block diagram showing a modeling device 1 according to the present invention. In the present embodiment,
An example of generating a three-dimensional model of a human head by deforming (fitting) a standard model created in advance based on measurement data (two-dimensional image) on the human head will be described.

As shown in FIG. 1, a modeling device 1
It comprises a processing device 10, a magnetic disk device 11, a medium drive device 12, a display device 13, a keyboard 14, a mouse 15, a camera 16, and the like.

The processing device 10 includes a CPU, a RAM, an RO,
M, a video RAM, an input / output port, and various controllers. Various functions are realized on the processing device 10 by the CPU executing programs stored in the RAM, the ROM, and the like.

The magnetic disk drive 11 has an OS (Operat
ing System), a modeling program PR for generating the three-dimensional model ML, other programs, a standard model (standard model data) DS, three-dimensional data (three-dimensional measurement data) DT, and reliability of the three-dimensional data DT. Reliability data DR, two-dimensional image (two-dimensional measurement data) FT,
The generated three-dimensional model ML, other data, and the like are stored. These programs and data are loaded into the RAM of the processing device 10 as appropriate. However, in the present embodiment, the three-dimensional data DT and the reliability data DR are not used.

The modeling program PR includes measurement processing, rough alignment, data reduction processing, deformation processing,
A program for partial area selection processing and other processing is included.

The medium drive device 12 is a CD-ROM
(CD), a floppy disk FD, or a recording medium such as a magneto-optical disk is accessed to read and write data or programs. An appropriate drive device is used depending on the type of the recording medium. The above-described modeling program PR can be installed from these recording media. Standard model DS, three-dimensional data DT, reliability data DR, and two-dimensional image F
T and the like can also be input via a recording medium.

On the display surface HG of the display device 13,
The various data described above, the three-dimensional model ML generated by the modeling program PR, and other data (images) are displayed.

The keyboard 14 and the mouse 15 are used for inputting data or giving commands to the processing device 10. The camera 16 is a digital camera,
This is for capturing a target object and acquiring a two-dimensional image FT. By photographing the object from a plurality of different directions, a plurality of two-dimensional images FT having different parallaxes can be obtained. As such a camera 16, a known digital camera can be used. Alternatively, a photograph may be taken using a silver halide camera, and the obtained photographic image may be digitized. There is also a method of using a plurality of cameras arranged with parallax with respect to an object. It is also possible to use a two-dimensional image FT taken from one direction.

The modeling device 1 can be configured using a personal computer or a workstation. The programs and data described above can be obtained by receiving them via the network NW.

Next, the overall processing flow of the modeling device 1 will be described with reference to flowcharts.
FIG. 2 is a flowchart showing the overall processing flow of the modeling device 1, FIG. 3 is a flowchart showing a deformation process, FIG. 4 is a diagram showing an example of a standard model DS1, FIG. FIG. 6 is a diagram showing a state of correspondence of contours;
FIG. 7 is a view schematically showing a state of alignment, FIG. 8 is a view schematically showing a contour RK of the standard model DS and a point P of the two-dimensional image FT, and FIG. It is a figure for explaining virtual spring of. [Preparation of Standard Model] In FIG. 2, first, a standard model DS for an object is prepared (# 11). In the present embodiment, since the object is a human head, it is most similar to the head of the object from among a plurality of standard model groups having various sizes and shapes, all around the head. Prepare a standard model DS1.

The standard model DS may be either a three-dimensional shape model defined by a polygon or a three-dimensional shape model defined by a free-form surface. 3 defined by polygon
In the case of a three-dimensional model, the surface shape is determined by the three-dimensional coordinates of the vertices of each polygon. In the case of a three-dimensional shape model defined by a free-form surface, the shape of the surface is determined by the function defining the surface and the coordinates of each control point.

When the model is a three-dimensional shape model defined by polygons, the vertices of each polygon are described as "constituent points". Further, at the time of fitting the standard model DS, a point used for deforming the standard model is referred to as a “control point”. The positional relationship between the control points and the constituent points of the polygon is arbitrary, and the control points may be set on the surface of the polygon or may be set away from the surface of the polygon. One control point is composed of multiple points (about 3 to 100)
And the associated constituent point moves in accordance with the movement of the control point. When fitting the standard model DS, the entire standard model DS is deformed by moving these control points. Even when the three-dimensional shape model is defined by a free-form surface, the arrangement of control points used for fitting is arbitrary.

The control points are arranged at a high density in a portion having a fine shape such as the outer corner of the eye or the end of the lip and a portion having a sharp change in shape such as the nose or lip. The other parts are uniformly arranged.

In the standard model DS, a characteristic contour RK and a characteristic point TT viewed from a certain direction are set. Contour R
As K, for example, a line of an eyelid, a line of a nose, a line of a lip, a line of a chin, or the like is set in an eye, a nose, a mouth, a chin, or the like. As the characteristic point TT, for example, there is no characteristic in itself, such as a part having an actual characteristic such as an end of an eye or a mouth, a top of a nose, or a lower end of a chin, or a middle part thereof. The part that can be easily specified is selected.

In the standard model DS1 shown in FIG. 4, the chin line, the lip line, and the eyelid line have the contours RK1 to RK3.
Is set as As can be seen in FIG.
Reference numeral 1 denotes an edge portion of the standard model DS1 when viewed from a certain direction. In the standard model DS1 shown in FIG. 4, only a part of the set feature points TT is actually shown in the figure. [Acquisition of Two-Dimensional Image] Next, an object is photographed from a plurality of directions to acquire a plurality of two-dimensional images FT (# 12). At this time, the camera parameters of the camera 16 are obtained at the same time. The camera parameters include, for example, the size and shape of the CCD, the attitude with respect to the object, the focal length of the lens, and the angle of view. The position and the angle of the two-dimensional image FT with respect to the object are obtained from the camera parameters.

Incidentally, the two-dimensional image FT may be described as "measurement data". Either preparation of the standard model DS and acquisition of the two-dimensional image FT may be performed first or may be performed in parallel. [Extraction of contours / feature points] As shown in FIGS. 5 and 6,
The contour RK and the feature point TT to be arranged at the same positions as the contour RK and the feature point TT preset in the standard model DS are arranged on the two-dimensional image FT (# 13). [Schematic alignment] Standard model DS and two-dimensional image FT
(# 14). In this process, as shown in FIG. 7, a partial line or point on the two-dimensional image FT is associated with a partial line or point on the standard model DS so that their distance is minimized. Change the orientation, size, and position of the standard model DS.

That is, the feature point TT on the two-dimensional image FT
And the distance between the corresponding feature point TT on the standard model DS and the corresponding energy point E (s) shown in the following equation (1).
i, αi, ti) of the standard model DS are derived such that i, αi, ti) is minimized.

[0031]

(Equation 1)

Where k: number of corresponding feature points Mk: feature points on the standard model after deformation x: feature points on the standard model before deformation Ck: feature points on the two-dimensional image Si: each of the standard models Αi: the amount of rotation of the standard model in each direction ti: the amount of movement of the standard model in each direction Further, the photographing by the camera 16 is limited to one direction. In such a case, the magnification (scale) in the depth direction (Z direction) may not be accurately obtained. In this case, it is assumed that there is no significant difference between the shape of the target object and the shape of the standard model DS, and as shown in the following equation (2), the magnification in the X and Y directions (S1, S2) is used. A method for correcting the magnification (S3) of the above can be used.

[0033]

(Equation 2)

Here, γ is a weight parameter for the line of sight x3 deformation. By performing such correction, the standard model DS is deformed into a three-dimensional image by using only one two-dimensional image FT photographed from one direction. The model ML can be generated, and acquisition and calculation of the two-dimensional image FT are easy. [Modification] The standard model DS is modified (# 15).
Here, the energy function e2 defined in relation to the distance between the contour of the standard model DS and the contour set in the two-dimensional image FT, the feature points of the standard model DS and the two-dimensional image F
Using an energy function e3 defined in relation to the distance between the feature points set to T and an energy function es defined to avoid excessive deformation, an energy function e obtained by combining them is expressed as Evaluate and deform the surface of the standard model DS so that the total energy function e is minimized.

Note that the total energy function e is e
Although it is desirable to use three functions of 2, e3, and es,
It is also possible to use only two of e2 and e3.
Next, each energy function will be sequentially described. [Distance between contour on standard model and contour on two-dimensional image]
Here, the distance between the contour RK specified on the two-dimensional image FT and the contour RK on the standard model DS is evaluated.

That is, the contour RK of the standard model DS is projected onto the two-dimensional image FT as shown in FIG. 6 using the camera parameters of the camera 16 that has captured the two-dimensional image FT. Then, as shown in FIG. 8, a perpendicular is dropped from the point Pk on the contour RK of the two-dimensional image FT to the corresponding point Qk on the contour RK on the standard model DS, and the shortest of the perpendiculars is set as the distance. . Note that a plurality of points are specified on the contour RK.

The difference energy e2 'for each contour RK of the standard model DS is calculated by the following equation (3) using the sum of squares of the distances.

[0038]

(Equation 3)

T2A: Control point group pk: Contour point on two-dimensional image qk: Drop foot point from contour point on two-dimensional image to corresponding model contour n: Correspondence to one model contour The number of contour points of the existing two-dimensional image d ^k : the direction of projection from the contour points on the two-dimensional image to the corresponding model contour d ^k = (qk-pk) / | qk-pk | l: various energies in the same unit Adjustment Scale for Handling It is possible to perform three-dimensional deformation of the standard model DS using the contour RK of the two-dimensional image FT photographed from a plurality of directions. However, it is necessary that the line-of-sight directions of each contour RK match each other.

That is, for example, a plurality of two-dimensional images FT are obtained by photographing an object from a plurality of different directions. At this time, as described above, the camera parameters are obtained.

The extraction of the contour RK can be performed automatically by image recognition, but there is also a method of performing the extraction manually.
Then, the standard model DS is projected on the two-dimensional image FT and deformed. Deformation is not performed on a plurality of two-dimensional images FT one by one.
Are simultaneously projected on the standard model DS, and the calculations are performed simultaneously so that their distances become shorter.

Normally, the projection of the standard model DS onto the two-dimensional image FT is performed on the memory, and the calculation is performed by the CPU. As a result of the calculation, the contour R of the standard model DS
When K moves, portions other than the contour RK move accordingly. [Distance between feature point on standard model and corresponding feature point on two-dimensional image] By setting feature point TT on a two-dimensional image instead of contour RK described above, two-dimensional image FT
The distance between the feature point TT set above and the feature point on the standard model DS is evaluated.

That is, the feature point TT on the two-dimensional image FT
Energy e between the feature point TT and the standard model DS
3 is calculated by the following equation (4) using the square distance of the corresponding feature point TT.

[0044]

(Equation 4)

T3A: control point group Fk: feature point on two-dimensional image Gk: feature point on standard model corresponding to feature point on two-dimensional image N: feature point on two-dimensional image and feature point on standard model L: Adjustment scale for handling various energies in the same unit [Stabilizing energy for avoiding excessive deformation] In addition to the difference energy described above, avoid excessive deformation A stabilizing energy es is introduced to perform

That is, it is assumed that control points used for deformation are connected by a virtual spring (elastic bar) KB shown in FIG. Based on the constraint of the virtual spring KB, a stabilizing energy es for stabilizing the shape of the surface S of the standard model DS is defined.

It should be noted that the virtual spring does not necessarily need to be set between control points. It is sufficient that the relationship between the control point and the virtual spring is clear. FIG. 9 shows a part of the surface S of the standard model DS to be fitted. Surface S
Is formed by a control point group U = | ui, i = 1... N |. A virtual spring KB is arranged between adjacent control points. The virtual spring KB gives a constraint between the control points by a tensile force, and functions to prevent abnormal deformation of the surface S.

That is, when the interval between adjacent control points u increases, the pulling force by the virtual spring KB increases accordingly. For example, when the point Qk approaches the point Pk and the movement of the point Qk increases the interval between the control points u, the tensile force of the virtual spring KB increases. Point Q
If the interval between the control points u does not change even if k moves, that is, if the relative positional relationship between the control points u does not change, the pulling force by the virtual spring KB does not change. A value obtained by averaging the tensile force by the virtual spring KB over the entire surface S is defined as stabilizing energy es. Therefore, when a part of the surface S protrudes and deforms, the stabilization energy es increases. If the entire surface S moves on average, the stabilization energy es is zero.

The stabilizing energy es is obtained by the following equation (5) according to the state of deformation of the virtual spring KB.

[0050]

(Equation 5)

TsA: Control point group U-m, V-m: Initial value of end point (control point) of virtual spring Um, Vm: End point of virtual spring after deformation L0m: Length of virtual spring in initial state , L0m = | U-m-V-m | M: Number of virtual springs c: Spring coefficient L: Adjustment scale for handling various energies in the same unit Therefore, if spring coefficient c is increased, virtual spring KB
Are hard and difficult to deform.

By introducing such a stabilizing energy function es, a certain constraint is imposed on the shape change of the surface S, and excessive deformation of the surface S can be prevented. [Total energy function] As described above, for each energy function e2, e3, e4, the control point group T2
A, T3A and TsA are used. Here, these control point groups T2A to TsA are the same, but can be different from each other as described later. Using the control point group TA, the standard model DS is deformed to obtain a control point group TA that minimizes the total energy function e (TA) shown in the following equation (6).

[0053]

(Equation 6)

Here, e2 ^s (T2A): difference energy between the contour on the two-dimensional image for each model contour e3 (T3A): difference energy eS (TSA) between the feature point of the two-dimensional image and the feature point on the model : Stabilizing energy wi, c for avoiding excessive deformation. Weight parameters of respective energies. TA = T1A = T2A = T3A = TSA. [Repeated deformation] Actually, repeated deformation is performed. In other words, the control point is moved and the deformation is repeated. Assuming that the total energy function after the n-th deformation is en (TA), when the condition of the following equation (7) is satisfied, it is determined that the total energy function en (TA) has converged.

[0055]

(Equation 7)

Now, the overall flow of the deformation process will be described with reference to FIG. First, a set of corresponding points is created between the two-dimensional image and the standard model DS (Pk in FIG. 8).
And Qk) (# 21).

The surface S is deformed (# 22), and the total energy function en (TA) after the deformation is calculated (# 23). The process is repeated until the total energy function en (TA) converges (Yes in # 24).

As a method of judging the convergence of the total energy function en (TA), as described above, a method of making the convergence when the total energy function en (TA) becomes smaller than a predetermined value, It is possible to use a known method such as a method in which convergence is achieved when the rate of change compared with the above becomes a predetermined value or less.

According to the above-described embodiment, since fitting is performed using the contour RK or feature point TT of the two-dimensional image FT, a three-dimensional model can be easily generated from the two-dimensional image without using three-dimensional data. can do.

In the above-described embodiment, the configuration, circuit, processing contents, processing order, processing timing, coefficient setting, and the like of the modeling device 1 can be appropriately changed in accordance with the gist of the present invention.

[0061]

According to the present invention, a three-dimensional model can be easily generated from a two-dimensional image without using three-dimensional data. Moreover, it is possible to generate a three-dimensional model using only two-dimensional images taken from one direction, and it is easy to obtain and calculate two-dimensional images.

According to the second aspect of the present invention, it is possible to make the local parts such as the eyes and the mouth better coincide.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a modeling device according to the present invention.

FIG. 2 is a flowchart showing the flow of the entire processing of the modeling device.

FIG. 3 is a flowchart showing a transformation process.

FIG. 4 is a diagram showing an example of a standard model.

FIG. 5 is a diagram showing a state of correspondence of feature points.

FIG. 6 is a diagram showing a state of correspondence of contours.

FIG. 7 is a diagram showing a schematic state of alignment.

FIG. 8 is a diagram schematically illustrating a contour of a standard model and points of a two-dimensional image.

FIG. 9 is a diagram for explaining a virtual spring for preventing abnormal deformation of the standard model.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Modeling apparatus (3D model generation apparatus) 10 Processing apparatus 16 Camera FT 2D image ML 3D model DS Standard model RK Contour TT Feature point PR Modeling program

 ──────────────────────────────────────────────────続 き Continuation of the front page (72) Inventor Hideo Fujii 2-3-13-1 Azuchicho, Chuo-ku, Osaka-shi, Osaka Inside the Osaka International Building Minolta Co., Ltd. (72) Inventor Tomashi Kovalchik 5-19 Hiroo, Shibuya-ku, Tokyo -9 Hiroo ON Building Gen-Tech Co., Ltd. F-term (reference) 5B046 FA18 5B050 BA09 BA12 EA05 EA06 EA13 EA28

Claims (3)

[Claims] 1. A method for generating a three-dimensional model by deforming a three-dimensional standard model, comprising: using a camera parameter at the time of shooting on a two-dimensional image obtained by shooting an object with a camera. Projecting the standard model, fitting the standard model to the two-dimensional image, and at that time, roughly aligning the standard model with the two-dimensional image, and obtaining the x direction obtained in the rough alignment And acquiring a scale in the z-direction based on the scale information in the y-direction, and deforming the standard model in the depth direction based on the acquired scale in the z-direction. 2. An outline or / and feature point is defined on the standard model, and the outline or / and feature point is defined by 2
The method for generating a three-dimensional model according to claim 1, wherein the fitting is performed so as to approach a corresponding contour or / and a feature point on the three-dimensional image. 3. An apparatus for generating a three-dimensional model by deforming a three-dimensional standard model, comprising: means for storing a two-dimensional image obtained by photographing an object with a camera; Means for projecting the standard model on the two-dimensional image by using, when roughly aligning the standard model and the two-dimensional image, in the x and y directions obtained in the rough alignment Means for acquiring a scale in the z-direction based on the scale information and deforming the standard model in the depth direction based on the acquired scale in the z-direction. apparatus.