JP2011008816A - Method for deforming standard model and modeling device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow better matching with measured three-dimensional data without causing local abnormal deformations of local parts such as eyes and a mouth.SOLUTION: In a method for deforming a standard model DS in accordance with measured three-dimensional data by using a processor including a CPU and a memory, the CPU executes: a step of storing the three-dimensional data and the standard model DS in the memory, and reducing the number of pieces of three-dimensional data in order to thin the three-dimensional data; a step of deforming the entire standard model DS to reduce a distance between an outline of thinned three-dimensional data and an outline of the standard DS model; a step of selecting a plurality of partial areas BRY having local features from three-dimensional data before thinning after the entire standard model DS is deformed; and a step of using three-dimensional data of the plurality of selected partial areas to deform only corresponding parts of the standard model DS.

Description

  The present invention relates to a standard model deformation method and modeling apparatus, and is used, for example, for generating a three-dimensional model in the field of computer graphics.

  In recent years, 3D CG (3D computer graphics) technology is often used for movies and games. In the three-dimensional CG, a three-dimensional model and a light are arranged and moved in a virtual three-dimensional space, so that the degree of freedom of expression is high.

  Conventionally, a non-contact type three-dimensional measuring apparatus using a light cutting method or the like has been put into practical use, and by performing measurement using this, three-dimensional data of an object can be created relatively easily. However, in order to use the three-dimensional data obtained by the measurement as it is for the three-dimensional CG, there are various problems such as complicated processing for reducing the amount of data by thinning out the obtained data.

  In order to deal with this problem, a method has been proposed in which a standard model of an object is prepared and the standard model is deformed in accordance with the measured three-dimensional data (Patent Document 1).

  In this conventional method, three-dimensional shape information of three-dimensional data obtained by measurement, that is, a point group existing in three dimensions is used as a fitting target, and the surface of the standard model is fitted to the three-dimensional point group.

JP-A-5-81377

  However, in the conventional method described above, the overall shapes are almost the same, but it is difficult to match the local features.

  That is, for example, when a three-dimensional model of a human head is created by a conventional method, the overall shape of the three-dimensional model matches the head of the person to be measured. Local parts that have a great influence on facial expressions such as eyes or mouth cannot be matched sufficiently. Therefore, it was difficult to express a person's fine facial expression.

  The present invention has been made in view of the above-described problems, and a method and apparatus that can better match the measured three-dimensional data without causing local abnormal deformation of the local area such as the eyes or mouth. The purpose is to provide.

  The method according to the present invention is a method of transforming a standard model in accordance with measured three-dimensional data using a processing device having a CPU and a memory, and storing the three-dimensional data and the standard model in the memory. In addition, in order for the CPU to thin out the three-dimensional data, the step of reducing the number of the three-dimensional data, and the distance between the thinned-out outline of the three-dimensional data and the outline of the standard model is small. Transforming the whole standard model so that it becomes, selecting a plurality of partial regions having features locally from the three-dimensional data before being thinned after transforming the whole standard model, The step of deforming only the corresponding part of the standard model using the three-dimensional data of the selected partial areas is executed.

  Further, control points for deforming the standard model are defined in the standard model, and for each partial region, the control points, a reduction rate for reducing the number of the three-dimensional data, or the standard model At least one of the evaluation functions for evaluating the degree of deformation is changed.

  In the apparatus according to the present invention, in order to thin out the three-dimensional data, a distance between the means for reducing the number of the three-dimensional data and the contour of the thinned-out three-dimensional data and the contour of the standard model is reduced. Means for deforming the whole of the standard model, and means for selecting a plurality of partial regions having features locally from the three-dimensional data before being thinned after the whole of the standard model is transformed, Means for deforming only the corresponding part of the standard model using the three-dimensional data of the plurality of partial areas.

  In the present invention, the fitting refers to a deformation process of a standard model or a series of processes including it.

  According to the present invention, it is possible to better match the measured three-dimensional data with respect to a local part such as an eye or a mouth without causing local abnormal deformation.

It is a block diagram which shows the modeling apparatus which concerns on this invention. It is a flowchart which shows the flow of the whole process of a modeling apparatus. It is a flowchart which shows a deformation | transformation process. It is a figure which shows the example of a standard model. It is a figure which shows a mode that three-dimensional data is acquired from a target object. It is a figure which shows the mode of a rough alignment. It is a figure which shows the mode of the extraction process of an outline and a feature point. It is a figure which shows typically the surface of a standard model, and the point of three-dimensional data. It is a figure for demonstrating the virtual spring for preventing the abnormal deformation | transformation of a standard model. It is a figure explaining the example of the method of acquiring the three-dimensional data and reliability data of a target object. It is a figure which shows the example of extraction of a partial area | region. It is a figure explaining the example of the method of deform | transforming a standard model using a partial area | region.

  FIG. 1 is a block diagram showing a modeling apparatus 1 according to the present invention. In the present embodiment, a standard model created in advance is deformed (fitted) based on measurement data (three-dimensional data or a two-dimensional image) about a human head, thereby obtaining a three-dimensional model of the human head. An example of generation will be described.

  As shown in FIG. 1, the modeling device 1 includes a processing device 10, a magnetic disk device 11, a medium drive device 12, a display device 13, a keyboard 14, a mouse 15, and a three-dimensional measuring device 16.

  The processing device 10 includes a CPU, a RAM, a ROM, a video RAM, an input / output port, various controllers, and the like. Various functions are realized on the processing device 10 by the CPU executing programs stored in the RAM and ROM.

  The magnetic disk device 11 includes an OS (Operating System), a modeling program PR for generating a three-dimensional model ML, other programs, a standard model (standard model data) DS, three-dimensional data (three-dimensional measurement data) DT, The reliability data DR indicating the reliability of the three-dimensional data DT, the two-dimensional image (two-dimensional measurement data) FT, the generated three-dimensional model ML, and other data are stored. These programs and data are loaded into the RAM of the processing device 10 at appropriate times.

  Note that the modeling program PR includes programs for measurement processing, rough alignment, data reduction processing, deformation processing, partial region selection processing, and other processing.

  The medium drive device 12 accesses a recording medium such as a CD-ROM (CD), a floppy disk FD, or a magneto-optical disk, and reads / writes data or programs. An appropriate drive device is used according to the type of the recording medium. The modeling program PR described above can be installed from these recording media. The standard model DS, the three-dimensional data DT, the reliability data DR, the two-dimensional image FT, and the like can also be input via the recording medium.

  Various data described above, the three-dimensional model ML generated by the modeling program PR, and other data (images) are displayed on the display surface HG of the display device 13.

  The keyboard 14 and the mouse 15 are used for inputting data or giving commands to the processing device 10. The three-dimensional measuring device 16 is for obtaining three-dimensional data DT of an object by, for example, a light cutting method. It is also possible to obtain the three-dimensional data DT directly by the three-dimensional measuring device 16, and the calculation is performed by the processing device 10 or the like based on the data output from the three-dimensional measuring device 16 and indirectly three-dimensionally. It is also possible to obtain data DT.

  Simultaneously with the three-dimensional data DT, it is also possible to acquire a two-dimensional image FT on the same line of sight for the same object as necessary. As such a three-dimensional measuring device 16, for example, a known device disclosed in JP-A-10-206132 can be used.

  As another known method for acquiring the three-dimensional data DT of the object, there is a method using a plurality of cameras arranged with parallax with respect to the object. From a plurality of images having parallax obtained from these cameras, three-dimensional data DT can be obtained by calculation using a stereoscopic photography method.

  In this method, for example, by using three cameras, data for determining the reliability of each point of the three-dimensional data DT can be acquired simultaneously. That is, an object is photographed by multi-viewing with three cameras, and three images are obtained. For these three images, the corresponding points are searched for. Based on the corresponding points for the two images, the three-dimensional data DT is obtained by a known calculation. The other image is used to obtain the reliability data DR.

  For example, as shown in FIG. 10, three cameras A, B, and C are used to photograph the object Q to obtain three images FA, FB, and FC. For each image FA, FB, FC, the respective image planes (un, vn) are shown (n = 1, 2, 3). A point QP on the object Q in the three-dimensional space M is projected onto points PA, PB, and PC on each image plane.

  Here, if the correspondence between the points PA, PB, and PC is obtained, the three-dimensional space M ′ can be reconstructed from these correspondences. In the three-dimensional space M ′, a point QP ′ corresponding to the points PA and PB is obtained. Ideally, the point PC ′ obtained by back projecting the reconstructed point QP ′ on the three-dimensional space M ′ onto the image plane (u 3, v 3) and the original point QP on the three-dimensional space M are displayed on the image plane. It should coincide with the point PC projected onto (u3, v3).

  However, since it is difficult to accurately determine the projection transformation and it is difficult to accurately determine the correspondence, they usually do not match. Therefore, the deviation between the point PC ′ and the point PC is taken as an error and used as an index of reliability.

  For example, the error between the point PC ′ and the point PC is indicated by the number of shifted pixels. If the point PC ′ and the point PC are on the same pixel, the error is “0”. If it is shifted by one pixel, the error is “1”. If it is shifted by two pixels, the error is “2”. This shifted pixel number can be used as the reliability data DR.

  As another method for determining the reliability data DR, for example, a method disclosed in Japanese Patent Application Laid-Open No. 61-125686 and other known methods can be used. The modeling apparatus 1 can be configured using a personal computer or a workstation. The programs and data described above can also be obtained by receiving via the network NW.

  Next, an overall processing flow of the modeling apparatus 1 will be described with reference to a flowchart.

2 is a flowchart showing the overall processing flow of the modeling apparatus 1, FIG. 3 is a flowchart showing the deformation processing, FIG. 4 is a diagram showing an example of the standard model DS1, and FIG. 5 is to acquire three-dimensional data DT from the object. FIGS. 6A and 6B are diagrams showing a schematic alignment, FIG. 7 is a diagram showing a contour and feature point extraction process, and FIG. 8 is a diagram showing the surface S of the standard model DS. FIG. 9 is a diagram schematically illustrating a point P of the three-dimensional data DT, FIG. 9 is a diagram for explaining a virtual spring for preventing abnormal deformation of the standard model DS, and FIG. 10 is a three-dimensional data DT and reliability of an object. It is a figure explaining the example of the method of acquiring data DR.
[Preparation of standard model]
In FIG. 2, first, a standard model DS for an object is prepared (# 11). In this embodiment, since the object is a human head, it is most similar to the head of the object from among a plurality of standard model groups with various sizes and shapes for the entire circumference of the head. A standard model DS1 is prepared.

  The standard model DS may be either a three-dimensional shape model defined by polygons or a three-dimensional shape model defined by free-form surfaces. In the case of a three-dimensional shape model defined by polygons, the surface shape is determined by the three-dimensional coordinates of the vertices of each polygon. In the case of a three-dimensional shape model defined by a free-form surface, the shape of the surface is determined by the function that defines the surface and the coordinates of each control point.

  In the case of a three-dimensional shape model defined by polygons, the vertices of each polygon are described as “component points”. In addition, when fitting the standard model DS, a point used to deform the standard model is referred to as a “control point”. The positional relationship between the control points and the constituent points of the polygon is arbitrary, and the control points may be set on the surface of the polygon or may be set away from the surface of the polygon. One control point is associated with a plurality of component points (about 3 to 100), and the associated component point moves in accordance with the movement of the control point. When fitting the standard model DS, the entire standard model DS is deformed by moving these control points. Even when the three-dimensional shape model is defined by a free-form surface, the arrangement of control points used for fitting is arbitrary.

  Control points are arranged at a high density in portions having fine shapes such as the corners of the eyes and lips, and in portions having rapid shape changes such as the nose and lips. The other portions are uniformly arranged.

  In the standard model DS, a characteristic contour RK and a characteristic point TT viewed from a certain direction are set. As the contour RK, for example, a wrinkle line, a nose line, a lip line, or a chin line is set for the eyes, nose, mouth, or jaw. As the feature point TT, for example, there is an actual characteristic part such as the end of the eye or mouth, the top of the nose, the lower end of the chin, or the middle of them, but there is no feature. A part that is easy to specify is selected.

In the standard model DS1 shown in FIG. 4, chin lines, lip lines, and heel lines are set as contours RK1 to RK3. As can be seen in FIG. 4, the contour RK1 is a portion that becomes an edge when the standard model DS1 is viewed from a certain direction. In the standard model DS1 shown in FIG. 4, only a part of the set feature points TT is actually shown in the figure.
[Acquisition of 3D data]
Next, three-dimensional measurement of the object is performed to obtain three-dimensional data DT (# 12). At that time, a two-dimensional image FT of the object is also acquired at the same time. Further, as described above, the reliability data DR indicating the reliability of each point of the three-dimensional data DT or information for obtaining the reliability data DR is acquired as necessary.

  For example, as shown in FIG. 5, a three-dimensional measuring device 16 is used to measure (capture) a human head that is an object. Thereby, the three-dimensional data DT and the two-dimensional image FT are acquired.

Note that the three-dimensional data DT and / or the two-dimensional image FT obtained by measuring the object may be referred to as “measurement data”. Either the preparation of the standard model DS or the acquisition of the three-dimensional data DT may be performed first or in parallel.
[Rough alignment] Rough alignment between the standard model DS and the three-dimensional data DT is performed (# 13). In this process, the orientation, size, and position of the standard model DS are changed so that the standard model DS and the three-dimensional data DT substantially match. At this time, the size of each standard model DS can be matched to the three-dimensional data DT by scaling the standard model DS individually to an arbitrary magnification in each of the X, Y, and Z directions.

  For example, as shown in FIG. 6A, by rotating the standard model DS and scaling in each direction with respect to the three-dimensional data DT as shown in FIG. And a standard model DSa of approximately the same size can be obtained. In addition, in order to make it intelligible, the thing which is not adjusting the position on drawing is shown.

As a method of rough alignment, as described below, there are two methods: (1) overall rough alignment and (2) local rough alignment. Among these methods, the method (1) can be automatically performed. In the method (2), it is difficult to automatically extract feature points in the method, and therefore, it is necessary to perform a part manually. In the fitting after the rough alignment, the standard model DS is basically locally deformed. Therefore, the method (1) is suitable when importance is attached to the shape, like an animation. The method (2) is preferable when the position is desired to match the shape. Further, if the user is accustomed to performing feature point extraction, it is effective to use the method (2) in order to shorten the processing time.
[Overall approximate alignment]
In the overall approximate alignment, the position, direction, and size of the standard model DS are changed so that the distance between the three-dimensional data DT and the standard model DS is minimized.

  That is, si, αi, ti that minimizes the energy function e (si, αi, ti) shown in the following equation (1) is derived. Note that f (si, αi, ti) is an energy function defined in relation to the distance between the three-dimensional data DT and the standard model DS. g (si) is a stabilization energy function for avoiding excessive deformation.

  In addition, using the two-dimensional image FT acquired at the same time when the three-dimensional measuring device 16 acquires the three-dimensional data DT, using the pattern matching on the two-dimensional image FT, the initial values of position, direction, and size are set. May be given.

However,
K: Number of constituent points of three-dimensional data dk: Distance between constituent points of three-dimensional data and standard model surface Wsc: Weight parameter for scaling stabilization S0: Initial scale Si: Amount of scaling in each direction (however, S3 is the depth direction)
αi: rotation of the standard model in each direction ti: movement amount of the standard model in each direction Here, the constituent points on the standard model DS move according to the following equation (2), and accordingly, the three-dimensional data DT The distance dk between the composing point and the surface of the standard model DS changes.

  In addition, when the measurement (photographing) by the three-dimensional measuring device 16 is limited from one direction, the amount of magnification in the depth direction (Z direction) may not be obtained accurately. In that case, it is considered that there is no significant difference between the shape of the three-dimensional data DT and the shape of the standard model DS, and as shown in the following equation (3), it depends on the amount of magnification (S1, S2) in the X and Y directions. A method of correcting the amount of magnification (S3) in the Z direction is also conceivable.

However,
γ: Line-of-sight direction x3 Weight parameter for deformation (local approximate alignment)
If the overall rough alignment described above is performed automatically and it does not fit well, it will be manually aligned. However, the local rough alignment described here is a manual alignment. It is a technique to do as easily as possible. In addition, the part which did not work automatically is reset once, and it starts it manually from the beginning.

  In the local approximate alignment, the characteristic line or point on the three-dimensional data DT and the characteristic line or point on the standard model DS are associated with each other so that the distance between them is minimized. Change position, direction, and size. When lines are associated with each other, the feature point is the point on one line and the shortest of the perpendiculars drawn from that point onto the other line, and multiple points are obtained on the line. It shall be.

  That is, the energy function E (si, αi, ti) shown in the following equation (4) is minimized with respect to the distance between the feature point on the three-dimensional data DT and the corresponding feature point on the standard model DS. Thus, ti, αi, and si of the standard model DS are derived.

However,
k: number of corresponding feature points Mk: feature points on the standard model after alignment x: feature points on the standard model before alignment Ck: feature points on the three-dimensional data Si: deviation in each direction of the standard model Double amount αi: Rotation of standard model in each direction ti: Amount of movement of standard model in each direction When the measurement (photographing) by the three-dimensional measuring device 16 is limited from one direction, the depth direction (Z direction) The scale of may not be obtained accurately. In that case, a method of correcting the scale in the Z direction using the following equation (5) is conceivable as in the case of the overall rough alignment described above.

[Extraction of contour and feature points]
Outlines and feature points are extracted on the three-dimensional data DT or the two-dimensional image FT (# 14). When the contour RK and the feature point TT for the standard model DS have been extracted in advance, the contour and the feature point to be arranged at the same position as the contour RK and the feature point TT are displayed on the three-dimensional data DT or the corresponding two-dimensional They are arranged on the image (see FIG. 7).

When the contour RK and the feature point TT are not extracted in advance for the standard model DS, they are also specified on the standard model DS together with the arrangement on the three-dimensional data DT or the two-dimensional image.
[Data reduction]
Next, in order to reduce the calculation amount and the error, data is reduced for the three-dimensional data DT, and only necessary and reliable data is extracted (# 15). By reducing the data, the calculation amount can be reduced without destroying the shape of the original three-dimensional data DT.

  In reducing the data, for example, data outside the object area is excluded and unnecessary data is excluded. For example, the face area is determined from the two-dimensional image FT, and only the three-dimensional data DT corresponding to the area is left. Alternatively, the region is determined using the difference in distance between the object and the background. In addition, there are various methods such as extracting a face region using the information on the approximate alignment. If the three-dimensional data DT includes the reliability data DR, only the highly reliable data is left. If there are many data in the neighborhood, the data is thinned out and the density is averaged.

  When the density is averaged by thinning out data, only the three-dimensional data DT that satisfies the condition expressed by the following equation (6) is employed.

However,
Pk: Composition point Pr: Already adopted composition point Rdet (x): Function representing the density around the composition point x According to the above equation (6), the data Pk of interest has been adopted so far If the distance to the remaining data Pr is greater than a certain value, the data Pk is adopted.
[Deformation]
The entire standard model DS is deformed (# 16). Here, the energy function e1 defined in relation to the distance between each constituent point of the three-dimensional data DT and the surface of the standard model DS is used, and in addition, the feature points of the standard model DS and the three-dimensional data are used. Energy function e3 defined in relation to the distance between the feature points specified for DT, related to the distance between the contour of the standard model DS and the contour specified for the three-dimensional data DT. The energy function e2 defined in order to avoid excessive deformation and the energy function es defined to avoid excessive deformation are used to evaluate the combined energy function e so that the total energy function e is minimized. The surface of the model DS is deformed.

It is most preferable to use four functions e1, e2, e3, and es as the total energy function e, but it is also possible to use only two of e1 to e3.
Next, each energy function will be described sequentially.
[Distance between standard model and 3D data]
In FIG. 8, one of the point groups constituting the three-dimensional data DT is indicated by a point Pk. On the surface S of the standard model DS, the point closest to the point Pk is indicated by Qk. The point Qk is an intersection when a perpendicular is drawn from the point Pk to the surface S. Here, the distance between the point Pk and the point Qk is evaluated.

  That is, the difference energy e1 between each point of the three-dimensional data DT and the surface of the standard model DS is a point Pk on the three-dimensional data DT after data reduction and a point Qk obtained by projecting the point Pk on the surface S of the standard model DS. Is calculated by the following equation (7).

However,
T1A: Control point group Pk: Composition point of 3D data after reduction Qk: Projection point from composition point to model surface K: Number of composition points after reduction dk: Projection direction from composition point to model surface,
dk = (Qk-Pk) / | Qk-Pk |
ρk: Reliability of the component point Pk
w (ρk): reliability function, w (ρk) = 1 / (α + ρk) ⁿ
W: Σw (ρk)
L: Scale for adjustment to handle various energies in the same unit [distance between contour on standard model and contour on measurement data]
Here, the distance between the contour RK designated on the three-dimensional data DT or the contour RK designated on the two-dimensional image FT and the contour RK on the standard model DS is evaluated.

  When the contour RK of the measurement data is specified on the three-dimensional data DT, a perpendicular is dropped from a point on the contour RK of the three-dimensional data DT to the corresponding contour RK on the standard model DS, and the shortest of the perpendiculars Is the distance. A plurality of points are designated on the contour RK.

  When the contour RK of the measurement data is designated on the two-dimensional image FT, the contour RK of the standard model DS is projected on the two-dimensional image FT using the camera parameters for the camera that captured the two-dimensional image FT. A perpendicular is dropped from a point on the contour RK of the two-dimensional image FT to the corresponding contour RK of the standard model DS, and the shortest one of the perpendiculars is taken as a distance. A plurality of points are designated on the contour RK.

  When the contour RK of the measurement data is specified on the three-dimensional data DT, the difference energy e2 for each contour RK of the standard model DS is calculated by the following equation (8) using the sum of squares of the distances. .

However,
T2A: control point group pk: contour point on 3D data qk: droop point from contour point on 3D data to corresponding model contour n: 3D data corresponding to one model contour Number of contour points d ^k : Projection direction from contour point of measurement data to corresponding model contour line,
d ^k = (qk-pk) / | qk-pk |
l: Adjustment scale for handling various energies in the same unit When the contour RK of the measurement data is specified on the two-dimensional image FT, the difference energy e2 ′ for each contour RK of the standard model DS is expressed by the following (9 ).

However,
T2A: control point group pk: contour point on 2D image qk: droop point from corresponding contour point on 2D image to corresponding model contour projected on 2D image n: corresponding to one model contour Number of contour points of measurement data attached d ^k : Projection direction from contour point on two-dimensional image to corresponding model contour,
d ^k = (qk-pk) / | qk-pk |
l: Adjustment scale for handling various energies in the same unit The reason for designating the contour RK of the measurement data on the two-dimensional image FT is, for example, when the three-dimensional data DT is ambiguous, the contour on the three-dimensional data DT This is because if the RK is specified, the contour RK itself becomes inaccurate. Therefore, the contour RK is extracted using the two-dimensional image FT instead.
[Distance between feature points on standard model and feature points on measurement data]
By setting the feature point TT on the measurement data, the feature point TT specified on the three-dimensional data DT or the feature point TT specified on the two-dimensional image FT and the feature point on the standard model DS The distance is evaluated.

The difference energy e3 between the feature point TT on the three-dimensional data DT and the feature point TT on the standard model DS is calculated by the following equation (10) using the square distance of the corresponding feature point TT.
When the feature point TT is specified on the two-dimensional image FT, the feature point TT of the standard model DS is projected on the two-dimensional image FT using the camera parameters, and the difference energy on the two-dimensional image FT is calculated. To do.

However,
T3A: Control point group Fk: Feature point of measurement data Gk: Feature point on standard model corresponding to feature point of measurement data N: Number of correspondence between feature point of measurement data and feature point on standard model L: Various Adjustment scale for handling energy in the same unit (stabilization energy to avoid excessive deformation)
In addition to the differential energy described above, a stabilization energy es is introduced to avoid excessive deformation.

  That is, it is assumed that the control points used for the deformation are connected by the virtual spring (elastic bar) KB shown in FIG. Based on the constraint of the virtual spring KB, a stabilization energy es for stabilizing the shape of the surface S of the standard model DS is defined.

  The virtual spring does not necessarily have to be stretched between the control points. It is sufficient if the relationship between the control point and the virtual spring is clear.

  In FIG. 9, a part of the surface S of the standard model DS to be fitted is shown. The surface S is formed by the control point group U = | ui, i = 1... N | A virtual spring KB is disposed between adjacent control points. The virtual spring KB acts to prevent abnormal deformation of the surface S by giving a restraint between the control points by a pulling force.

  That is, when the interval between adjacent control points u increases, the pulling force by the virtual spring KB increases accordingly. For example, when the point Qk approaches the point Pk, if the distance between the control points u increases with the movement, the pulling force by the virtual spring KB increases. Even if the point Qk moves, if the distance between the control points u does not change, that is, if the relative positional relationship between the control points u does not change, the pulling force by the virtual spring KB does not change. A value obtained by averaging the pulling force by the virtual spring KB over the entire surface S is defined as a stabilization energy es. Therefore, the stabilization energy es increases when a part of the surface S protrudes and deforms. If the entire surface S moves on average, the stabilization energy es is zero.

  The stabilization energy es is obtained by the following equation (11) depending on the deformation state of the virtual spring KB.

However,
TsA: Control point group Um, Vm: Initial value of end point (control point) of virtual spring Um, Vm: End point of virtual spring after deformation L0m: Length of virtual spring in initial state,
L0m = | U ~ m -V ~ m |
M: Number of virtual springs c: Spring coefficient L: Adjustment scale for handling various energies in the same unit Therefore, when the spring coefficient c is increased, the virtual spring KB becomes hard and difficult to deform.

By introducing such a stabilization energy function es, a constant constraint is provided for the shape change of the surface S, and excessive deformation of the surface S can be prevented.
[Total energy function]
As described above, the control point groups T1A, T2A, T3A, and TsA are used for the energy functions e1, e2, e3, and e4, respectively. Here, these control point groups T1A to TsA are the same, but can be different from each other as will be described later. Using these control point groups TA, the standard model DS is deformed to obtain a control point group TA that minimizes the total energy function e (TA) shown in the following equation (12).

However,
e1 (T1A): Difference energy between the constituent points of the 3D data and the model surface e2 ^s (T2A): Difference energy between the contours of the measured data for each model contour e3 (T3A): Feature points of the measured data and the model Energy es (TSA): Stabilization energy wi, c: Weight parameter for each energy TA = T1A = T2A = T3A = TSA
[Repeated deformation]
Actually, the deformation is repeatedly performed (# 17). That is, the control point is moved and repeatedly deformed. Assuming that the total energy function after the n-th deformation is en (TA), it is determined that the total energy function en (TA) has converged when the condition of the following equation (13) is satisfied.

  Now, the overall flow of the deformation process will be described with reference to FIG. First, a pair of corresponding points is created between the measurement data and the standard model DS (Pk and Qk in FIG. 8) (# 21).

  The surface S is deformed (# 22), and the total energy function en (TA) after deformation is calculated (# 23). The process is repeated until the total energy function en (TA) converges (Yes in # 24).

As a method of judging the convergence of the total energy function en (TA), as described above, the method of convergence when the total energy function en (TA) becomes smaller than a predetermined value, and the previous calculation were compared. It is possible to use a known method such as a method of converging when the rate of change becomes a predetermined value or less.
[Use of different control points]
In the expression (12) described above, the same control point group is used for energy functions e1 to e4 having different fitting objects (composition points, contours RK, and feature points TT of the three-dimensional data DT). The example uses a different control point group for each fitting object, that is, for each energy function. That is, the control point groups T1A, T2A, T3A, TsA are made different from each other.

  In this case, the above equation (12) can be used as the total energy function e (TA). However, the control point groups T1A, T2A, T3A, and TSA used there are different from each other and have the following relationship.

TA⊃T1A
TA⊃T2A
TA⊃T3A
TA = TSA
That is, as described above, since the feature points are points, the local movement occurs with respect to the energy between the feature points. For example, when trying to match the positions of the eyes of the three-dimensional data DT and the standard model DS, there is a possibility that only the portion where the feature point TT is set is strongly pulled and deformed into an irregular shape. In such a case, it is preferable to make the movement as a whole.

  On the other hand, for the constituent points of the three-dimensional data DT, I want the side of the eyes to move finely. However, when only a small number of control points are used, the constituent points do not move finely. Therefore, a large number of control points are used for the constituent points of the three-dimensional data DT, and a small number of control points are used for the feature points. The contour RK is an intermediate amount.

  For example, for a portion where the contour RK changes rapidly, the control point is made fine. Stabilization energy is applied to all control points. Such control point selection is performed when the standard model DS is prepared.

Although the control point groups T1A, T2A, T3A, and TsA are different from each other, some of the control points included in each control point group are commonly used.
[Change of weight according to reliability]
In the above equation (12), the total energy function e (TA) is evaluated on the assumption that the reliability of each information is equivalent. However, in the example shown here, the weight is set according to the reliability of each information. change. Accordingly, it is possible to make a determination by placing emphasis on more reliable information among various types of information.

  In addition, the reliability of each information shall be acquired at the time of three-dimensional measurement or the automatic extraction of an outline and a feature point. In this case, a control point group TA that minimizes the total energy function e (TA) shown in the following equation (14) is obtained.

However,
ρi: reliability of information on each energy function ei (TA) W (ρi): reliability function TA = T1A = T2A = T3A = TSA
(Partial area deformation)
By the processing up to step # 17, a more precise three-dimensional model ML can be obtained as compared with the conventional technique. However, since the processing up to step # 17 is performed for the entire target object, a precise fitting is not necessarily performed for a locally characteristic portion. Therefore, a characteristic part of the object or a part that has not been sufficiently fitted in the processing up to step # 17 is extracted from the three-dimensional data DT, and only the extracted three-dimensional data DT is used to further extract the standard model DS. (# 18).

  FIG. 11 is a diagram illustrating an example of extraction of the partial region BRY, and FIGS. 12A, 12B, and 12C are diagrams illustrating an example of a method for performing deformation of the standard model DS using the partial region BRY.

  As shown in FIG. 11, the partial region BRY is extracted from the three-dimensional data DT. Each of the partial regions BRYa, BRYb, and BRYc is three-dimensional data obtained by extracting the eyes and the periphery thereof, the mouth and the periphery thereof, the nose and the periphery thereof from the three-dimensional data DT, respectively. Each partial area BRY is subjected to data reduction in order to reduce the data amount as necessary. The data reduction process is performed in the same procedure as the data reduction performed in step # 15, for example. In the partial area modification described later, since the fitting area is limited to the partial area BRY, the right side of the equation (6) is made closer to 0 than in step # 15 to lower the data reduction rate, thereby reducing the partial area. Even when the precision of BRY is maintained, the fitting process does not require much time. Accordingly, the data reduction rate can be set for each partial area BRY in consideration of the range of the partial area BRY, the required precision, the processing time, and the like. The extraction of the partial area BRY may be performed in advance in the process of step # 12, that is, when the three-dimensional data DT is obtained, or may be performed after the process up to step # 17 is completed.

  As shown in FIG. 12, the standard models DS1a and DS1b show the process of partial area deformation of the standard model DS1.

  In step # 18, the standard model DS1 of FIG. 12A obtained by the processing up to step # 17 is fitted using the partial region BRYa as shown in FIG. 12B to obtain the standard model DS1a. . The fitting is performed in the same procedure as the deformation or the repetitive deformation as in Step # 16, and the control point group T1A, the point Pk, the point Qk, the equation (8) and the equation (9) in the equation (7). Control point group T2A, point pk, point qk, control point group T3A of equation (10), point Fk, point Gk, control point group TsA of equation (11), point Um, point Vm, or (13 A suitable value may be used for the threshold value ε and the like of the formula) according to the partial region BRY.

  For each partial region BRY, partial region deformation is sequentially performed (Yes in # 19, # 18). For each partial region BRY, fitting is performed with different control point groups. As a result, the standard model DS1 can be fitted more precisely for each partial region BRY.

  For example, after obtaining the standard model DS1a shown in FIG. 12B, fitting is performed using the partial region BRYb as shown in FIG. 12C to obtain the standard model DS1b (three-dimensional model ML).

  Each partial region BRY may overlap with a part of another partial region BRY, or may be included in another partial region BRY. For example, as shown in FIG. 11, a part of the partial area BRYa and a part of the partial area BRYc may overlap, or after the partial area deformation is performed on the partial area BRYb, only the upper lip is replaced with a new partial area. It may be extracted as BRY and subjected to partial region deformation.

  According to the above-described embodiment, since the standard model DS is partially fitted using the partial region BRY after the standard model DS is entirely fitted, the local area such as the eyes or the mouth is localized. They can be better matched without causing abnormal deformation. Therefore, a three-dimensional model ML that is closer to the object can be generated.

  In the embodiment described above, the configuration, circuit, processing content, processing order, processing timing, coefficient setting, and the like of the modeling apparatus 1 can be appropriately changed in accordance with the spirit of the present invention.

1 Modeling device (3D model generator)
10 Processing device DT 3D data (measurement data)
FT 2D image (measurement data)
BRY Partial area DS, DS1, DS1a Standard model DS1b Standard model (3D model)
ML 3D model TA Control point group Pk Component point (point group)
S-plane PR modeling program

Claims (4)

A method of transforming a standard model according to measured three-dimensional data using a processing device having a CPU and a memory,
Storing the three-dimensional data and the standard model in the memory;
By the CPU
Reducing the number of three-dimensional data in order to thin out the three-dimensional data;
Transforming the entire standard model so that the distance between the thinned outline of the three-dimensional data and the outline of the standard model is reduced;
Selecting a plurality of partial regions having features locally from the three-dimensional data before being thinned after deforming the whole of the standard model;
Performing only the corresponding part of the standard model using the three-dimensional data of the selected partial regions,
A standard model deformation method characterized by the above. In the standard model, control points for deforming the standard model are defined,
Changing at least one of the control points, the reduction rate for reducing the number of the three-dimensional data, or the evaluation function for evaluating the degree of deformation of the standard model for each partial region;
The method for deforming a standard model according to claim 1. A modeling device that generates a three-dimensional model by transforming a standard model according to measured three-dimensional data,
Means for reducing the number of the three-dimensional data in order to thin out the three-dimensional data;
Means for deforming the entire standard model so that the distance between the thinned outline of the three-dimensional data and the outline of the standard model is small;
Means for selecting a plurality of partial regions having features locally from the three-dimensional data before being thinned after the whole of the standard model is deformed;
Means for deforming only the corresponding part of the standard model using the three-dimensional data of the plurality of selected partial regions;
A modeling apparatus characterized by comprising: The means for deforming only the part is:
For each partial region, at least a control point defined in the standard model for deforming the standard model, a reduction rate for reducing the number of the three-dimensional data, or an evaluation function for evaluating the degree of deformation of the standard model Change one,
The modeling apparatus according to claim 3.

Images