JP4761670B2 - Moving stereo model generation apparatus and method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an apparatus and method for creating a moving 3D model, and more particularly to a 3D modeling technique for generating a moving 3D model (3D model) of an object from a plurality of image data of a moving object.
[0002]
[Prior art]
Conventionally, a motion capture system has been put to practical use as a method for acquiring motion data and applied to movie production and the like. This is mainly for the purpose of acquiring human motion data. By attaching markers or sensors to human joints and detecting their positions according to human motion, human skeleton motion data is obtained. It is.
[0003]
More specifically, there are two types of motion capture systems, one using a magnetic sensor and the other using an optical technique. In the case of using a magnetic sensor, a magnetic sensor is attached to a joint, and each sensor obtains its position based on a magnetic field in space. The position of each sensor changes according to human movement, and this is acquired as joint movement data. However, in this method, it is necessary to connect a cable to each sensor, and this causes a problem that the movement of a person becomes inconvenient. In addition, there is a problem that accurate motion data cannot be obtained in an environment where the magnetic field is disturbed.
[0004]
On the other hand, in the optical type, for example, an LED is attached to a joint as a marker. This is observed with a plurality of cameras, and the position of each LED is detected by the triangulation principle. However, this method has a problem that the data input space must be darkened, and it is difficult to acquire motion data in real time.
[0005]
Furthermore, in either the magnetic method or the optical method, only motion data is acquired, and three-dimensional shape data of the moving object (person) itself is not acquired.
[0006]
[Problems to be solved by the invention]
On the other hand, CMU (Carnegie Mellon University) has announced a method in which a person operates in a space in which a plurality of cameras are arranged and three-dimensional data with movement of a target object (person) is obtained by a stereo method.
[0007]
However, even with this method, although 3D data with motion can be obtained,
(1) Since it is based on the passive stereo method, the shape data is extremely inaccurate when there is not enough texture information on the object. (2) Since the list of 3D data is obtained, the amount of acquired data is It has the problem of becoming extremely large, and it cannot be said that it provides data suitable for actual application fields.
[0008]
The present invention has been made in view of the above-described problems of the prior art, and the object thereof is a moving object model (three-dimensional moving image data) with high accuracy without being affected by the surface color of the object. It is an object of the present invention to provide an apparatus and a method capable of obtaining the above.
[0009]
[Means for Solving the Problems]
In order to achieve the above object, the present invention provides a dynamic solid model generation apparatus for generating three-dimensional shape data, color data, and motion data of a three-dimensional object. Irradiating means for irradiating the three-dimensional object, pattern light receiving means for acquiring the three-dimensional shape data and motion data by sequentially receiving the pattern light reflected from the three-dimensional object, and photographing the three-dimensional object Photographing means for obtaining color data, and processing means for combining the color data and the three-dimensional shape data obtained at the same time, and the pattern light has a wavelength other than the sensitivity wavelength of the photographing means. It is characterized by having.
[0010]
In this apparatus, it is preferable that the pattern light is random pattern light. In the present apparatus, it is preferable that the pattern light has a near infrared wavelength. Further, in the present apparatus, means for generating skeleton data from the synthesized three-dimensional shape data, means for dividing the skeleton data into a plurality of rigid body groups from the movement of the skeleton data, and for each rigid body group Means for calculating the motion parameter.
[0011]
Here, the voxel data that is the three-dimensional shape data is acquired over a continuous time, and the means for generating the skeleton data executes the thinning process of the voxel data acquired at a certain time over at least a plurality of times. It is preferable to generate the skeleton data by
[0012]
Further, the means for classifying the rigid body groups can classify the skeleton elements having substantially the same movement direction and size of the skeleton data as belonging to the same rigid body group.
[0013]
Further, the means for dividing into the rigid body group describes the motion direction and size of the skeleton data as a linear function of a part of the continuous voxel coordinates after the thinning process, and the linear function description It is also possible to classify consecutive voxels whose errors are below a certain value as belonging to the same rigid body group.
[0014]
The apparatus preferably further includes means for storing the skeleton data and the motion parameters for each rigid body group separately from the three-dimensional shape data and the color data.
[0015]
Here, it is also possible to have means for synthesizing the skeletal data and the motion parameters for each rigid body group with the three-dimensional shape data and color data obtained for different objects.
[0016]
The present invention also provides a dynamic solid model generation method for generating three-dimensional shape data, color data, and motion data of a three-dimensional object. In this method, the step of irradiating the three-dimensional object in motion with pattern light and the step of acquiring the three-dimensional shape data and the movement data by sequentially receiving the pattern light reflected from the three-dimensional object And acquiring the color data by photographing the three-dimensional object, and synthesizing the three-dimensional shape data and the color data acquired at the same time, and the pattern light is visible It has a wavelength other than the light wavelength.
[0017]
In the present method, the pattern light is preferably random pattern light. In the present method, it is preferable that the pattern light has a near infrared wavelength. Further, in the method, the step of generating skeleton data from the synthesized three-dimensional shape data, the step of dividing the skeleton data into a plurality of rigid body parts from the movement of the skeleton data, and for each rigid body part Preferably, calculating a motion parameter.
[0018]
Here, the step of generating the skeleton data may include a step of thinning the voxel data acquired at a certain time over at least a plurality of times.
[0019]
In the step of dividing into the rigid body portions, the division can be made based on the direction and magnitude of the movement of the skeleton data.
[0020]
Further, in the step of segmenting into the rigid body portions, the motion direction and size of the skeleton data are described as a linear function of a part of continuous voxel coordinates after the thinning process, and the linear function description It is also possible to classify consecutive voxels whose errors are below a certain value as belonging to the same rigid body group.
[0021]
The method may further include the step of storing the skeletal data and the motion parameters for each rigid body part in a computer memory separately from the three-dimensional shape data and the color data.
[0022]
The method may further include a step of synthesizing the skeletal data and the motion parameters for each rigid body part with the three-dimensional shape data and color data obtained for different objects stored in the memory in advance. it can.
[0023]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0024]
<First Embodiment>
First, the basic concept of this embodiment will be described. In order to input 3D moving image data with high accuracy regardless of the surface color of the target object, an active method is effective instead of a passive method. That is, it is a method of obtaining the shape of an object by irradiating the object with some pattern light and observing the pattern light. Typical patterns used here include structured light such as slit light, random light, and the like.
[0025]
Now, when it is assumed that three-dimensional data with movement of the entire circumference is acquired, it is necessary to irradiate a pattern simultaneously from a plurality of directions, not just one direction. Here, since structured light such as slit light is meaningful in the pattern itself, if the same object is irradiated simultaneously from multiple directions, the pattern light will interfere with each other, resulting in incorrect shape data. Become.
[0026]
On the other hand, in the case of random light, the pattern itself has no meaning. Therefore, there is no problem even if simultaneous irradiation from a plurality of directions is performed. However, a plurality of input cameras are required, and the stereo method may be applied with these.
[0027]
In addition, in order to acquire color (texture) data from all directions for the target object, it is necessary to photograph the target object from a plurality of directions with a color camera. The texture data acquisition must be performed simultaneously with the input of the three-dimensional data. However, the pattern light should not be reflected in an image photographed by a color camera. This is because this pattern light gives unnecessary information other than the color of the original object.
[0028]
From the above, it is preferable to use an input unit composed of the following.
[0029]
(A) Other than visible light, for example, a random pattern irradiation projector using near infrared rays, (b) a near infrared camera array, (c) a color camera, and a plurality of units arranged on the object to move around the entire object. It is possible to input 3D data.
[0030]
FIG. 1 shows the configuration of the dynamic three-dimensional model generation device of the present embodiment. FIG. 1A is an input unit of the dynamic stereoscopic model creation apparatus, and FIG. 1B is an overall configuration diagram of the dynamic stereoscopic model generation apparatus.
[0031]
The input unit 10 includes a projector 10a that emits a random pattern of near infrared rays, a near infrared camera array 10b that receives a random pattern irradiated from the projector 10a and reflected by the subject, and a color for acquiring a color texture of the subject. It consists of a camera 10c.
[0032]
A plurality of such input units, for example, four input units 10, 12, 14, 16 as shown in FIG. 1B, are arranged around the object (for example, a person), The near-infrared image obtained by the near-infrared camera array 10b and the color image obtained by the color camera 10c are supplied to the processing computer 18. The relative positional relationship between the near-infrared camera array 10b and the color camera 10c in each unit is known and fixed, and the relative positional relationship between the units is accurately calculated in advance. Further, it is desirable that the near-infrared camera array 10b is synchronized with each other among all the units arranged, and it is desirable that the color camera 10c is also synchronized.
[0033]
FIG. 2 is a block diagram showing the configuration of the computer 18 in FIG. It has an input / output interface I / F, CPU, ROM, image memory, three-dimensional shape database, and skeleton database.
[0034]
Image data obtained by the near-infrared camera and the color camera is taken into the computer 18 from the interface I / F and stored in the image memory.
[0035]
The CPU executes processing to be described later from the image data stored in the image memory, acquires three-dimensional shape data and color data, and registers them in the three-dimensional shape database. In this embodiment, in order to model a moving three-dimensional object, the three-dimensional shape database stores temporally continuous three-dimensional shape data. Three-dimensional shape data at a certain time can be called a frame. Further, the CPU separates and extracts the skeleton data and its motion data from the three-dimensional shape data and the color data stored in the three-dimensional shape database, and registers them in the skeleton database.
[0036]
The ROM stores a processing program for executing a modeling process, which will be described later, and the CPU reads a program stored in the ROM and executes it sequentially to model a moving three-dimensional object. In this embodiment, the program is stored in the ROM, but may be stored in another storage medium such as a hard disk or a CD-ROM.
[0037]
The flow of processing of image data of an object input at a certain time by the above apparatus is as follows.
[0038]
(1) Acquisition of partial depth data (2) Calculation of all three-dimensional data (3) Acquisition of textured three-dimensional data These processes will be sequentially described.
[0039]
<Acquisition of partial depth data>
Acquisition of partial depth data refers to acquiring partial three-dimensional data acquired in each unit. Here, the object is irradiated with a random pattern from the near-infrared projector 10a. An image obtained by photographing this image with the near-infrared camera array 10b becomes an input image. By using a well-known method such as multi-baseline stereo, distance data (depth data) from the camera array 10b can be obtained for the common visual field portion of all the cameras in the object.
[0040]
The principle of multi-baseline stereo is as follows. Now, as shown in FIG. 3, three cameras A, B, and C are arranged around the object 100, and images obtained by the respective cameras are an image 102 (an image of camera A) and an image 104 (a camera B). ) And image 106 (camera C image). As corresponding points of the point a in the image 102 of the camera A, there are a plurality of corresponding point candidates in the image 104 of the camera B, as shown in FIG. 4, b1, b2, and b3. The depth is calculated as z1, z2, and z3 according to each of these corresponding point candidates. At this stage, it is not possible to specify which is a correct depth candidate. However, when three or more cameras are used, it is possible to narrow down from these multiple candidates. For example, when the camera C exists as shown in FIG. 5, a plurality of corresponding point candidates corresponding to the point a of the image 102 of the camera A exist in C, and the respective depths exist. However, among these candidates, the correct corresponding point candidates should have the same depth value. That is, after obtaining corresponding point candidates using a plurality of cameras, those having the same depth (Q in the figure) are finally regarded as true corresponding points, and the depth value of that point is the final depth data. And However, these depth data are only partial information viewed from each unit at this point. Next, these need to be integrated.
[0041]
<Calculation of all three-dimensional data>
When the input unit is composed of four input units 10, 12, 14, and 16 as shown in FIG. 1B, a common space exists in the field of view of the four units. And it can be assumed that the object exists in this common space. Otherwise, data cannot be acquired for at least a part of the object. This common space is expressed by a voxel space, that is, a set of minute cubes.
[0042]
The partial depth data is calculated for each unit by the partial depth data acquisition process described above. If the depth data from each unit is Dz and the distance between each voxel in the voxel space and the unit is Dv, these partial depth data are integrated by the following processing.
[0043]
(1) As shown in FIG. 6, if Dv> Dz for each unit and for all voxels, one point is added to that voxel. That is, one point is added to the voxels farther than the depth data among all the voxels. By performing for four units, each voxel takes one of the values 0-4. Incidentally, 0 is a voxel located in front of the depth data, and corresponds to the outside of the object.
[0044]
(2) Only voxels that have acquired 4 points are extracted, and these are regarded as voxels corresponding to the object.
[0045]
When n input units are used, the voxel that has acquired n points in step (2) may be extracted. If n is large, voxels that have acquired (n-1) points, more generally (nm) points (n> m) in step (2) may be extracted.
[0046]
Through the above processing, voxels corresponding to the object are extracted, and data expressed in the voxel space is converted into a polygon representation.
[0047]
<Acquisition of textured 3D data>
Color data (texture data) to be assigned to each polygon is acquired with reference to images taken by the color camera 10c at the same time.
[0048]
By performing the above three processes for all times, accurate three-dimensional moving image data can be obtained.
[0049]
The calculated three-dimensional moving image data is registered in the three-dimensional shape database, and the CPU can read it and output it to the output device 20 as necessary.
[0050]
Second Embodiment
In the first embodiment described above, a list of 3D image data is obtained. However, by further separating motion data and shape data from this list, it is possible to express 3D moving image data with a very small amount of data, and it is possible to extract only motion data and apply it to other shape data. , Etc. are generated.
[0051]
Therefore, in this embodiment, a method for separating motion data and shape data will be described. The configuration of the apparatus is the same as in FIG. 1 and FIG. 2, and all images are taken in synchronization every 1/30 seconds, and these images taken at the same time are called the same time frame. Using the information of the same time frame, three-dimensional data (shape + color) in each frame is obtained.
[0052]
The flow of processing is as follows.
[0053]
(1) Generation of skeleton data in each frame (2) Comparison of skeleton data between frames (3) Generation of final skeleton data Hereinafter, these processes will be sequentially described.
[0054]
<Generation of skeleton data between frames>
As described in the first embodiment, the three-dimensional shape data in each frame is once expressed as voxel data. Here, a thinning process well known in the image processing field is performed. The thinning process is described, for example, in “Introduction to Computer Image Processing” (Hideyuki Tamura, edited by Japan Industrial Technology Center). Thinning is an operation of narrowing the line width from a given figure and extracting the center line of width 1. “Throughout the boundary points in the image, all the pixels that are erasable and are not the end points of the line Based on the “erase” algorithm. This basic algorithm can be executed as a single operation that is performed on the pixels of the entire image, and is repeated until there are no more pixels to be erased.
[0055]
In general, the thinning process is performed on a two-dimensional image expressed in a pixel space, but it is easy to extend this to a three-dimensional image, that is, a voxel space. The result of the thinning process corresponds to an object, for example, a human skeleton.
[0056]
FIG. 7 schematically shows an object (person) that has been thinned. FIG. 7A is a two-dimensional representation of voxel data, and FIG. 7B is a thinning process of this voxel data. The solid line in FIG. 7B is a connection of unit voxels.
[0057]
<Comparison of skeleton data between frames>
Here, the position of the point sequence obtained as a result of the thinning process is compared between consecutive frames to obtain the motion of the point sequence.
[0058]
Specifically, the following processing is performed.
[0059]
(1) Find feature points (end points, branch points) of a point sequence.
[0060]
(2) Corresponding neighboring feature points in adjacent frames.
[0061]
(3) The point sequence between the feature points is associated.
[0062]
Here, it is assumed that the movement of the object is continuous and that the movement is sufficiently slow with respect to the shooting time interval. Therefore, it is assumed that the feature points are sufficiently close between adjacent frames. For the feature points of the point sequence in (1), a corresponding point in the next frame may be searched for in the vicinity of each feature point in the previous frame. This is the process (2). Each point sequence sandwiched between feature points is associated with each other.
[0063]
Here, the distance between the feature points is essentially the same, but does not always match due to an image processing error or the like. For this reason, it is preferable to use a method such as assigning almost average.
[0064]
<Final skeleton data generation>
Here, the movement vector between each frame is calculated based on the inter-frame correspondence of all points. Then, point grouping is performed. Specifically, it is assumed that the movement vectors in the adjacent points have the same direction and have substantially the same size are in the same group. Further, it is determined whether or not the same group of members satisfies the constraint that the body motion is rigid. For example, when the arm is bent, the point sequences belonging to the skeletons of the lower arm and the upper arm perform the same rigid body motion, but those of different skeletons correspond to different rigid body motions. . Therefore, grouping can also be expressed as separating a point sequence into a plurality of rigid body groups. Finally, rigid body motion parameters (translation and rotation) are determined for these grouped point sequences.
[0065]
As described above, the skeleton and its motion data can be automatically extracted.
[0066]
The extracted skeleton and motion data are registered in the skeleton database in FIG. The obtained skeleton data and motion data are extremely useful because they can be applied to other data having a similar skeleton generated by CG or the like and given the same motion.
[0067]
Also, by reading out the skeleton and motion data registered in the skeleton database, and replacing the separated shape data with the shape data for other objects, it is the same as the object having an arbitrary shape on the computer. It is also possible to make such a movement. In other words, if the skeleton and motion data are Φ and the shape data is Ψ, the original dynamic solid model is Φ + Ψ, but instead of Ψ, three-dimensional information about other objects registered in advance in the three-dimensional shape database By generating Φ + Ω using the shape data Ω, Φ can be moved to the shape of Ω.
[0068]
In the above, the method for automatically obtaining the skeleton data has been described. However, the skeleton may be manually assigned in a certain frame, and the movement data may be obtained by tracking the movement of the skeleton based on this.
[0069]
【The invention's effect】
As described above, according to the present invention, it is possible to acquire a moving three-dimensional model (three-dimensional moving image data) of a target object with high accuracy without being affected by the surface color of the target object.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of an embodiment.
FIG. 2 is a configuration diagram of a computer in FIG. 1;
FIG. 3 is a diagram for explaining the principle of a stereo method (No. 1).
FIG. 4 is a diagram for explaining the principle of a stereo method (No. 2).
FIG. 5 is a diagram for explaining the principle of a stereo method (No. 3).
FIG. 6 is an explanatory diagram showing calculation processing of three-dimensional data.
FIG. 7 is an explanatory diagram of thinning processing.
[Explanation of symbols]
10, 12, 14, 16 input unit, 18 computer, 20 output device.

Claims (4)

A moving and solid model generating device for generating three-dimensional shape data, color data, and motion data of a three-dimensional object, the photographing means for obtaining the color data by photographing the three-dimensional object, and the tertiary having motion Irradiation means for irradiating the original object with pattern light having a wavelength other than the sensitivity wavelength of the imaging means; and sequentially receiving the pattern light reflected from the three-dimensional object, the three-dimensional shape data and motion data are obtained. In a dynamic three-dimensional model generation device comprising: a pattern light receiving unit to acquire; and a processing unit that combines the three-dimensional shape data acquired at the same time with the color data .
Means for generating skeleton data from the synthesized three-dimensional shape data;
Means for dividing a skeleton element that is substantially the same from the direction and size of movement of the skeleton data into a plurality of rigid body groups as belonging to the same rigid body group;
Means for calculating a motion parameter for each rigid body group;
A dynamic three-dimensional model generation apparatus characterized by comprising: The apparatus of claim 1.
An apparatus for generating a dynamic solid model, comprising means for storing the skeletal data and motion parameters for each rigid group separately from the three-dimensional shape data and color data . The apparatus of claim 2.
A dynamic three-dimensional model generation apparatus comprising means for synthesizing the skeleton data and the motion parameters for each rigid body group with the three-dimensional shape data and color data obtained for a different object . A moving solid model generation method for generating three-dimensional shape data, color data, and motion data of a three-dimensional object, the step of acquiring the color data by photographing the three-dimensional object, and the three-dimensional motion Irradiating an object with pattern light having a wavelength other than a visible light wavelength ; obtaining the three-dimensional shape data and motion data by sequentially receiving the pattern light reflected from the three-dimensional object ; Combining the three-dimensional shape data and the color data acquired at the same time ,
Generating skeleton data from the synthesized three-dimensional shape data;
Dividing a skeleton element that is substantially the same from the direction and size of the movement of the skeleton data into a plurality of rigid body groups as belonging to the same rigid body group;
Calculating a motion parameter for each rigid body part;
A dynamic three-dimensional model generation method characterized by comprising:

Images