JP2011149952A - Model input device and model generation system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a model input device and model generation system for accurately acquiring a moving solid model (three-dimensional moving image data) of an object. <P>SOLUTION: This model input device includes a photographing section 10c for acquiring color data of the object, an irradiation section 10a for irradiating the object with light having a wavelength other than that of the visible light, a light receiving section 10b for receiving light reflected by the object, a unit 10 including the photographing section, irradiation section, and light receiving section, and an interface capable of outputting, to the computer, the color data acquired by the photographing section and information related to the reflected light acquired by the light receiving section. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a model input device and a model generation system, and more particularly to a three-dimensional modeling technique for generating a motion solid model (three-dimensional model) of an object from a plurality of image data of a moving object.

  Conventionally, a motion capture system has been put to practical use as a method for acquiring motion data and applied to movie production and the like. This is mainly for the purpose of acquiring human motion data. By attaching markers or sensors to human joints and detecting their positions according to human motion, human skeleton motion data is obtained. It is.

  More specifically, there are two types of motion capture systems, one using a magnetic sensor and the other using an optical technique. In the case of using a magnetic sensor, a magnetic sensor is attached to a joint, and each sensor obtains its position based on a magnetic field in space. The position of each sensor changes according to human movement, and this is acquired as joint movement data. However, in this method, it is necessary to connect a cable to each sensor, and this causes a problem that the movement of a person becomes inconvenient. In addition, there is a problem that accurate motion data cannot be obtained in an environment where the magnetic field is disturbed.

  On the other hand, in the optical type, for example, an LED is attached to a joint as a marker. This is observed with a plurality of cameras, and the position of each LED is detected by the triangulation principle. However, this method has a problem that the data input space must be darkened, and it is difficult to acquire motion data in real time.

Furthermore, in either the magnetic method or the optical method, only motion data is acquired, and three-dimensional shape data of the moving object (person) itself is not acquired.
JP-A-11-219422 JP 2001-082940 A JP-A-11-175692

  On the other hand, CMU (Carnegie Mellon University) has announced a method in which a person operates in a space in which a plurality of cameras are arranged and three-dimensional data with movement of a target object (person) is obtained by a stereo method.

However, even with this method, although 3D data with motion can be obtained,
(1) Since it is based on the passive stereo method, the shape data is extremely inaccurate when there is not enough texture information on the object. (2) Since the list of 3D data is obtained, the amount of acquired data is It has the problem of becoming extremely large, and it cannot be said that it provides data suitable for actual application fields.

  The present invention has been made in view of the above-described problems of the prior art, and the object thereof is a moving object model (three-dimensional moving image data) with high accuracy without being affected by the surface color of the object. It is an object to provide a model input device and a model generation system that can acquire the model.

  In order to achieve the above object, the model input device of the present invention irradiates the object with a photographing unit (for example, the color camera 10c) that acquires color data of the object and light having a wavelength other than visible light. An irradiation unit (for example, projector 10a), a light receiving unit (for example, near-infrared camera array 10b) that receives the light reflected by the object, and the imaging unit, the irradiation unit, and the light receiving unit inside A unit (e.g., input unit 10), and an interface (e.g., interface I / F) that can output to the computer the color data obtained by the photographing unit and the information about the reflected light obtained by the light receiving unit And.

  In the present apparatus, it is preferable that the photographing unit and the irradiation unit are arranged in the unit as far apart as possible.

  In the present apparatus, three-dimensional shape data and motion data can be acquired based on the information related to the reflected light.

  In the present apparatus, it is preferable that the light receiving section includes a plurality of elements capable of receiving light having a wavelength other than visible light.

  In addition, the model generation system of the present invention includes an imaging unit (for example, a color camera 10c) that acquires color data of an object, and an irradiation unit (for example, a projector) that irradiates the object with light having a wavelength other than visible light. 10a), a light receiving unit (for example, near-infrared camera array 10b) that receives the light reflected by the object, and a unit (for example, the image capturing unit, the irradiation unit, and the light receiving unit) Model input comprising: an input unit 10); and an interface (for example, interface I / F) capable of outputting the color data obtained by the photographing unit and the information about the reflected light obtained by the light receiving unit to a computer A three-dimensional shape database in which three-dimensional shape data is registered and connected to the model input device via the interface Reflected light model synthesizer for synthesizing said registered a motion data on the three-dimensional shape database three-dimensional shape data based on information about (e.g., computer 18), characterized in that it comprises a, a.

  In the present system, the model synthesizing apparatus may further include an image memory for storing the color data acquired by the photographing unit.

  In this system, it is preferable that the model synthesizing device synthesizes the three-dimensional shape data based on the information about the reflected light and the color data acquired at the same time with the information about the reflected light.

  The system may further include an output device (for example, the output device 20) that is connected to the model synthesis device and outputs a dynamic solid model synthesized by the model synthesis device.

  Here, the voxel data that is the three-dimensional shape data is acquired over a continuous time, and the means for generating the skeleton data executes the thinning process of the voxel data acquired at a certain time over at least a plurality of times. It is preferable to generate the skeleton data by

  Further, the means for classifying the rigid body groups can classify the skeleton elements having substantially the same movement direction and size of the skeleton data as belonging to the same rigid body group. Further, the means for dividing into the rigid body group describes the motion direction and size of the skeleton data as a linear function of a part of the continuous voxel coordinates after the thinning process, and the linear function description It is also possible to classify consecutive voxels whose errors are below a certain value as belonging to the same rigid body group.

  The apparatus preferably further includes means for storing the skeleton data and the motion parameters for each rigid body group separately from the three-dimensional shape data and the color data.

  Here, it is also possible to have means for synthesizing the skeletal data and the motion parameters for each rigid body group with the three-dimensional shape data and color data obtained for different objects.

  As described above, according to the present invention, it is possible to acquire a moving three-dimensional model (three-dimensional moving image data) of a target object with high accuracy without being affected by the surface color of the target object.

It is a block diagram of embodiment. It is a block diagram of the computer in FIG. It is principle explanatory drawing (the 1) of a stereo method. It is a principle explanatory view (the 2) of the stereo method. It is a principle explanatory view (the 3) of the stereo method. It is explanatory drawing which shows the calculation process of three-dimensional data. It is thinning process explanatory drawing.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
First, the basic concept of this embodiment will be described. In order to input 3D moving image data with high accuracy regardless of the surface color of the target object, an active method is effective instead of a passive method. That is, it is a method of obtaining the shape of an object by irradiating the object with some pattern light and observing the pattern light. Typical patterns used here include structured light such as slit light, random light, and the like.

  Now, when it is assumed that three-dimensional data with movement of the entire circumference is acquired, it is necessary to irradiate a pattern simultaneously from a plurality of directions, not just one direction. Here, since structured light such as slit light is meaningful in the pattern itself, if the same object is irradiated simultaneously from multiple directions, the pattern light will interfere with each other, resulting in incorrect shape data. Become.

  On the other hand, in the case of random light, the pattern itself has no meaning. Therefore, there is no problem even if simultaneous irradiation from a plurality of directions is performed. However, a plurality of input cameras are required, and the stereo method may be applied with these.

  In addition, in order to acquire color (texture) data from all directions for the target object, it is necessary to photograph the target object from a plurality of directions with a color camera. The texture data acquisition must be performed simultaneously with the input of the three-dimensional data. However, the pattern light should not be reflected in an image photographed by a color camera. This is because this pattern light gives unnecessary information other than the color of the original object.

  From the above, it is preferable to use an input unit composed of the following.

(A) Other than visible light, for example, a random pattern irradiation projector using near infrared rays, (b) a near infrared camera array, (c) a color camera, and a plurality of units arranged on the object to move around the entire object. It is possible to input 3D data.

  FIG. 1 shows the configuration of the dynamic three-dimensional model generation device of the present embodiment. FIG. 1A is an input unit of the dynamic stereoscopic model creation apparatus, and FIG. 1B is an overall configuration diagram of the dynamic stereoscopic model generation apparatus.

  The input unit 10 includes a projector 10a that emits a random pattern of near infrared rays, a near infrared camera array 10b that receives a random pattern irradiated from the projector 10a and reflected by the subject, and a color for acquiring a color texture of the subject. It consists of a camera 10c.

  A plurality of such input units, for example, four input units 10, 12, 14, 16 as shown in FIG. 1B, are arranged around the object (for example, a person), The near-infrared image obtained by the near-infrared camera array 10b and the color image obtained by the color camera 10c are supplied to the processing computer 18. The relative positional relationship between the near-infrared camera array 10b and the color camera 10c in each unit is known and fixed, and the relative positional relationship between the units is accurately calculated in advance. Further, it is desirable that the near-infrared camera array 10b is synchronized with each other among all the units arranged, and it is desirable that the color camera 10c is also synchronized.

  FIG. 2 is a block diagram showing the configuration of the computer 18 in FIG. It has an input / output interface I / F, CPU, ROM, image memory, three-dimensional shape database, and skeleton database.

  Image data obtained by the near-infrared camera and the color camera is taken into the computer 18 from the interface I / F and stored in the image memory.

  The CPU executes processing to be described later from the image data stored in the image memory, acquires three-dimensional shape data and color data, and registers them in the three-dimensional shape database. In this embodiment, in order to model a moving three-dimensional object, the three-dimensional shape database stores temporally continuous three-dimensional shape data. Three-dimensional shape data at a certain time can be called a frame. Further, the CPU separates and extracts the skeleton data and its motion data from the three-dimensional shape data and the color data stored in the three-dimensional shape database, and registers them in the skeleton database.

  The ROM stores a processing program for executing a modeling process, which will be described later, and the CPU reads a program stored in the ROM and executes it sequentially to model a moving three-dimensional object. In this embodiment, the program is stored in the ROM, but may be stored in another storage medium such as a hard disk or a CD-ROM.

  The flow of processing of image data of an object input at a certain time by the above apparatus is as follows.

(1) Acquisition of partial depth data (2) Calculation of all three-dimensional data (3) Acquisition of textured three-dimensional data These processes will be sequentially described.

<Acquisition of partial depth data>
Acquisition of partial depth data refers to acquiring partial three-dimensional data acquired in each unit. Here, the object is irradiated with a random pattern from the near-infrared projector 10a. An image obtained by photographing this image with the near-infrared camera array 10b becomes an input image. By using a well-known method such as multi-baseline stereo, distance data (depth data) from the camera array 10b can be obtained for the common visual field portion of all the cameras in the object.

  The principle of multi-baseline stereo is as follows. Now, as shown in FIG. 3, three cameras A, B, and C are arranged around the object 100, and images obtained by the respective cameras are an image 102 (an image of camera A) and an image 104 (a camera B). ) And image 106 (camera C image). As corresponding points of the point a in the image 102 of the camera A, there are a plurality of corresponding point candidates in the image 104 of the camera B, as shown in FIG. 4, b1, b2, and b3. The depth is calculated as z1, z2, and z3 according to each of these corresponding point candidates. At this stage, it is not possible to specify which is a correct depth candidate. However, when three or more cameras are used, it is possible to narrow down from these multiple candidates. For example, when the camera C exists as shown in FIG. 5, a plurality of corresponding point candidates corresponding to the point a of the image 102 of the camera A exist in C, and the respective depths exist. However, among these candidates, the correct corresponding point candidates should have the same depth value. That is, after obtaining corresponding point candidates using a plurality of cameras, those having the same depth (Q in the figure) are finally regarded as true corresponding points, and the depth value of that point is the final depth data. And However, these depth data are only partial information viewed from each unit at this point. Next, these need to be integrated.

<Calculation of all three-dimensional data>
When the input unit is composed of four input units 10, 12, 14, and 16 as shown in FIG. 1B, a common space exists in the field of view of the four units. And it can be assumed that the object exists in this common space. Otherwise, data cannot be acquired for at least a part of the object. This common space is expressed by a voxel space, that is, a set of minute cubes.

  The partial depth data is calculated for each unit by the partial depth data acquisition process described above. If the depth data from each unit is Dz and the distance between each voxel in the voxel space and the unit is Dv, these partial depth data are integrated by the following processing.

(1) As shown in FIG. 6, if Dv> Dz for each unit and for all voxels, one point is added to that voxel. That is, one point is added to the voxels farther than the depth data among all the voxels. By performing for four units, each voxel takes one of the values 0-4. Incidentally, 0 is a voxel located in front of the depth data, and corresponds to the outside of the object.

(2) Only voxels that have acquired 4 points are extracted, and these are regarded as voxels corresponding to the object.

  When n input units are used, the voxel that has acquired n points in step (2) may be extracted. If n is large, voxels that have acquired (n-1) points, more generally (nm) points (n> m) in step (2) may be extracted.

  Through the above processing, voxels corresponding to the object are extracted, and data expressed in the voxel space is converted into a polygon representation.

<Acquisition of textured 3D data>
Color data (texture data) to be assigned to each polygon is acquired with reference to images taken by the color camera 10c at the same time.

  By performing the above three processes for all times, accurate three-dimensional moving image data can be obtained.

  The calculated three-dimensional moving image data is registered in the three-dimensional shape database, and the CPU can read it and output it to the output device 20 as necessary.

Second Embodiment
In the first embodiment described above, a list of 3D image data is obtained. However, by further separating motion data and shape data from this list, it is possible to express 3D moving image data with a very small amount of data, and it is possible to extract only motion data and apply it to other shape data. , Etc. are generated.

  Therefore, in this embodiment, a method for separating motion data and shape data will be described. The configuration of the apparatus is the same as in FIG. 1 and FIG. 2, and all images are taken in synchronization every 1/30 seconds, and these images taken at the same time are called the same time frame. Using the information of the same time frame, three-dimensional data (shape + color) in each frame is obtained.

  The flow of processing is as follows.

(1) Generation of skeleton data in each frame (2) Comparison of skeleton data between frames (3) Generation of final skeleton data Hereinafter, these processes will be sequentially described.

<Generation of skeleton data between frames>
As described in the first embodiment, the three-dimensional shape data in each frame is once expressed as voxel data. Here, a thinning process well known in the image processing field is performed. The thinning process is described, for example, in “Introduction to Computer Image Processing” (Hideyuki Tamura, edited by Japan Industrial Technology Center). Thinning is an operation of narrowing the line width from a given figure and extracting the center line of width 1. “Throughout the boundary points in the image, all the pixels that are erasable and are not the end points of the line Based on the “erase” algorithm. This basic algorithm can be executed as a single operation that is performed on the pixels of the entire image, and is repeated until there are no more pixels to be erased.

  In general, the thinning process is performed on a two-dimensional image expressed in a pixel space, but it is easy to extend this to a three-dimensional image, that is, a voxel space. The result of the thinning process corresponds to an object, for example, a human skeleton.

  FIG. 7 schematically shows an object (person) that has been thinned. FIG. 7A is a two-dimensional representation of voxel data, and FIG. 7B is a thinning process of this voxel data. The solid line in FIG. 7B is a connection of unit voxels.

<Comparison of skeleton data between frames>
Here, the position of the point sequence obtained as a result of the thinning process is compared between consecutive frames to obtain the motion of the point sequence.

  Specifically, the following processing is performed.

(1) Find feature points (end points, branch points) of a point sequence.

(2) Corresponding neighboring feature points in adjacent frames.

(3) The point sequence between the feature points is associated.

  Here, it is assumed that the movement of the object is continuous and that the movement is sufficiently slow with respect to the shooting time interval. Therefore, it is assumed that the feature points are sufficiently close between adjacent frames. For the feature points of the point sequence in (1), a corresponding point in the next frame may be searched for in the vicinity of each feature point in the previous frame. This is the process (2). Each point sequence sandwiched between feature points is associated with each other.

  Here, the distance between the feature points is essentially the same, but does not always match due to an image processing error or the like. For this reason, it is preferable to use a method such as assigning almost average.

<Final skeleton data generation>
Here, the movement vector between each frame is calculated based on the inter-frame correspondence of all points. Then, point grouping is performed. Specifically, it is assumed that the movement vectors in the adjacent points have the same direction and have substantially the same size are in the same group. Further, it is determined whether or not the same group of members satisfies the constraint that the body motion is rigid. For example, when the arm is bent, the point sequences belonging to the skeletons of the lower arm and the upper arm perform the same rigid body motion, but those of different skeletons correspond to different rigid body motions. . Therefore, grouping can also be expressed as separating a point sequence into a plurality of rigid body groups. Finally, rigid body motion parameters (translation and rotation) are determined for these grouped point sequences.

  As described above, the skeleton and its motion data can be automatically extracted.

  The extracted skeleton and motion data are registered in the skeleton database in FIG. The obtained skeleton data and motion data are extremely useful because they can be applied to other data having a similar skeleton generated by CG or the like and given the same motion.

  Also, by reading out the skeleton and motion data registered in the skeleton database, and replacing the separated shape data with the shape data for other objects, it is the same as the object having an arbitrary shape on the computer. It is also possible to make such a movement. In other words, if the skeleton and motion data are Φ and the shape data is Ψ, the original dynamic solid model is Φ + Ψ, but instead of Ψ, three-dimensional information about other objects registered in advance in the three-dimensional shape database By generating Φ + Ω using the shape data Ω, Φ can be moved to the shape of Ω.

  In the above, the method for automatically obtaining the skeleton data has been described. However, the skeleton may be manually assigned in a certain frame, and the movement data may be obtained by tracking the movement of the skeleton based on this.

  10, 12, 14, 16 input unit, 18 computer, 20 output device.

Claims (8)

An imaging unit for obtaining color data of an object;
An irradiation unit for irradiating the object with light having a wavelength other than visible light;
A light receiving unit for receiving the light reflected by the object;
A unit comprising the imaging unit, the irradiation unit, and the light receiving unit inside;
An interface capable of outputting, to a computer, the color data obtained by the photographing unit and information about the reflected light obtained by the light receiving unit;
A model input device comprising: The model generation apparatus according to claim 1,
The model input device, wherein the photographing unit and the irradiation unit are arranged in the unit as far apart as possible. The model generation apparatus according to claim 1,
A model input device that acquires three-dimensional shape data and motion data based on information about the reflected light. The model generation apparatus according to claim 1,
The model input device, wherein the light receiving unit includes a plurality of elements capable of receiving light having a wavelength other than visible light. An imaging unit that acquires color data of an object, an irradiation unit that irradiates the object with light having a wavelength other than visible light, a light receiving unit that receives the light reflected by the object, and the inside A unit including an imaging unit, the irradiation unit, and the light receiving unit; and an interface capable of outputting the color data obtained by the imaging unit and information on the reflected light obtained by the light receiving unit to a computer. A model input device;
A three-dimensional shape database connected to the model input device via the interface and registered with three-dimensional shape data, and motion data based on information on the reflected light and the third order registered in the three-dimensional shape database A model synthesizer that synthesizes the original shape data;
A model generation system comprising: The model generation system according to claim 5, wherein
The model synthesizing apparatus further includes an image memory for storing the color data acquired by the photographing unit. The model generation system according to claim 5, wherein
The model synthesizing apparatus synthesizes three-dimensional shape data based on information about the reflected light and color data acquired at the same time with the information about the reflected light. The model generation system according to claim 5, further comprising an output device connected to the model synthesis device and outputting a dynamic solid model synthesized by the model synthesis device.