JP3456029B2 - 3D object recognition device based on image data - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an autonomous mobile vehicle that recognizes an environment outside the vehicle by image processing technology and discovers a target object (automobile, obstacle, building, etc.) of known shape, or an autonomous mobile robot. A visual recognition device for detecting a three-dimensional positional relationship between a known target object and itself by image processing technology, or a model-based coding device for constructing a three-dimensional video database and transmitting an ultra-low bit rate image The present invention relates to a three-dimensional object recognition device based on image data that can be used for such purposes.

[0002]

2. Description of the Related Art In recent years, with the increase in demand for automation equipment for FA inspection, unmanned monitoring equipment, unmanned mobile vehicles, or robot vision represented by visual equipment for autonomous mobile robots, movement of objects and the environment There is a demand for an apparatus capable of recognizing shapes, colors, and the like and instantly and freely creating three-dimensional images viewed from different viewpoints. In addition, next-generation video coding technology for ultra-low bit rate image transmission such as TV phones and realistic communication of 3D video is being researched. There is a demand for automatic estimation of movement and deformation. Further, even in the image synthesizing / editing technique using computer graphics, there is a great demand for automating modeling of an object, and such a demand is increasing more and more due to the advent of artificial reality technology.

For example, for an autonomous mobile robot for FA that handles a work whose shape is known in a factory, three-dimensional position / orientation based on two-dimensional visual information (image information) from a monocular camera (for example, CCD camera). Is required to be recognized. In this case, the goal is to calculate the three-dimensional current position and orientation of the monocular camera based on feature amount extraction and shape parameter calculation from the image processing target fixed in the absolute coordinate system.

As such a position / orientation recognition method for FA, for example, JP-A-6-262568 and JP-A-6-262568 are available.
As disclosed in Japanese Patent No. 258028, the image feature points indicating the feature of the shape of the target object are extracted based on the image information, and the parameter space expressing the position and orientation of the camera is quantized at the grid interval of the target accuracy, It is considered that the optimum posture position is searched by minimizing the error evaluation function of the image feature amount corresponding to each quantized spatial point. Since such a position and orientation recognition method is intended to handle an object with a known shape in a known indoor environment inside a factory, the appearance of the object can be limited, and the lighting conditions are almost constant. Therefore, the feature points can be binarized and extracted, and the search area can be limited from the machine stop accuracy. Therefore, the optimum position and orientation can be estimated by repeating the cyclic one-dimensional search for each parameter several times.

[0005]

However, the above-described position and orientation recognition method is used when used in a place where a noise component level such as a change in brightness or reflection of light is large, such as in an outdoor environment, or in a non-target. This is difficult to apply in a situation where feature points cannot be stably extracted, such as when the appearance of an object cannot be limited due to overlapping of objects and images. Further, there is a situation in which the application becomes difficult even when the feature points cannot be defined as in the case where the object itself is mainly formed of a curved surface shape. In other words, in the conventional configuration, there are restrictions on the shape of the object, its appearance and position / orientation range, lighting conditions, etc., which has been a practical obstacle. Moreover, in the conventional configuration, since the still image processing is basically used, the motion information of the moving object cannot be estimated, and such a point is also one of the problems.

On the other hand, conventionally, as a moving object recognition device based on video information, feature point extraction of an object is performed, such as those described in Japanese Patent Application Laid-Open Nos. 5-73681 and 5-73682. Although the predicates are known, it is difficult to apply them to curved objects for the same reason as described above.

The present invention has been made in view of the above circumstances, and an object thereof is to stabilize the extraction of feature points of an object under a situation where there are many noise components such as fluctuations in brightness and reflection of light. Even in situations where it is not possible to do so, it is possible to freely generate 3D images of moving objects viewed from different viewpoints,
In addition, such an image having an effect that it is possible to generate such a three-dimensional image without trouble even when the object is mainly configured by a curved surface shape or the situation in which the posture position of the imaging unit is unknown It is to provide a three-dimensional object recognition device based on data.

[0008]

In order to achieve the above-mentioned object, a three-dimensional object recognition apparatus according to the present invention is provided with an image pickup means for photographing an object whose three-dimensional shape is known, and the image pickup means picks up an image. In a device for recognizing a three-dimensional position and orientation of an object by performing moving image processing based on image data of a plurality of consecutive frames in a series image, a two-dimensional motion appearing between screens of the plurality of frames. Moving body area extracting means for extracting a moving body area based on a vector, storage means for storing shape data modeling the three-dimensional structure of the object, and moving body extracted by the moving body area extracting means Spatial quantization means for estimating the position and orientation of the object by searching the position and orientation of the image pickup means for obtaining the two-dimensional shape of the region, and the space quantization means The image estimating means is configured to generate a virtual real image of the object by three-dimensional structural modeling based on the position and orientation of the elephant and the shape data stored in the storage means. 1).

In this case, the spatial quantizing means calculates the error between the virtual real image generated by the image estimating means and the actual observed image by a predefined error evaluation function and evaluates the error. The configuration may be such that the three-dimensional position and orientation with the function taking the minimum value is output as the final estimation result (claim 2).

Further, the spatial quantizing means sets a plurality of search area centers from the length information of the long axis and the short axis of the two-dimensional shape of the moving body area extracted by the moving body area extracting means, A configuration may be adopted in which one position / orientation is finally estimated based on data obtained by parallelly one-dimensionally searching these search areas (claim 3).

Further, a portion of the object in which the feature is easily grasped is set as a feature region, and the image estimation means makes the pixel density other than the feature region coarser than that of the feature region in generating the virtual real image. It is also possible to adopt a configuration in which the above interpolation is performed (claim 4).

The moving body area extracting means may be configured to extract the moving body area in block units using a block matching method and perform labeling.

In this case, the moving body area extracting means may be provided with a function of removing a block containing a noise component from the moving body area extracted in block units.

Further, when the area surrounded by the moving body areas extracted in block units includes an area in which no motion vector exists, the moving body area extracting means labels the blocks in the same manner as the surrounding blocks. function can be configured and a child with a (claim 7).

Further, the moving body region extracting means may be provided with a function of removing a portion corresponding to the shadow of the object from the moving body region extracted in block units by color space division processing. (Claim 8).

The storage means is configured to store the shape data of the object as wire frame data, and the image estimation means attaches real images obtained by photographing the object from a plurality of directions to the wire frame. The virtual real image may be generated by performing such texture mapping (claim 9).

Further, a smoothing process may be performed on a reproduced video including a plurality of virtual real images generated by the final estimated position / orientation information by a moving average according to a time series of estimated positions ( Claim 10).

[0018]

In the apparatus according to the first aspect of the present invention, the moving body region extracting means includes a two-dimensional motion vector (optical) appearing between the screens of a plurality of consecutive frames in the series image taken by the imaging means. The mobile body region is extracted based on the flow). The two-dimensional shape of the moving body region thus extracted has a shape corresponding to the position and orientation of the image pickup unit, and the spatial quantizing unit obtains the two-dimensional shape of the extracted moving body region. The three-dimensional position and orientation of the object is estimated by searching the position and orientation of the image pickup means. The image estimation means is based on the three-dimensional position and orientation of the object estimated by the spatial quantization means as described above and the shape data stored in the storage means, that is, the shape data obtained by modeling the three-dimensional structure of the object. A virtual real image of the object is generated by three-dimensional structural modeling.

As a result, it is possible to freely generate a three-dimensional image of the object viewed from different viewpoints even in a situation where the posture position of the image pickup means is unknown. In this case, in particular, unlike the conventional position and orientation recognition method, it is not necessary to extract feature points by binarization processing, so noise component levels such as brightness fluctuations and light reflections such as in an outdoor environment can be reduced. Even if you use it in a large place, or if you can not limit the appearance of the object due to overlapping of non-object and image,
It becomes possible to generate a three-dimensional image of an object viewed from different viewpoints as described above without any trouble. Even when the object itself is mainly composed of a curved surface, if the three-dimensional structure of the object can be modeled, that is, the shape data modeling the three-dimensional structure is stored in the storage means. As long as the above conditions are satisfied, it is possible to generate a three-dimensional image of the same object as described above without any trouble.

In the apparatus according to the second aspect, the spatial quantizing means calculates an error between the virtual real image generated by the image estimating means and the actual observed image by a predefined error evaluation function, and The three-dimensional position / orientation with the error evaluation function taking the minimum value is output as the final estimation result. As a result, the accuracy of the virtual real image finally generated by the image estimation means can be improved.

In the apparatus according to the third aspect, the spatial quantizing means uses a plurality of search area centers based on the length information of the long axis and the short axis of the two-dimensional shape of the moving body area extracted by the moving body area extracting means. Is set, and one position / orientation is finally estimated based on the data obtained by parallelly one-dimensionally searching these search areas. Therefore, it becomes possible to prevent the local optimum from falling during the search of the target object, and the process itself of extracting the long axis and the short axis from the two-dimensional shape is relatively simple. The processing for setting can be easily performed, and the amount of calculation in the spatial quantization means can be reduced.

In the apparatus according to the fourth aspect, a portion of the object in which the feature is easily grasped visually is set as the feature region, and the image estimating means determines the pixel density other than the feature region in generating the virtual real image. Since the interpolation is performed so as to be coarser than the characteristic region, the amount of calculation in the image estimating means can be reduced.

In the apparatus according to the fifth aspect, the moving body area extracting means extracts the moving body area in block units using the block matching method and performs labeling. In this case, since the block matching method has already been established as a practical technique, it becomes possible to reliably extract the moving body region.

In the apparatus according to the sixth aspect, the moving body region extracting means removes the block including the noise component from the moving body region extracted in block units by the block matching method. As a result, it is possible to improve the extraction accuracy of the moving body region without being adversely affected by the noise caused by the uneven brightness and the noise of the image itself.

In the apparatus according to the seventh aspect, the moving body region extraction means, when the region surrounded by the moving body regions extracted in block units includes a region in which no motion vector exists, surrounds the block. Labeling will be the same as for blocks. As a result, even if there is a block in which the motion vector cannot be detected due to reflection of light from the object, there is no risk that the accuracy of extracting the moving body region will decrease.

In the apparatus according to the eighth aspect, the moving body region extracting means removes the portion corresponding to the shadow of the object from the moving body region extracted in block units by color space division processing. Even in the situation where the shadow of the target object exists, it is possible to accurately extract the moving body region.

In the apparatus according to the ninth aspect, the shape data of the object is stored in the storage means as the wire frame data, and the image estimating means photographs the object from the plural directions with respect to the wire frame. Since the virtual real image is generated by performing texture mapping of pasting the real image, the virtual real image can be generated relatively easily.

In the apparatus according to the tenth aspect, the smoothing process by the moving average according to the time series of the estimated position is performed on the reproduced video including the plurality of virtual real images generated by the final estimated position and orientation information. Since this is performed, the virtual real image between the reproduced images shows a smooth motion, and a high quality object recognition result can be obtained.

[0029]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment in which the present invention is applied to a three-dimensional position recognizing device having a function of estimating the three-dimensional motion of a car from an original image obtained by photographing a car driving scene will be described with reference to the drawings To do.

FIG. 1 shows a basic structure of a three-dimensional position recognition apparatus according to this embodiment by combining functional blocks. First, this structure will be briefly described. That is, in FIG. 1, the A / D conversion unit 1 is, for example, R
A series image captured by a CCD camera 2 (corresponding to an image capturing unit according to the present invention) that generates a GB image signal, that is, a video source that supplies a scene including an object is digitized, and the video after the digitization is digitized. The source is subjected to noise removal and filtering by the preprocessing unit 3. The moving body extraction unit 4 corresponding to the moving body region extraction means in the present invention extracts a moving body region image from the output data (video source) of the preprocessing unit 3, and the feature amount extraction unit 5 uses the preprocessing unit. From the output data of No. 3, the feature amount other than the motion vector (feature point position, region division, edge information, etc.) is extracted.

The structure / feature quantity modeling unit 6 has a function of modeling a three-dimensional structure of an object as a wire frame, and the modeling is performed, for example, by manpower. The structural data is registered (stored) in the model database 7 (corresponding to the storage means in the present invention). The wireframe information registered in the model database 7 is selectively accessed by the object information extracted by the request interpreting unit in the structure / feature quantity modeling unit 6 and stored in the object memory 8. Has become.

The texture mapping unit 9 uses the wireframe data stored in the object memory 8 as a reference,
In addition, the position / orientation information output from the spatial quantizer 10 is used to perform texture mapping of the object on the wire frame, and the image estimator 11 that constitutes the image estimator according to the present invention together with the texture mapper 9 A virtual real image of the object is generated based on such texture mapping. The space quantizer 10 constitutes a space quantizer according to the present invention, and the estimation of the position and orientation of the moving object using the extracted image of the moving body region by the moving body extractor 4 will be described later. In addition to the above, the position and orientation search by collating the estimated image based on the virtual real image by the image estimation unit 11 with the extracted image of the moving body region by the moving body extraction unit 4 is performed as described below. It is designed to be output as optimum position and orientation information. The attribute calculator 12 outputs the motion / color / shape and environment information based on the position / orientation information from the space quantizer 10 and the a priori information, structure information, color information, surrounding environment information, etc. regarding the object. .

The structure / feature quantity modeling unit 6 takes in a request relating to recognition of a target object or environment in accordance with an instruction such as voice, language, or numerical data inputted from the outside, and converts it into an operation mode and data format in the apparatus. Have the function to Further, when registering the unknown object in the model database 7, the structure / feature amount modeling unit 6 performs a classification according to the category, a numerical value input request, a search for an already registered object, and an object memory 8. It also has the function of transferring structure and image attribute data. Further, in the structure / feature quantity modeling unit 6, the input data is managed by interactively teaching the structure data of the unknown object.

The image compression / reproduction system 13 compresses an image data group such as a video source, an environmentally generated virtual image, or a composite image of the both obtained by the image compositing unit 14 and sequentially stores it in the image storage unit 15. The communication interface 16 has a function of taking in compressed image information, model database information, or a processing request command obtained by an information communication network into the device and transmitting data to another device. The D / A converter 17 outputs the video data generated in the device as an analog video image, and the video data can be reproduced on the monitor 18.

FIG. 2 shows the operation contents of the three-dimensional position recognizing apparatus according to this embodiment, which will be described below together with the structure and operation of relevant parts.

The outline of the operation contents shown in FIG. 2 is as follows. A block calculated by the CCD camera 2 at time n (where n is a natural number) and each frame of image at time (n + 1) is read (steps S1 and S2) and calculated based on the image shift amount between these frames. The moving body area is extracted in block units based on each motion vector (moving body area extraction routine S3). Then, the search initial values of the posture and position parameters of the CCD camera 2 are set from the extracted two-dimensional shape of the moving body area and the a priori information obtained in advance, or a plurality of search candidate areas are set. Then, a hierarchical parallel search for quantized spatial points is performed for these candidate areas (search area limiting routine S4). Further, in order to reproduce the image of the object (the automobile in this case) by the model-based encoding, the virtual real image is estimated by texture mapping in the three-dimensional structure modeling routine S5, and the estimated image and the observed image are combined. An operation of estimating the position and orientation when the error evaluation function between the two takes the minimum value, obtaining the position and orientation information obtained in this way for each time n, and calculating the trajectory vector based on this (step S6) repeat.

Specifically, the extraction of the moving body area in the moving body area extraction routine S3 is performed as follows.
The processing content of this routine S3 corresponds to the function of the moving body region extraction means in the present invention.

First, assuming that the object (the automobile in this embodiment) is within the field of view of the CCD camera 2, two consecutive frames obtained from the series image taken by the camera 2 are taken. A two-dimensional motion vector distribution (optical flow) for each block pixel is detected based on the image data. In this case, the distribution of the two-dimensional motion vector for each pixel to be encoded that constitutes a block is detected by interframe processing based on the block matching method adopted in the MPEG moving image compression standard and the like. As a result, an optical flow as shown in FIG. 3A is obtained.

Incidentally, FIG. 3 (a) shows an image of the second frame, A shows an automobile as an object, and arrows show the motion vector of each block (however, in FIG. 3 shows a state in which the number of blocks is reduced in order to prevent the drawing from becoming complicated). In FIG.
Motion vectors generated in other parts are noise components. Further, the block size in the normal block matching method is generally set to about 8 × 8 pixels surrounding the pixel to be encoded, but can be appropriately changed according to the processing target. . For example, when the object is an automobile as in this embodiment, 24 (vertical) ×
The block size is set to 12 (horizontal) pixels.

Then, each block is labeled based on the generation status of the optical flow as shown in FIG. 3A obtained as described above. Note that, in FIG. 3B, the blocks in which the motion vector is detected are shown in a state in which the blocks are divided into hatched bands.

By the way, generally, in the block matching method, the motion vector is not always generated only in the area where the moving object exists due to the unevenness of the brightness in the image, the noise of the image itself, the influence of the reflection of light, and the like. There is a circumstance that inclusion of noise components is unavoidable. Therefore, when extracting the moving body region,
Each process for removing noise components such as

Threshold processing Two methods are prepared as threshold processing, and these methods are selectively used according to the preconditions. Specifically, when it is known that the automobile, which is the object, does not make a sharp turn, and when detecting the approximate position of the object, the first method is used, and in other cases The second method is used for (for example, a configuration that uses only the second method when the calculation speed is sufficiently high).

In the first method, the threshold value processing is used to determine whether or not the vector is a motion vector. That is, the threshold value Vth
Is set and the labeling is performed by regarding the corresponding block as a motion vector generation area only when the magnitude V of the motion vector has a relationship of V ≧ Vth. According to this method, the calculation speed is high. Become.

In the second method, when the error evaluation function between block pixels is calculated when the image shift amount is calculated by the block matching method, when the condition of the following expression (1) is satisfied, the motion vector becomes (0 , 0). However, the equation (1) is used for the nth frame and the (n-1) th frame.
The block range and the search range in the second frame are shown in FIG.
It is premised on being defined by XY coordinates as shown in (a) and (b), and (rij, gij, bij)
Is the RG of the pixel at the position (i, j) in the nth frame
B component, (r'ij, g'ij, b'ij) is the (n'th)
-1) RGB of the pixel at the position (i, j) of the frame
(Xs, ys) and (xe, ye) are the coordinates of the diagonal vertices of the pixel block, and ΔOF is a preset threshold value.

[0045]

[Equation 1]

Specifically, in the case of this embodiment, the block size is 24 × 12 pixels, the width of the region for which matching is calculated is dx = dy = 15 pixels, and the threshold value ΔOF = 2 × 24 ×.
It was set to 12.

... Isolated Point Removal Processing In the labeling information of the motion vector generating block obtained as described above, it is judged that the block existing in isolation is erroneously detected in the shift amount, and the block is detected. The motion vector of is set to (0,0). For example, as shown in FIG. 5, with respect to the isolated point removal processing target block, the motion vectors of adjacent blocks in eight directions are checked, and all of the stationary blocks (the motion vector is (0,
0)), the target block is regarded as an isolated point block and is removed from the image processing target.

Reflection removal treatment The optical flow caused by the reflection and gloss of the metal surface is
By utilizing the property that the flow and direction of the entire surface and the state of time change are often different, the block h that becomes a hole in the moving body region is labeled in the same manner as the surroundings, as shown in FIG. 3B. . In this case, the magnitude of the motion vector of the block h is obtained by the interpolation calculation using the data of the surrounding blocks.

After the above processing is performed, the block area is extracted by labeling. In this case,
For each row in units of blocks, the blocks located at the left and right ends of the block in which the optical flow is generated and the blocks located between them are labeled. In this way, assuming that the moving object is an object (an automobile in this embodiment) continuously distributed in the space, as shown in FIG. A moving object area (a portion surrounded by a thick line) including the shadow is obtained in block units.

After that, the shadow is removed by color space division. That is, generally, in the block matching method, the shadow of the object is also extracted as the moving body region as described above depending on the illumination condition. Therefore, the shadow portion is separated by a well-known color space clustering method, and finally a moving body region as shown in FIG. 3D is extracted. Note that, in this clustering, for example, K-means clustering or LBG method can be considered, but in particular, the property that the shadow part is achromatic in most cases is utilized empirically and used as a seed for the vector of each RGB component. The value is set.

More specifically, when the body color of the object is red, for example, the distribution value of RGB components at the center of the initial cluster at the time of clustering as described above is set to 256 gradations (8 bits), About part (RGB) = (2
55,0,0) and for the shadow part (RGB) =
It will be set to (85, 85, 85).

Further, when the optical flow detection area is extracted as described above, the image of the object may be missing, so that the moving object area is actually extracted one size larger than that. Further, an average motion vector is calculated for the labeling area obtained by performing the noise removal processing as described above.

On the other hand, in the search area limiting routine S4, C
It is determined whether or not the video source captured from the CD camera 2 is the initial frame (that is, n = 1 or not), and if it is not the initial frame, the parameters (γ, θ, φ, The initial search value of r) is set to be the parameter (γ, θ, φ, r) n−1 in the previous frame.

By the way, in this embodiment, it is premised that the object is observed by the CCD camera 2 from all directions. Therefore, as shown in FIG. 6, assuming a dome-shaped space surrounding the object (vehicle A), the appearance when the object is observed from all directions is estimated.
To this end, the camera coordinate system for the object is defined by the position r,
It is represented by six parameters of θ, φ and postures α, β, γ, and the parameter space is quantized as described later to estimate a virtual real image.

In this case, when the video source captured from the CCD camera 2 is the initial frame, it is difficult to limit the search area, and therefore the following processing is executed. That is, FIG. 7 shows the flow of this processing. First, the size of the two-dimensional shape of the moving body region extracted by the moving body region extraction routine S3 is determined by calculation, and as shown in FIG. The tilt angles ψ of the long axis a, the short axis b, and the long axis a are calculated from the two-dimensional shape, and the search area regarding the distance r and the angles θ and φ is limited from the calculated information. In this case, in order to estimate all appearances of the object observed from all directions with a virtual real image, there is a situation that the amount of calculation increases significantly. The appearance is narrowed down, and based on this, a plurality of search candidate areas Q1 to Qi are set as shown in FIG. 8, and one-dimensional searches of these candidate areas are performed in parallel.

In the process of narrowing down the appearance that is a candidate for the search area as described above, the long axis a and the short axis b of the moving body area are selected.
In calculating the inclination angle Ψ of the long axis a, the following first shape determination method and second shape determination method are conceivable.

First Shape Determination Method As shown in FIG. 10, the center position of the vertical coordinate component of the block group BL existing on the leftmost side and the block group existing on the rightmost side with respect to the moving body area extracted in block units. The center positions of the vertical coordinate components of BR are calculated, and the center positions are connected by a straight line. Next, a straight line orthogonal to the straight line is drawn from the midpoint of the straight line, and the intersection of the straight line and the outer circumference of the moving body region is obtained. Of the two line segments thus obtained, the longer one is the major axis Lmax, the shorter one is the minor axis Lmin, and the angle between the major axis and the horizontal axis is ψ0. In this case, the center of gravity of the moving body region is the intersection of the long axis Lmax and the short axis Lmin, but the center of gravity and the image center generally do not coincide.

Second Shape Judging Method As shown in FIG. 11, it is assumed that a large number of straight lines exhibiting an arbitrary angle with respect to the horizontal axis are drawn from the center of gravity of the moving body region extracted in block units. The distance from the center of each block located on the outer periphery of the moving body area with respect to the straight line is calculated, and the optimum angle Ψo is obtained by the least squares method using the sum of the two as an evaluation function. The corresponding line segment is the long axis Lmax. Further, a straight line orthogonal to the long axis is drawn from the center of gravity of the moving body region, and the portion where the straight line is included in the moving body region is defined as the short axis Lmin.

The major axis Lmax, the minor axis Lmin, and the angle Ψ obtained by either of the first and second shape determining methods as described above.
The candidate area of the camera position (r, θ, φ) is narrowed down based on the information of 1. In this case, the search area regarding the distance r and the angles θ and φ is limited based on the information. Thereafter, several tens of candidate values for the search center are set in a dome-shaped space as shown in FIG. 6, and one-dimensional search is performed for each search center in parallel to finally output one position and orientation.

Specifically, the search area is limited by the following method, for example. That is, as shown in FIG. 6, in the coordinate system of the camera with respect to the object (automobile A) represented by the six parameters of position r, θ, φ and postures α, β, γ,

[0061]   α = β = γ = 0 [deg] Fixed with   r = rmin + iΔr (where i is a non-negative integer, r ≦ rmax)   θ = θmin + jΔθ (where j is a non-negative integer, θ ≦ θmax)   φ = φmin + kΔφ (where k is a non-negative integer, φ ≦ φmax)

For each of the parameters r, θ, and φ, a virtual real image of the model of the automobile A is generated, and the long axis Lmax as shown in FIG. 10 or 11 that is the feature of the two-dimensional shape of the virtual real image. (R, θ, φ), short axis Lmin (r,
θ, φ) and tilt angle Ψ (r, θ, φ)
And the second shape determination method. Where rmin, θmin, φmin and rmax, θmax, φma
x is the minimum and maximum search value for each parameter. Thereby, a correspondence table of (r, θ, φ) → (Lmax, Lmin, Ψ) [r, θ, φ] is created in advance. By referring to this correspondence table, from (Lmax, Lmin, Ψ) to (r, θ, φ)
Can be calculated. In this case, in the present embodiment, a correspondence table is created for θmin, θmax, φmin, and φmax for all directions. Also, rmin and rmax are (Lmax
, Lmin, Ψ 0), or based on a rough scale correspondence table, or set appropriately according to the application target.

When the above correspondence table is searched, a set of (r, θ, φ) satisfying the relation of the following expression (2) (ro)
i, θ0i, φ0i). However, i is a natural number (i = 1, 2, ..., N) according to the number N of candidates. Further, in the equation (2), L0max and L0min are the lengths of the major axis and the minor axis calculated by the above-described first shape determining method or the second shape determining method, and ΔLmax, ΔLmin and ΔΨ are threshold values. .

[0064]

[Equation 2]

Next, as shown in FIG. 12, the attitude parameters α and β are obtained from the center of gravity (xG, yG) of the moving body region by the following equation. However, F is the focal length of the CCD camera 2,
xG is the horizontal coordinate of the center of gravity of the moving body area with the image center as the origin, and yG is the vertical coordinate of the center of gravity of the moving body area with the center of the image as the origin.

Α = arctan (yG / F) β = arctan {(− xG / F) × cos α} Regarding the attitude parameter γ, in the case of the present embodiment,
It is assumed that there is almost no deviation in the γ direction in the relationship between the postures of the automobile A, which is the object, and the CCD camera 2, and therefore γ = 0.

Then, thus obtained (roi, θ
0i, φ0i) and (α, β, γ) are candidate values for the position and orientation of the moving body with respect to the CCD camera 2. The number of candidate values N
Is dependent on the values of ΔLmax, ΔLmin, and ΔΨ, but a candidate value is preferentially selected based on an error evaluation function of a shape feature described later from the one having the smallest error, and the processor that controls the arithmetic function of the apparatus of the present embodiment. N is determined according to the computing power and the real-time property.

Here, attitude parameters α and β at the time of observation
Has the problem that it cannot be limited to α = β = 0 at the time of creating the correspondence table, but generally, in the range of the angle of view of the CCD camera 2, Lmax, Lmin, and Ψ0 are not so different, and There is no particular problem if it is considered that the accuracy of the primary estimation, that is, the limitation of the search area, is sufficient. Of course, when the computing capacity has a margin, the values of α and β in the above equation are incorporated to calculate the correspondence table. In the present embodiment, the search width of the one-dimensional search as described above is set for each of the parameters r, θ, φ, α, β, γ, for example, as shown in Table 1 below.

[0069]

[Table 1]

In the three-dimensional structure modeling routine S5, the following processing is performed. In calculating the perspective transformation matrix, the perspective transformation matrix is calculated from the viewpoint of the quantized space point in the dome-shaped parameter space represented by the position and orientation parameters r, θ, φ, α, β, γ of the camera coordinate system. To do.

In estimating a virtual real image using such a perspective transformation matrix, when the object is a car,
For example, as shown in FIG. 13A, a wire frame in which a three-dimensional structure of an object is modeled is created in advance,
By pasting the real images P1 to P4 captured in four directions in the front, rear, left, and right in advance by texture mapping, an arbitrary number of virtual real images at arbitrary positions viewed from an arbitrary viewpoint as shown in FIG. Can be generated. in this case,
If the number of virtual real images is too small, it becomes impossible to generate a visible surface when viewed from all directions, and if it is too large, problems occur in terms of data amount and calculation speed, so consider these conditions. To decide. The ceiling of the object is regarded as a flat plate, and a flat plate of a color sampled from the actual image is attached to the ceiling.

The texture of the object image required when the virtual real image is generated by such a method may be obtained from a previously photographed object image or may be obtained in real time from the observed image. These methods will be individually described below.

(A) Method of Obtaining Texture from Object Image Preliminarily Photographed In this case, the actual image of the object can be photographed under favorable illumination conditions and position / orientation, and the actual size for wireframe preparation can be obtained. Since the measurement can be performed at that time, the quality of the generated virtual real image can be improved. Further, it is possible to attach the wire frame to the obtained image with sufficient time, and there are the following four levels in such attachment.

A method of changing the perspective transformation matrix while interactively operating the computer and pasting on the screen by manual cut-and-try. ... A method of performing perspective transformation by interactively operating a computer, specifying only the extraction and correspondence of feature points, and solving the perspective n-point problem on a computer. A method for obtaining a perspective transformation matrix by automatically solving by a computer from extraction of feature points and determination of correspondence to calculation of perspective n-point problem, and manually inspecting and correcting the result. A method of performing the above without human intervention (a method of inspecting and correcting the calculation result of the perspective transformation matrix only by the autonomous evaluation of the computer).

(B) Method of Obtaining Texture in Real Time from Observation Image In this case, the wire frame must be automatically attached to the observation image. Therefore, first, fitting of the wire frame structure to the extracted moving body area is performed. At the time of fitting,
Ensure that the conditions of are met.

As shown in FIG. 14A, the outer periphery of the wire frame WF should not exceed the moving body area MA. ... As shown in FIG. 14B, the outer circumference of the wire frame WF and the outer circumference of the moving body area MA should not be extremely separated. ... As shown in FIG. 14 (c), the wire frame WF and the long axis in the moving object region MA, the minor axis and the tilt angle is not a large one another Lena Ikoto.

In addition to this, the initial position / orientation may be obtained by a texture model of a virtual real image using a pre-captured image, and an estimated image satisfying these conditions is as shown in FIG. 14 (d). Also, in this case,
The correction of the wire frame fitted in the initial position and orientation will be performed by the autonomous evaluation of the computer.

Although not shown in FIG. 2, in the present embodiment, in order to improve the quality of the virtual real image generated,
Variable density interpolation (sampling) and color adaptation processing in the following estimation image generation are performed, and these will be described below.

(A) Variable Density Interpolation in Estimated Image Generation In this case, a portion of the object in which the feature is easily recognized visually is set as the feature region based on the human sensitivity.
For example, in an automobile A as shown in FIG. 15, headlights,
A number plate, a wheel of a tire, etc. are set as the characteristic regions, and the interpolation density is made variable such that the characteristic regions are fine and the other patch surfaces are coarse. As a result, the amount of calculation is reduced and statistical smoothing is achieved.
In the case of the present embodiment, the interpolation density per patch surface and the interpolation density of the characteristic region in the virtual real image generation are 16 ×.
Although 16 points (or 20 × 20 points) are sampled, in reality, the characteristic region has a smaller area than a general patch surface, so that the characteristic region side is equivalently sampled. The chemical density is high.

(B) Color adaptation processing In this case, in the following case, a color obtained by sampling pixels in the patch surface from another pixel area or an average color of the object area according to the property of the object. Fill with. ... Regions of different colors that exist as isolated points in the actual image for modeling, ... Surfaces that are not observed in the model image.

By the way, in a situation where the object moves in an unknown outdoor environment as in the present embodiment, the degree of overlap between the non-object and the image and the brightness change, and the appearance of the object is changed. Since there are various conditions such as being incapable of limiting, stable feature point extraction becomes difficult, and the risk of falling into so-called local optimization when searching for quantized spatial points of position and orientation parameters increases. In order to avoid such a risk, in the present embodiment, when estimating the position and orientation when the error evaluation function between the estimated image and the observed image takes the minimum value in the three-dimensional structure modeling routine S5, The error evaluation function was improved as follows.

Specifically, the area error, the pixel value error, the two-dimensional position error of the characteristic region, and the evaluation of the shape feature are comprehensively evaluated. First, the basis of the comprehensive evaluation. Each of the factors will be explained individually.

(A) Area error As shown in FIG. 16, the estimated image based on the virtual real image is superimposed on the moving body region, and the error evaluation is calculated by the individual load in the following cases. ... If the number of pixels overlapping the outside of the moving body region is a certain value or more, the estimated image is rejected. If the moving object region is outside the estimated image region, the error area Ea (the area of the region where the moving object region extracted image and the virtual actual image do not overlap) is calculated in pixel units, and the error load wa is increased. In this case, Ea = waΣ (number of pixels) is obtained (wa is about “5”).

(B) Pixel value error In the area where the actual image (observed image) and the estimated image (virtual actual image) overlap, the sum of absolute error of pixel values (or Sum of squared distances). Ev = wvΣ (pixel value error) Specifically, in the present embodiment, calculation was performed using the pixel value error evaluation function of the following expression (3).

[0085]

[Equation 3]

However, the meaning of each symbol in the formula (3) is as follows. P: A set of sample pixels on each patch surface projected on the screen of the virtual real image. N: the number of pixels included in P. r1xy, g1xy, b1xy: RGB component values of the pixel (x, y) in the moving body region extracted from the observed image. r2xy, g2xy, b2xy: RGB component values of the pixel (x, y) sampled in the virtual real image. Wxy: Weighting coefficient depending on the coordinate xy. However, the weighting coefficient Wxy is set in the following three ways.

First case: When the pixel located at (x, y) is on the characteristic region in the virtual real image → w1 Second case: The pixel located at (x, y) is the virtual real image In the case where it is not on the characteristic region and is included in the moving body region extracted from the observed image → w2 Third case: the pixel located at (x, y) is not on the characteristic region in the virtual real image, In addition, if it is not included in the moving body region extracted from the observed image → w3

The setting of such weighting factors produces the following effects. That is, by the setting of the first case, the error is minimized when the portion of the characteristic region of the model overlaps the corresponding portion of the moving body region extracted from the observed image. By the settings of the second case and the third case, the value of the pixel value error Ev becomes large when the virtual real image is outside the moving body region, and when the virtual real image is inside the moving body region. Is the smallest. In this embodiment, the weighting factors w1 to w3 in the pixel value error evaluation function are set, for example, as shown in Table 2 below.

[0089]

[Table 2]

(C) Error in the two-dimensional position of the characteristic region If the two-dimensional position of the characteristic region can be detected by pixel processing, the error Ep in the position of the corresponding characteristic region as shown in FIG.
Is calculated as one of the evaluation functions. For example, if the sum of squared distances is used as an error evaluation scale, the following formula (4) is obtained.

[0091]

[Equation 4]

Here, Nf is the number of target characteristic regions,
uvi and vvi are the feature points (u,
v) Position on coordinates, umi, vmi are positions on (u, v) coordinates of feature points in the feature area of the moving body area.

(D) Evaluation of shape feature The shape feature is evaluated by focusing on the long axis and the short axis of the moving body area, and the function Es for performing the evaluation is obtained by the following equation (5). .

[0094]

[Equation 5]

However, L1max: the length of the long axis calculated from the moving body region extracted from the observed image L1min: the length of the short axis calculated from the moving body region extracted from the observed image Ψ1: the moving body extracted from the observed image Inclination angle L2max of the long axis calculated from the region: Length L2min of the long axis calculated from the virtual real image: Short axis length Ψ2 calculated from the virtual real image: Inclination angle wsi of the long axis calculated from the virtual real image: Weighting factor (i = 1, 2, 3). The value of this weighting coefficient is set as follows, for example. Where Δmax and Δ
min and ΔΨ are threshold values. ws1 = 0 (when 0 ≦ L1max−L2max <Δmax) ws1> 0 (other than the above case) ws2 = 0 (when 0 ≦ L1min−L2min <Δmin) ws2> 0 (other than the above case) ws3 = 0 ( (When the absolute value of Ψ1−Ψ2) is smaller than ΔΨ) ws3> 0 (other than the above case)

By incorporating such a shape feature error evaluation function, it becomes possible to prevent the virtual real image from becoming extremely smaller than the moving body region (the state as shown in FIG. 14B). In the present embodiment, the weighting factors ws1 to ws3 and the thresholds Δmax, Δmin and ΔΨ in the shape feature error evaluation function are set as shown in Table 3 below.

[0097]

[Table 3]

Therefore, the four evaluation functions E described above are used.
In order to obtain the error evaluation function E in which a, Ev, Ep, and Es are integrated, for example, the following linear combination is obtained.

[0099]

[Equation 6]   E = αa · Ea + αv · Ev + αp · Ep + αs · Es (6)

In this case, αa, αv, αp, and αs are arbitrary coefficients, and if these coefficients are set to “0”, the evaluation functions Ea, Ev, Ep, and Es can be easily selected. However, at least αv and αs are preferably set to values other than “0”.

Then, using the error evaluation function E obtained as described above, the one-dimensional search of the quantized spatial point with the plurality of candidate values obtained in the search area limiting routine S4 as the search center is executed. . The order of the parameters of such a one-dimensional search is changed according to the actual application example.
When the object is an automobile as in the present embodiment, FIG.
The order is as shown in 7. One-dimensional search like this
As mentioned earlier, it is performed in parallel for dozens of candidate values of the search center set in the dome-shaped space surrounding the object (vehicle), and the position and orientation with the smallest evaluation function is calculated. Output as the final estimated value. In the case of the present embodiment, the interpolation density per patch plane in the virtual real image generation is, for example, about 16 × 16 points, and the number of quantization point searches may be about 10 times for each parameter.

On the other hand, the search areas after the third frame can be set based on the estimation of the movement of the center of gravity. That is, when the video source of the kth frame (k ≧ 3) is obtained, the moving body region is extracted from each image of the (k−1) th frame and the kth frame. The average (xOF, yOF) of motion vectors for each block is calculated based on the image shift amount between frames at this time, and the search centers of the posture parameters α and β are set as follows.

[0103]   αc [k] = arctan (tan αE [k-1] -yOF / F [k-1])   βc [k] = arctan {(tan βE [k-1] −xOF / F [k-1]) × cosα} here, αE [k-1]: Estimated value of attitude parameter α at time k-1 βE [k-1]: Estimated value of attitude parameter β at time k-1 F [k-1]: Camera focal length at time k-1

It is This means that the amount of movement of the center of gravity of the motion vector (optical flow) within one frame time is used as the temporal change amount (Δα, Δβ) of the optical axis deviation to perform initial estimation of the three-dimensional position of the k-th frame. . on the other hand,
The position parameters (r, θ, φ) are (r [k-1], θ [k-1], φ [which are the estimation results obtained based on the position and orientation search at time k-1. k−1]) is the search center for the k-th frame.

Further, in the initial frame, a one-dimensional search is carried out in parallel for a plurality of search candidate regions. However, since the k-th frame and thereafter can estimate the movement of the center of gravity, the search region can be limited to one. The function is calculated in the same manner as the equation (6), and for example, similar to the means described in the invention of Japanese Patent Application No. 5-247229, a search for each parameter such that the evaluation error value by the above calculation is minimized. It suffices to perform a one-dimensional search for the points independently, and thereby the optimum value of the position and orientation is output for each frame.

By the way, in the image reproducing method as described above, the position / orientation search other than the estimation of the movement of the center of gravity is independently performed for each still image frame, and therefore the continuously reproduced image is an object. The image may be jerky as if the car were traveling on a bumpy road. Therefore, in order to obtain a smooth motion estimation result that is close to the actual traveling state of the automobile, it is sufficient to perform smoothing processing by a moving average such as the following equation on the time series of the estimated position and orientation.

[0107]

[Equation 7]

If the estimated image obtained by the above-described processing is collated with the moving body area obtained from the actual image and the above-described evaluation function and quantization space point search method are used,
For example, the three-dimensional motion information (r, θ, φ, α,
The estimation result of β, γ) can be obtained from a plurality of estimated images. Then, if smoothing processing by moving average is applied to the estimation result of the position and orientation, and the motion of the car is reproduced by the virtual real image by the model-based reproduction,
It is possible to obtain a smooth motion estimation result corresponding to the continuous automobile traveling scene.

In short, according to the configuration of the present embodiment described above, the appearance of the object is virtually determined from the two-dimensional shape of the moving body area extracted based on the two-dimensional motion vector appearing between two temporally consecutive screens. Since it is estimated from a real image, it is possible to freely generate a three-dimensional image of a moving object viewed from different viewpoints based on the same image source even when the position of the camera is unknown. . Particularly in this case, JP-A-6-262568 or JP-A-6-2
Unlike the conventional position / orientation recognition method as disclosed in Japanese Patent No. 58028, since it is not necessary to extract feature points by binarization processing, noise components such as fluctuations in brightness and reflection of light like in an outdoor environment. Even when it is used in a place where the level is large, or when the appearance of the moving object cannot be limited due to the overlap of the non-object and the image, etc., the three-dimensional image of the moving object viewed from different viewpoints as described above. Can be generated without any trouble. Therefore, it can be applied to the follow-up running of automobiles and the remote control technology of robots.

Even when the object itself is mainly composed of a curved surface, if the three-dimensional structure of the object can be modeled, that is, the shape data modeling the three-dimensional structure is used as the model database 7. 3D image of the same object as described above can be generated without any problem if it is stored in.

According to the configuration of this embodiment, it is possible to recognize not only the estimation of the three-dimensional motion of the moving object but also the posture of the object, and the three-dimensional model. If the position and orientation of the moving object is estimated by collating with, the detailed parts included in the object (in the case of a car,
It is also possible to estimate the position and orientation of the mirror position). Furthermore, if the shape and position / orientation of the obstacle are known, it is possible to estimate the three-dimensional movement of the moving object even in a situation where the moving object is partially blocked by the obstacle.

If the object area can be limited by color information or the like, the position and orientation of the stationary object can be detected.
Further, the image can be encoded as an object after recognizing the presence, position / orientation, motion, color, etc. of the object. With this, construction of a 3D image database of the object,
It is also possible to apply the present invention to a model-based encoding / decoding device for transmitting an ultra-low bit rate image for a TV phone using a public telephone line.

The error between the virtual real image estimated as described above and the actual observed image is defined in advance by an error evaluation function,
In particular, the area error, the pixel value error, the error in the two-dimensional position of the feature area, and the shape feature are calculated by an integrated error evaluation function, and the position when the error evaluation function takes the minimum value is calculated. Since the posture is output as the final estimation result, the accuracy of the estimated virtual real image can be improved.

Furthermore, a plurality of search area centers are set from the length information of the long axis and the short axis of the two-dimensional shape of the moving body area extracted as described above, and these search areas are one-dimensionally searched in parallel. Since it is configured to finally estimate one position / orientation based on the data, it becomes possible to prevent a local optimum from falling during the search for an object. Since the process itself for extracting the short axis is relatively simple, the process for setting the center of the search area can be easily performed, and the amount of calculation can be reduced.

[0115] Further, in the case of generating a virtual real image, the above-mentioned characteristics are set by setting a portion (a headlight, a number plate, a wheel of a tire, etc. in the case of an automobile) of an object in which a characteristic can be easily grasped visually. Since the pixel density in areas other than the area is interpolated so as to be coarser than that in the characteristic area, the amount of calculation can be reduced also by this.

At the time of extracting the moving body area, the moving body area is extracted and labeled in block units by the block matching method already established as a practical technique, so that the moving body area can be surely extracted. Become. In this case, blocks that include noise components such as luminance unevenness in the image and noise in the image itself are removed from the moving body region extracted in block units, so that the noise components may be adversely affected. Therefore, the extraction accuracy of the moving body area can be improved.

When the area surrounded by the moving body areas extracted in block units as described above includes an area in which no motion vector exists, the block is labeled in the same manner as the surrounding blocks. Even if there is a block in which the motion vector cannot be detected due to reflection of light from the target object or the like, there is no fear that the extraction accuracy of the moving body region will decrease. Further, since the portion corresponding to the shadow of the target object is removed by the color space division processing from the mobile object region extracted in block units, it is possible to extract the mobile object region even when the shadow of the target object exists. You will be able to do it accurately.

Further, the shape data of the object is stored as wire frame data, and texture mapping is performed by pasting real images obtained by shooting the object from a plurality of directions to the wire frame to generate a virtual real image. Since the virtual real image is generated, the virtual real image can be generated relatively easily.

Further, since the smoothing process by the moving average according to the time series of the estimated position is performed on the reproduced video including the plurality of virtual real images generated by the final estimated position / orientation information, for example, an automobile When reproducing the running scene of, the virtual real image between the reproduced images shows a smooth motion, and a high quality object recognition result can be obtained.

In the above embodiment, in limiting the search area, (r, θ, φ) → (Lmax, Lmin, Ψ)
Although the method of referring to the correspondence table of [r, θ, φ] is adopted,
Instead of this, it is also possible to adopt a method in which the following means are combined.

Limitation of θ and φ: The distance from the object to the viewpoint is set to a certain reference value r 0 (for example, 10 m), and the perspective transformation of the wire frame model of the object is set to (θ, φ). Calculated by scanning the azimuth and using the first shape determination method (see FIG. 10) or the second shape determination method (see FIG. 11),
The major axis a0 and the minor axis b0 are obtained. From this result, a correspondence table between (c0, a0, Ψ0), which is a combination of these ratios, that is, c0 = b0 / a0 and the major axis a0 and its inclination angle Ψ0, and (θ, φ) above. Is calculated.

Next, using this correspondence table, from the ratio of the major axis a and the minor axis b obtained from the observed image, that is, c = b / a and the inclination angle Ψ of the major axis obtained from the observed image as well. , As a shape feature evaluation function, for example

[Equation 8]

If a predetermined criterion, for example, Esc ≦ ΔEsc is satisfied (ΔEsc is a threshold value), θ and φ in the above correspondence table are regarded as search candidate values.

Limitation of r Using the magnitudes of the major axis a0 and the major axis a obtained from the observed image with respect to the candidate values of θ and φ obtained above, a rough estimated value rE of r is obtained from the following equation.

[0125] rE = r0 · (a0 [r0, θ, φ] / a)

The center of gravity of the moving object region in which α and β are limitedly extracted generally does not coincide with the center of the image. Therefore, the values of the center deviations α and β are set, for example, in Japanese Patent Laid-Open No.
-262568 or JP-A-6-258028. However, in practice, if the target object is within the screen, the calculation of the correspondence table will not be hindered even if the calculation is performed with α = β = 0. Therefore, only when the computing capacity has a margin, it is sufficient to calculate the correspondence table dynamically incorporating the values of α and β calculated from the observed image.

[Brief description of drawings]

FIG. 1 is an overall functional block diagram showing an embodiment of the present invention.

FIG. 2 is a flowchart for explaining operation contents.

FIG. 3 is a schematic diagram for explaining a moving body region extracting operation.

FIG. 4 is an XY coordinate diagram for explaining threshold processing when an optical flow is detected by a block matching method.

FIG. 5 is a schematic diagram for explaining a process of removing an isolated block of a moving body region extracted in block units.

FIG. 6 is a schematic diagram for explaining parameters required for a quantized space point search operation.

FIG. 7 is a flowchart showing a process for limiting a search area in an initial frame.

FIG. 8 is a schematic diagram for explaining a search area in an initial frame.

FIG. 9 is a diagram for explaining a method of determining a two-dimensional shape of a moving body region.

FIG. 10 is a diagram for explaining a method of extracting a long axis and a short axis of a moving body region, part 1

FIG. 11 is a diagram for explaining a method of extracting a long axis and a short axis of a moving body region, part 2

FIG. 12 is an XY coordinate diagram for explaining an example of obtaining a posture parameter from the center of gravity of a moving body region.

FIG. 13 is a diagram schematically showing a method of generating a virtual real image by texture mapping.

FIG. 14 is a schematic diagram showing an example of a relationship between a wire frame and a moving body region.

FIG. 15 is a schematic diagram for explaining the position of a characteristic region.

FIG. 16 is a schematic diagram showing the contents of the error evaluation function.

FIG. 17 is a flowchart showing a one-dimensional search method.

[Explanation of symbols]

2 ... CCD camera (imaging means), 4 ... moving body extracting section (moving body area extracting means), 7 ... model database (storage means), 9 ... texture mapping section, 10 ... spatial quantizing section (spatial quantizing means) , 11 ... Image estimating unit (image estimating means).

Continuation of front page (56) Reference JP-A-6-259560 (JP, A) JP-A-6-247246 (JP, A) JP-A-5-143737 (JP, A) JP-A-6-262568 (JP , A) JP-A-6-258028 (JP, A) JP-A-5-73681 (JP, A) JP-A-5-73682 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB) Name) G01B 11/00-11/30 G06T 1/00 G06T 7/00 G06T 7/60

Claims (10)

(57) [Claims] 1. An image pickup means for picking up an object having a known three-dimensional shape, wherein moving image processing is performed based on image data of a plurality of consecutive frames in a series of images picked up by the image pickup means. In the device for recognizing the three-dimensional position and orientation of the object by means of a moving body area extracting means for extracting a moving body area based on a two-dimensional motion vector appearing between screens of the plurality of frames, the object Storage means for storing shape data modeling a three-dimensional structure, and the object by searching the position and orientation of the imaging means for obtaining the two-dimensional shape of the moving body region extracted by the moving body region extracting means. 3D based on the spatial quantization means for estimating the position and orientation of the object, and the position and orientation of the object estimated by the spatial quantization means and the shape data stored in the storage means. Concrete modeled by the object of the virtual real image a three-dimensional object recognition device based on image data comprising the image estimating means for generating a. 2. The spatial quantizing means is provided so as to calculate an error between a virtual real image generated by the image estimating means and an actual observed image by a predefined error evaluation function. The three-dimensional object recognizing device based on image data according to claim 1, wherein the three-dimensional position and orientation in a state where the error evaluation function takes the minimum value is output as a final estimation result. 3. The spatial quantization means sets a plurality of search area centers from the length information of the long axis and the short axis of the two-dimensional shape of the moving body area extracted by the moving body area extracting means, The three-dimensional object recognition device based on image data according to claim 1, wherein one position / orientation is finally estimated based on data obtained by parallelly one-dimensionally searching these search regions. . 4. A feature region is set as a portion of a target object in which the feature is easily grasped visually, and the image estimation means
The three-dimensional object recognition device based on image data according to claim 1, wherein, when generating the virtual real image, interpolation is performed so that a pixel density other than the characteristic region is coarser than that of the characteristic region. 5. The image data according to claim 1, wherein the moving body region extracting means is configured to extract and label a moving body region in block units using a block matching method. Based 3D object recognition system. 6. The moving body region extracting means has a function of removing a block containing a noise component from the moving body region extracted in block units.
A three-dimensional object recognition device based on the described image data. 7. The moving body region extracting means, when a region surrounded by moving body regions extracted on a block-by-block basis includes a region in which no motion vector exists, labels the block in the same manner as surrounding blocks. The three-dimensional object recognition device based on image data according to claim 5, which is provided with a function. 8. The moving body region extracting means has a function of removing a portion corresponding to the shadow of the object from the moving body region extracted in block units by color space division processing. A three-dimensional object recognition device based on the image data according to claim 5. 9. The storage means stores shape data of the target object as wire frame data, and the image estimating means captures real images of the target object taken from a plurality of directions with respect to the wire frame. The three-dimensional object recognition device based on image data according to claim 1, wherein the three-dimensional object recognition device is configured to generate a virtual real image by performing texture mapping of pasting. 10. A smoothing process using a moving average according to a time series of estimated positions is performed on a reproduced video including a plurality of virtual real images generated by final estimated position and orientation information. A three-dimensional object recognition device based on image data according to claim 1.