JPH0528246A - Three-dimensional object recognition device - Google Patents

Abstract

PURPOSE:To accurately generate three-dimensional form data. CONSTITUTION:An arithmetic processing unit 3 includes a characteristic point detection circuit 31 detecting a pair of corresponding characteristic points from two pictures, a characteristic polyhedron generation circuit 32 generating a characteristic polyhedron by considering an error as to a corresponding characteristic point picture element, a characteristic polyhedron synthesis circuit 33 synthesizing the produced characteristic polyhedron a parameter setting error setting circuit 34 setting the various parameters of a stereo camera device and the errors, a view polyhedron generation circuit 35 generating a view polyhedron considering the error based on the picture element area on a projection surface, a view polyhedron synthesis circuit 36 synthesizing the generated view polyhedrons and a three-dimensional form restoration circuit 37 restoring a three-dimensional form based on the synthesized characteristic polyhedron and view polyhedron. Thus, the characteristic polyhedron and the view polyhedron are generated by considering the error in the stereo camera device 1, and the three-dimensional form is restored after the polyhedrons are synthesized.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates generally to a three-dimensional object recognition device, and more particularly to a three-dimensional object data recognition system capable of more accurately generating three-dimensional shape data on the basis of an imaged image in consideration of an error. The present invention relates to an object recognition device.

[0002]

2. Description of the Related Art Conventionally, in the technical field of image processing, 3
A method is known in which a three-dimensional situation or object is imaged using an imaging device, and a three-dimensional situation or object is represented by data based on the obtained image. The three-dimensional object recognition has a wide application field, for example, creation of traveling environment map data of a self-supporting mobile robot and creation of identification data for identifying a three-dimensional object.

To restore a three-dimensional situation or object based on an image, the principle of point projection has been conventionally applied. In point projection, each point of an object placed in a three-dimensional space is projected on a projection surface or screen based on the projection center point. One point on the screen of the captured image corresponds to one point in the three-dimensional space or a straight line connecting the point on the screen and the projection center point of the imaging device.

However, an actual image pickup device cannot realize ideal point projection. For example, in an ideal point projection, the position of the projected point may change continuously, whereas in an actual imaging device, the position of the projected point is a discrete point determined by the position of the pixel. Is represented by a value. In addition to this, in ideal point projection, the focal length and the installed position of the image pickup apparatus can be accurately known, but the actual image pickup apparatus cannot accurately know these parameters. That is, in an actual image pickup device, these parameters are given by estimation. Furthermore, it is pointed out that in an actual image pickup apparatus, there are noises caused by various causes.

It is desirable that the estimation of the parameters of the image pickup device be highly accurate. In the conventional estimation method, a plurality of indexes placed in a three-dimensional space are photographed, and the parameters of the imaging device are estimated from the relationship between the projected position on the screen and the installation position.

In order to estimate the parameters with high accuracy, it is necessary to set and fix the index with high accuracy in the three-dimensional space. In addition to this, in order to save the state at the time of estimation, it is necessary to take measures such as image stabilization and fixing of the imaging device. As a result, the parameter estimation procedure is complicated. Generally, it is desirable that the restoration accuracy of the three-dimensional data is high, while it is practical that the parameters of the image pickup device are low.

A decrease in the estimation accuracy of the parameters of the image pickup device causes a decrease in the accuracy of the restoration result. In order to prevent a decrease in the accuracy of the restoration result, it is effective to repeatedly image the same object from different positions in different directions. Each of the restoration results is obtained based on each imaged image,
A method of improving the accuracy of the entire restoration result from these intersections can be considered. There are some problems in the method of integrating the restoration results.

For example, when the image to be used has noise, the restoration result derived from the noise and the restoration result of the actual object are obtained in a mixed manner. In principle, the two cannot be distinguished in the restored shape. This means that the restored shape that does not intersect is not limited to the restored object derived from noise. When a real image in which noise is potentially present is targeted, noise may remain in the final restoration result, and the correct result may be lost.

Even if there is no noise in the image, 3
When three-dimensional data reconstructed from the obtained images are superposed on a single three-dimensional map by sequentially capturing images while moving over a wide range in a three-dimensional space, in such a case, the field of view of each image is limited. In a situation where the fields of view also do not intersect spatially, there is no intersection.

In addition, due to the limitation of the image pickup device, there are cases where the image pickup device is sequentially moved in a three-dimensional space from a plurality of different directions and positions, and the images are processed in the image pickup order. There, the shapes are restored in the order of imaging, and the intersection determination of the restoration results is performed. When a method is adopted in which only the restored results that intersect are sequentially arranged on the 3D map, the result of the 3D map may be different depending on the order of intersection determination of the restored results (Set Membership A
proch to the Propagation
of Uncertain Geometric I
nformation, A.N. Sabater, F.M. Th
mas, Proc. of the 1991 IEEE
E Int. Conf. on Robotic an
d Automation, Apr. 1991). This point will be briefly described below with reference to FIG.

FIG. 17 is a schematic diagram showing that the restoration result is different depending on the difference in the order of intersection determination in the conventional technique. Referring to FIG. 17, here, the same subject is imaged from four different positions P1 to P4, and an image I
Suppose that 1 to 14 are obtained. In the first processing order, the shape 61 is restored based on the images I1 and I2. The shape 62 is restored from the shape 61 and the image I3. The shape 63 is restored from the shape 62 and the image I4. Similarly, in the second processing order, images I3 and I4
The shape 64 is restored based on Shape 64 and image I2
The shape 65 is restored from and. The shape 66 is restored from the shape 65 and the image I1.

Therefore, although the two shapes 63 and 66 are restored from the processing according to the first and second processing orders, these shapes 63 are restored by the conventional cross processing method.
And 66 did not match. In other words, the restored shape is different depending on the processing order, and an error based on the processing order different from the error based on the parameter occurs.

[0013]

FIG. 16 is a schematic diagram showing a region in a three-dimensional space where points corresponding to pixels on the projection plane (or screen) may exist. Referring to FIG.
In this figure, point projections from two positions P1 and P2 are assumed. In the point projection from the position P1, the projection area from the point projection center point FR of one pixel PR on the screen IR is shown. In the point projection from the position P2, the projection area from the point projection center point FL of the pixel PL on the screen IL is shown.
By finding the overlapping part of the two projected areas, the intersection point O
A quadrangle (a hexahedron in the three-dimensional space) consisting of 1, O2, O3 and O4 is obtained.

Therefore, based on the two point projections, it is predicted that the points corresponding to the pixels PR and PL are present in the hexahedron determined by the intersection points O1 to O4 in the actual three-dimensional space. However, in the conventional method, estimation using an ellipsoid or a normal distribution is performed instead of the above-mentioned hexahedron (Geometrical).
Fusion Method for Multi-
Sensor Ronotic Systems, Y.M.
Nakamura, Y .; Xu, Proc. ofthe
1991 IEEE Int. Conf. on Rob
otice and Automation, Apr. 1
991).

Therefore, as shown in FIG. 16, the curve E
An ellipsoid in three-dimensional space as indicated by 1 or E2 is obtained as a possible region of pixels PR and PL. However, these ellipsoids E1 and E
2 has a different shape compared to the hexahedron defined by the intersection points O1 to O4. That is, the ellipsoid E1
Is obtained by inscribed in the above-mentioned hexahedron,
A region where the points corresponding to the pixels PR and PL may exist in the three-dimensional space is left (hatched portion in the figure).

On the other hand, the ellipsoid E2 is obtained as the one circumscribing the above-mentioned hexahedron, and therefore includes the entire area of the hexahedron, but in reality there are areas where points corresponding to pixels cannot exist. Contains.

In reality, the position data of the projection center point,
That is, the focal length data often contains an error. For example, the projection center point FL at the position P2
Exists in FL ', the projection area of the point corresponding to the pixel PL changes as shown by the dotted line in the figure. Therefore, the hexahedral region defined by the intersections O1 'to O4' is obtained as a region where the points corresponding to the pixels PR and PL can exist. Therefore, when the error included in the position data of the projection center point is also taken into consideration, it becomes necessary to estimate the ellipsoid E3 shown in FIG.

The ellipsoids E1, E2 and E shown in FIG.
As can be seen by comparing 3 and the hexahedron defined by the intersections O1 to O4, the regions in which it is estimated that the points corresponding to the pixels PR and PL can exist in the three-dimensional space are different. Therefore, an area in which a point corresponding to a pixel may exist cannot be accurately and efficiently expressed in a three-dimensional space, and accurate three-dimensional data of a subject cannot be obtained.

The present invention has been made to solve the above problems, and in a three-dimensional object recognition apparatus,
It is an object of the present invention to more accurately generate three-dimensional shape data based on a captured image in consideration of an error in image capturing.

[0020]

A three-dimensional object recognition apparatus according to the present invention is an image pickup means for picking up an image of the same three-dimensional object from a plurality of first different positions to generate a first plurality of image signals. And an error setting unit that sets error data that defines an error that the image capturing unit has, and a third plurality of characteristic points that correspond between the first plurality of images in response to the first plurality of image signals. A feature polyhedron for defining a region in which a corresponding feature point detected by the feature point detecting unit can exist in a three-dimensional space in response to the feature point detecting unit for detecting and the error data set by the error setting unit. A characteristic polyhedron generating means for generating data, and a visual field polyhedron for defining an area of a visual field that can be commonly imaged by the imaging means from a first plurality of different positions based on the width of the imaging pixel area on the imaging surface. data Including a field polyhedron generating means generate, in response to the characteristic polyhedron data and the field-polyhedron data and the three-dimensional shape data generating means for generating three-dimensional shape data of the object.

[0021]

In the three-dimensional object recognizing device according to the present invention, the object 3 is detected from the feature polyhedron data and the field polyhedron data.
Since the three-dimensional shape data is generated, more accurate three-dimensional shape data can be obtained.

[0022]

FIG. 2 is a hardware block diagram of a three-dimensional object recognition apparatus showing an embodiment of the present invention. With reference to FIG. 2, this three-dimensional object recognition device is a stereo camera device 1 including two left and right imaging cameras 11 and 12.
And an image memory 2R for storing the image data IR on the right side
And an image memory 2L for storing the left image data IL
An arithmetic processing unit 3 for performing arithmetic processing based on the image data IR and IL; a three-dimensional map memory 4 for storing the three-dimensional data obtained by the arithmetic processing; and a CRT 5 for displaying a three-dimensional shape if necessary. Including. The stereo camera device 1 includes an A / D converter 13 that converts a video signal captured by the camera 11 into a digital signal, and a camera 12
A to convert the video signal captured by
/ D converter 14 is included.

FIG. 1 is a block diagram of the arithmetic processing unit 3 shown in FIG. With reference to FIG. 1, the arithmetic processing unit 3 is
Image data I captured by the stereo camera device 1
A feature point detection circuit 31 that detects a feature point in an image based on R and IL; a feature polyhedron generation circuit 32 that generates a feature polyhedron indicating an area in which a pixel indicating the feature point can exist in a three-dimensional space; A feature polyhedron integration circuit 33 that integrates the generated feature polyhedrons, a circuit 34 that sets various parameters in the imaging by the stereo camera device 1 and their errors, and a visual field polyhedron determined by an imaging pixel area on the imaging surface. A visual field polyhedron generating circuit 35, a visual field polyhedron integrating circuit 36 that integrates the generated visual field polyhedrons, and a three-dimensional shape restoring circuit 37 that restores the three-dimensional shape of the subject based on the integrated feature polyhedron and visual field polyhedron. Including and

FIG. 3 is a schematic diagram for briefly explaining the operation of the three-dimensional object recognition device shown in FIG. To briefly describe the operation, the stereo camera device 1 is moved to the n positions P1 to Pn, and the subject is imaged at each position. Therefore, the images IL and IR are obtained from the left and right cameras by, for example, imaging at the position P1. A feature polyhedron PG1 is generated by calculation based on the left and right images IL and IR. In addition to this, the viewing polyhedron VG1 is based on the size of the image pickup device area on the image pickup surface.
Is obtained by calculation. Similar polynomials PG2 to PGn and visual field polyhedrons VG2 to VGn are obtained by performing similar processing on images obtained at other positions P2 to Pn.

By subjecting the characteristic polyhedron data PG1 to PGn and the visual field polyhedron data VG1 to VGn obtained by the above processing to arithmetic processing, three-dimensional shape data of the object is generated. The above description is for explaining the concept of processing in the three-dimensional object recognition device.
It does not indicate the order of processing. That is, the feature polyhedron data and the visual field polyhedron data are cumulatively processed in the embodiment, and the parallel processing is not necessarily performed as shown in FIG.

4 and 5 are flowcharts of the processing in the arithmetic processing unit shown in FIG. 1 and 4
The processing of the three-dimensional object recognition device will be described below with reference to FIG.

First, in step 41, parameter setting and error setting are performed. These settings are made using an input device such as a keyboard.

FIG. 7 is a parameter diagram showing an example of parameters in the stereo camera device 1. With reference to FIG. 7, a three-dimensional space is defined by the X, Y and Z axes.
An example of parameters of the stereo camera device 1 placed at the i-th position Pi is shown. The camera 11 on the right side has a focal point FR as a projection center point in point projection. Focus FR
The projection screen IR is placed at a position away from the focal length fR. Similarly, the left camera 12 has a focal point FL as a projection center. The projection screen IL is placed at a position away from the focus FL by the focal length fL. Two cameras 11 and 1
The distance between the two central axes is given as the distance b. Further, the error α for the position Pi is given.

The above parameters are only examples, and further, the left and right swing angles of the stereo camera device 1, the sizes (width) of the projection screens IR and IL of the projection screens IR and IL of the cameras 11 and 12, Other parameters and their associated errors are set in step 41 shown in FIG. Note that, as shown in FIG. 7, it is pointed out that the position of the stereo camera device 1 is moved with rotation. In addition, other parameters can be changed as needed.

In step 42, the position Pi of the stereo camera device 1 is set. For example, as shown in FIG. 3, the stereo camera device 1 is placed at the position P1 and the subject is imaged by the two cameras 11 and 12 (step 43). Therefore, the image data IR and IL are obtained.

In step 44, the feature points in the image are detected based on the image data IR and IL, and the corresponding feature point pairs are detected. More detailed processing in this step is shown in the flowchart of FIG.

Referring to FIG. 6, in step 441, a change in shading is detected for each of images IR and IL by image processing. Next, in step 442, the image I
A window is set for each of R and IL in the vicinity of the gradation change point.

In step 443, the images IR, IL
In between, window pairs with similar shade changes are detected. Further, in step 444, the maximum grayscale similarity in each detected window is recognized as a corresponding feature point.

It is pointed out that the above-described feature point detection processing is an example, and that it is possible to apply other conventionally known feature point detection processing.

As another method, for example, the feature point detection processing is performed by Moravec Operato for each image.
This is performed by image processing using r. As a result of the processing, the feature points are detected and the position of each feature point on the screen is determined. Corresponding point detection is performed by referring to the above processing result and the image data before the above processing and using a template matching method. Based on the position of the feature point on one screen, the surrounding area is cut out from the original image to create a small image (template). A template matching technique is applied to detect the most similar part of the template from the opposite image. As a result, the association of the feature points is completed.

Next, in step 45 shown in FIG. 4, one feature point pixel pair PRj, PLj is selected.
Then, in steps 46 and 47, the focal length f1
And f2 are respectively selected, and the feature polyhedron PGj is generated in step 48. These step 4
Details of the processing at 6, 47 and 48 are described below with reference to FIGS. 8 and 9.

FIG. 8 is a schematic diagram showing the principle of generating a characteristic polyhedron. Referring to FIG. 8, a pixel P on the left screen IL
L is projected in the three-dimensional space with the focus FL as the projection center. On the other hand, the pixel PR on the right projection screen IR is projected toward the three-dimensional space with the focus FR as the projection center. The respective vertices of the pixel PL on the screen IL are indicated by L1, L2, L3 and L4. Each vertex of the pixel PR on the screen IR is indicated by R1, R2, R3 and R4. Therefore, a region where points corresponding to the pixels PL and PR can exist in the three-dimensional space is obtained as the characteristic polyhedron PG having six faces. That is, the feature polyhedron PG is
As overlapping projection areas obtained by left and right point projection,
It is defined by the intersections O1 to O8. Features polyhedron PG
When the parameter of the stereo camera system and the position of the pixel change due to an error, the direction and position of the corresponding projection line change, and therefore the feature polyhedron PG moves and deforms.

Although the true value of the stereo camera system parameter and the position of the feature point pair is unknown, the eight projection lines are set to the error range of each parameter by taking into account the error in setting the existence range of the true value. Moves in the three-dimensional space according to. As a result, the movement and deformation range of the hexahedron can be determined in consideration of the parameter error.

FIG. 9 is a schematic diagram showing the generation of the characteristic polyhedron when the error of the focal length is taken into consideration. When the error of the focal length is taken into consideration, here, as shown in FIG. 9, the error of the focal length is taken as a change in the position of the projection screen for convenience (the projection center is fixed as point O). Figure 9
(A) shows a side view of the generated feature polyhedron, while
FIG. 9B shows a plan view of the generated feature polyhedron.

In FIG. 9, it is assumed that the projection screens of the left and right cameras are placed at the focal lengths f1 and f2, taking into account the set error of the focal length. That is, as can be seen from FIG. 9 (B), the projection screens IR1 and IL are located at a distance f1 from the projection center points OR and OL.
1 is placed. Further, projection screens IR2 and IL2 are placed at positions f2 away from projection center points OR and OL. The distances f1 and f2 are determined based on the error regarding the focal length set in step 41. That is, the distance f1 is the minimum focal length when the error is taken into consideration, while the distance f2 is the maximum focal length when the error is taken into consideration.

In FIG. 9, the Y axis indicates the height direction and the Z axis indicates the depth direction. The projection center points OL and OR of the left and right cameras are placed on the X axis, they do not converge, and the projection axes CL and CR of each camera are Z.
Assume parallel to the axis. In addition, the pixel shape is assumed to be rectangular. The position of the corresponding pixel is Y
The positions in the directions are the same on both screens, and the coordinates of the upper and lower ends of the pixels are y1 and y1 + 1. The coordinates of the left and right ends of the pixels on the right screen are xr and xr + 1, and the coordinates of the left and right ends of the pixels on the left screen are xl and xl + 1.

Next, with reference to FIGS. 9 (A) and 9 (B), a process for generating a feature polyhedron in which an error is taken into consideration will be described. A projection line connecting the apex of each pixel and the projection center points OL and OR represents the ridgeline of the feature polyhedron. Figure 9
In (B), the projection lines connecting the projection center point OL of the left camera and the projection positions xl and xl + 1 at the focal length f1 are defined as L1 and L3, respectively, and the projection lines connecting the projection positions xl and xl + 1 at the focal length f2 are defined. L2 and L4 respectively
And By taking the error into account, the focal length is f
Since it moves from 1 to f2, the projection line L1 moves to L2 and the projection line L3 moves to L4. On the other hand, the projection position xr, xr + at the projection center point OR of the right camera and the focal length f1
Let R1 and R3 be the projection lines that connect with 1 and R2 and R4 that are the projection lines that connect with the projection positions xr and xr + 1 at the focal length f2. As the focal length moves from f1 to f2, the projection line R1 goes to R2 and the projection line R3 goes to R2.
Move to 4.

Further, as shown in FIG. 9A, the projection position y at the projection center point O of the camera and the focal length f1.
Projection lines connecting 1 and yl + 1 are defined as P1 and P3, respectively, and projection lines connecting projection positions yl and yl + 1 at the focal length f2 are defined as P2 and P4, respectively. Focal length is f
As it moves from 1 to f2, the projection line P1 changes its position to P2, and the projection line P3 changes its position to P4. As a result, the focal length f
The hexahedron restored from the pixel corresponding to the case of 1 is the intersection A1 of the projection lines L3 and R1 and the projection lines L1 and R1.
B1 of the projection lines, C1 of the projection lines L1 and R3, and D1 of the projection lines L3 and R1. On the other hand, in the side view, the hexahedron is the point A11 on the projection line P1.
, A point B11 on the projection line P1 and a point C1 on the projection line P1
1, a point C12 on the projection line P3, and a point B on the projection line P3
12 and a point A12 on the projection line P3.

The hexahedron restored by the pixels on the projection screen placed at the distance f2 is the projection line L4 in FIG. 9B.
And an intersection A2 of R2, an intersection B2 of projection lines L2 and R2, an intersection C2 of projection lines L2 and R4, and a projection line L
4 and the intersection D2 of R4. Further, in FIG. 9A, the hexahedron is a point A21 on the projection line P2,
A point B21 on the projection line P2 and a point C21 on the projection line P2
And a point C22 on the projection line P4 and a point B2 on the projection line P4.
2 and a point A22 on the projection line P4.

Taking into account the error in the focal length,
The polyhedron based on the stereo camera system moves / deforms in a range sandwiching the above two hexahedrons. A polyhedron circumscribing these two hexahedrons is generated as a characteristic polyhedron PGj in step 48 shown in FIG. That is, the feature polyhedron PG
As j, a dodecahedron consisting of the surfaces of the above two hexahedra and the surface stretched by the ridgeline connecting the corresponding vertices is determined.

In FIG. 9B, the feature polyhedron PGj is
Surface A1B1C1D1, Surface A1A2D2D1, Surface A1
A2B2B1, surface A2B2C2D2, and surface C1D1D
2C2 and surface B1C1C2B2. Therefore,
The feature polyhedron PGj has these six faces and the line segment A1B1.
, Line segment B1B2, line segment B2C2, and line segment C2D2
, And a line segment D2D1 and a line segment D1A1 and has six faces perpendicular to the XZ plane.

In the above description, the feature polyhedron PGj is generated in consideration of the error of the focal length, but it is also possible to generate the feature polyhedron by a similar method in consideration of the errors of other parameters. be pointed out. In step 49 shown in FIG. 4, the feature polyhedron integration process is performed. The generated feature polyhedrons PGj are integrated in the feature polyhedron integration circuit 33 shown in FIG.

In step 49, the feature polyhedron generated by the feature polyhedron generating circuit 32 is placed on the three-dimensional map. In the feature polyhedron generating circuit 32, the feature polyhedron is generated in consideration of not only the error of the focal length but also the error of other parameters. The generated feature polyhedra are cumulatively generated. In the example of the process shown in FIG. 4, only the feature polyhedron taking into account the error of the focal length is described, and therefore, only the integration of the feature polyhedron PGj based on it is described here. The polyhedral integration processing will be described in detail later.

Referring to FIG. 5, in step 50, all feature point pixel pairs PRj, PLj (j = 1, 2, ...).
It is determined whether or not steps 45 to 49 have been performed for m). When the process is not completed, the process returns to step 45, another feature point pixel pair is selected, and the feature polyhedron generation and the cumulative integration process of the generated polyhedra are repeated (steps 46 to 49). As a result, feature polyhedra are generated for all m feature point pixel pairs and their integration is performed. Finally, the integrated feature polyhedron PG is cumulatively obtained.

In step 51, the view polyhedron VGi at the i-th position Pi of the stereo camera device 1 is generated. This generation processing is performed in the visual field polyhedron generation circuit 35 shown in FIG.

FIG. 10 is a schematic diagram showing the principle of generating a visual field polyhedron. FIG. 10 (A) shows a side view of the viewing polyhedron of the stereo camera system, and FIG. 10 (B) shows its plan view. The Y axis indicates the height direction, and the Z axis indicates the depth direction.
The projection center points OL and OR of the left and right cameras are set on the X axis so that they do not converge and the projection axes CL and CR of the cameras are arranged parallel to the Z axis. In FIG. 10A, the projection screen is indicated by IP, in FIG. 10B, the left screen is indicated by IL and the right screen is indicated by IR. The projection screen here corresponds to an area in which an image pickup element involved in image pickup is arranged. Two
With the focal length of one camera being f, the two screens are placed at a distance f from the XY plane in parallel. Figure 10
In (A), the upper and lower ends of the screen IP are set to I5 and I6. In FIG. 10B, the left and right edges of the left screen IL are I1 and I2, and the left and right edges of the right screen IR are I3 and I4.
And

The boundary surface of the entire field of view of the stereo camera system is the projection center point O of each camera shown in FIGS. 10 (A) and 10 (B).
The projection plane connects L, OR, and O with each end of the screen. In FIG. 10A, the field of view of the screen is a range from a projection plane P1 connecting I5 and O to a projection plane P2 connecting I6 and O. In FIG. 10B, the field of view of the left camera is I2 and O from the projection plane PL1 that connects I1 and OL.
It is a range up to the projection plane PL2 connecting with L. The field of view of the right camera is I from the projection plane PR1 connecting I3 and OL.
It is a range up to the projection plane PR2 connecting 4 and OL.

The full field of view of the stereo camera is the part where the fields of view of both screens of the two cameras intersect. Figure 10
The relevant part in (B) is a part in front of the intersection P between the projection planes PL2 and PR1 and sandwiched between the projection planes PL2 and PR1. In FIG. 10 (A), it is the front surface of the line segment EF and the portion sandwiched by the projection planes P1 and P2.

When the parameters of the stereo camera system fluctuate due to an error, the position and orientation of the projection plane forming the polyhedron of the visual field fluctuate, so that the visual field polyhedron moves / deforms accordingly. Although the true values of the parameters of the stereo camera system are unknown, the existence area of the viewing polyhedron is determined as follows in consideration of the error of the parameters. In the following example, the focal length is selected as an example of the parameter, and the generation of the visual field polyhedron in consideration of the error will be described.

FIG. 11 is a schematic diagram showing the generation of the visual field polyhedron when the error of the focal length is taken into consideration. 11
FIG. 11A is a side view of the visual field polyhedron of the stereo camera system in which variations due to parameter errors are expected, and FIG.
(B) shows the top view. Y-axis shows the height direction,
The Z axis indicates the depth direction. Projection center point O of two cameras
L and OR are set on the X axis without congestion, and the projection axes CL and CR of each camera are set parallel to the Z axis.

In FIG. 11A, the projection screen is shown by IP, and in FIG. 11B, the left screen is IL and the right screen is IR.
, The two screens are arranged parallel to the XY plane. In FIG. 11A, the upper and lower edges of the screen placed at the focal length f1 are I51 and I61, respectively, and the focal length f
The upper and lower ends of the screen placed at 2 are designated as I52 and I62, respectively. In FIG. 11B, upper and lower ends of the left screen placed at the focal length f1 are I11 and I21, respectively, and upper and lower ends of the right screen are I31 and I41, respectively. The upper and lower ends of the left screen placed at the focal length f2 are I12 and I22, and the upper and lower ends of the right screen are I32 and I42.

When the size of the projection screen is constant, if the focal length of the stereo camera system increases, the total field of view of the camera becomes narrower. In FIG. 11A, the focal length of the camera is f
When 1, the field of view of the camera is in a range from a projection plane P1 connecting the projection center points O and I51 to a projection plane P2 connecting the projection center points O and I61. When the focal length changes to f2, the range changes from the projection plane P3 connecting the projection center points O and I52 to the projection plane P4 connecting the projection center points O and I62.

In FIG. 11B, the focal length of the camera is f
When 1, the total field of view of the left camera is the projection center points OL and I1.
It is the range from the projection plane L1 connecting 1 to the projection plane L2 connecting the projection center point OL and I21. When the focal length changes to f2, the projection plane L3 connecting the projection center point OL and I12
To the projection plane L4 connecting the projection center point OL and I22. The entire field of view of the right camera is the projection center point OR
And I31 from the projection plane R1 connecting the projection center points OR and I
41 is a range up to the projection plane R2 connecting with 41. When the focal length changes to f2, the projection plane R3 connecting the projection center points OR and I32 to the projection plane R connecting the projection center points OR and I42.
It changes to the range up to 4.

If the focal lengths of the two cameras are f1, then 2
FIG. 11B shows an intersection of all fields of view of the two cameras (full field of view of the stereo camera system).
2, which is in front of the intersection P3 with the projection planes R1 and L2.
It is the part sandwiched between and. In FIG. 11A, the relevant portion is a front surface of the line segment AB and is a portion sandwiched by the projection planes P1 and P2. In the case where the focal lengths of the two cameras are f2, the intersection of the total fields of view of the two cameras (the total field of view of the stereo camera system) is shown in FIG. 11B, and is in front of the intersection P4 between the projection planes R3 and L4. And is a portion sandwiched by the projection planes R3 and L4. In FIG. 11 (A), the relevant portion is a portion in front of the line segment CD and sandwiched by the projection planes P3 and P4. As the focal length of each camera increases, the total field of view of the stereo camera system becomes narrower.

Considering the error of the focal length, FIG.
As shown in, the position of the projection screen changes equivalently in the range from f1 to f2. Due to this error, the visual field polyhedron of the stereo camera system moves / deforms within a range sandwiching the above two pentahedrons. Therefore, no matter how the entire field of view is deformed due to error in these ranges,
The part that is always covered is generated as a visual field polyhedron that takes into account the parameter error. Here, it corresponds to the entire visual field V of the stereo camera system with the focal length f2.

In the above example, the generation of the visual field polyhedron when the error of the focal length is taken into consideration has been described. However, even when the error of the other parameters is taken into consideration, the visual field polyhedron can be similarly generated. it can. That is, the visual field polyhedron is generated in consideration of the error of each parameter.

Next, in step 52 shown in FIG. 5, a field polyhedron integration process is performed. The integration processing of the visual field polyhedron is performed in the visual field polyhedron integration circuit 36 shown in FIG. As described above, the visual field polyhedron generating circuit 3
In 5, since the visual field polyhedron is generated in consideration of the errors of various parameters, the integrated processing of all the generated visual field polyhedrons is performed. In the example of the processing shown in FIG. 5, since the visual field polyhedron is generated in consideration of the error of the focal length, the cumulative integration processing of the visual field polyhedrons generated in consideration of the error of the focal distance is performed. .. less than,
An example of the integration process in steps 49 and 52 will be briefly described.

In the feature polyhedron integration circuit 33 and the visual field polyhedron integration circuit 36 shown in FIG. 1, the number of polyhedra to be processed is large, and in addition, the number of faces of each polyhedron is large. Therefore, it is difficult to determine intersections between polyhedrons, and as a result, not only time is required for processing, but also a huge memory capacity is required to store the processing results. In order to avoid such a practical problem, the following 8-branch tree description method is used in the polyhedral integration processing. For more information on 8-branch tree descriptions, see the Oct-trees and therir use.
in presenting three-dim
mental objects, C.I. L. Jack
ins, S.S. L. Tanimoto, Comput. G
graphics Image Process
g. , Vo 1.14, 1980), which will now be briefly described with reference to FIGS. By using the 8-branch tree description,
It is possible to efficiently describe the combination of hexahedra and the three-dimensional arrangement in the three-dimensional map. Management of the number of intersections of polyhedra
A numerical value is given to each hexahedron forming an 8-branch tree in the three-dimensional map, and this is realized by recording and updating the numerical values.

If one large hexahedron covering the entire three-dimensional map is divided into two along the X, Y and Z axes, it can be divided into eight hexahedrons. The position of the 8-divided hexahedron in the three-dimensional space can be restored by referring to the relative positional relationship between the original hexahedron and the 8-divided hexahedron. From the relationship between the recursive division of a hexahedron and the relative positions of the hexahedron before division and the small hexahedron after division, a solid that occupies a three-dimensional space with an arbitrary shape in an arbitrary position in a three-dimensional map It can be expressed as a three-dimensional array of large and small hexahedra.

When the feature polyhedron and the visual field polyhedron are expressed as a three-dimensional array of large and small hexahedra, if the numerical value of the corresponding polyhedron is always incremented by 1, the hexahedron in the three-dimensional map after the processing of all the polyhedra is completed. The numerical value of has the number of intersections of each polyhedron.

Processing such as recursive division of a hexahedron and updating of numerical values of each hexahedron is recursively determined by the inclusion relation between the polyhedron to be placed and the hexahedron in the three-dimensional map. This will be briefly described with reference to the drawings.

12 to 15 are schematic diagrams showing examples of the relationship between two polyhedra in the 8-branch tree description process. FIG. 12 shows three polyhedrons P to be processed.
It is shown that the hexahedron C in the three-dimensional map exists in a completely included state. FIG. 13 shows a state in which the polyhedron P to be processed and the hexahedron C are completely separated. FIG. 14 shows a state in which the polyhedron P to be processed and the hexahedron C partially intersect.

In the case of FIG. 12, the count value of the corresponding hexahedron C is incremented by 1, and then the processing of the polyhedron P to the hexahedron C is stopped.

In the case of FIG. 13, the count value of the corresponding hexahedron C is stored as it is. After that, polyhedron P
The process for the hexahedron C targeted for is stopped.

The case of FIG. 14 corresponds to the case where neither FIG. 12 nor FIG. In the case of 13, the corresponding hexahedron C is divided into eight (FIG. 15). The count values of all the small hexahedrons divided into eight are the count values of the hexahedron C before division. After that, the inclusion relation with the polyhedron P is obtained for each of all the hexahedrons, and the process according to the inclusion result is repeatedly executed.

Therefore, if the count value of the hexahedron is initially set to 0, the large and small hexahedrons existing at the time when the above-described inclusion determination processing for all polyhedrons are completed are Each of them has a count value corresponding to the number of intersections obtained by determining the intersections of all the hexahedra arranged on the three-dimensional map. In the integration process for the feature polyhedron, this process is repeated for the feature polyhedron, and thus the count value of the hexahedron corresponds to the number of intersections of the feature polyhedron. In the same manner, by repeating the above process for the visual field polyhedron, the count value of the hexahedron indicates the number of intersections of the visual field polyhedron. As a result, the number of intersections in the three-dimensional space is obtained for each of the feature polyhedron and the view polyhedron.

In step 53 shown in FIG.
It is determined whether or not the above process is performed for all the positions P1 to Pn shown in FIG. When the processing is not completed for all the positions, the process returns to step 42,
The above process is repeated. After the above processing is performed for all positions, the processing proceeds to step 54.

In step 54, the three-dimensional shape is restored based on the data obtained by the above processing. That is, the intersection information and the number of intersections of the three-dimensional map and the polyhedron arranged on the map obtained by the integration of the feature polyhedron, and the intersection of the three-dimensional map obtained by the integration processing of the view polyhedron and the polyhedron arranged on the map. By referring to the information and the number of intersections, the existence probabilities of all the feature polyhedra and the reliability of the probabilities are calculated. The number of polyhedron intersections at the designated position in the three-dimensional space can be known from the count value of the polyhedron occupying the corresponding position in the three-dimensional map.

The part where the feature polyhedron does not exist in the three-dimensional map indicates that the part is not imaged or the imaged part where the object does not exist. The distinction between the two can be determined from the data of the corresponding portion of the three-dimensional map output from the visual field polyhedron integration circuit 36. Furthermore, even if the feature polyhedron exists, the reliability of the observation result varies depending on the number of observations. For example, the existence probability of an object is
From the ratio of the number of intersections of the feature polyhedron and the number of intersections of the field polyhedron, the reliability of the probability is calculated from the number of observations. As a result, in the shape restoration circuit 37, the feature polyhedron integration circuit 33
A three-dimensional map is created by adding information on the existence probability and the reliability to the three-dimensional map generated in (3). Based on this three-dimensional map, the feature polyhedrons are integrated as a restored shape in accordance with the existence probability and reliability setting, and final three-dimensional map data is generated.

In step 55 shown in FIG. 5, in FIG.
The restored three-dimensional shape is displayed on the screen of the CRT 5 shown in FIG. 3, and the three-dimensional shape data is stored in the three-dimensional map memory 4. Although the stereo camera device 1 having two cameras is used as the imaging device in the above-described embodiment, it is pointed out that the number of cameras is not limited to two.
That is, the same processing can be performed by capturing images from a plurality of positions with one camera. Further, in the above embodiment, the processing of the data is performed cumulatively, but it is pointed out that it is possible to prepare the data in parallel and process them all at once.

By the above processing, the following advantages can be obtained. First, by expressing the restored shape of the feature point pixel on the projection screen by a polyhedron, it is possible to consider the structural error of the imaging device. Therefore, the error can be expressed as deformation and movement of the polyhedron according to the error of the imaging parameter. As a result, it is possible to omit the complicated processing for estimating the parameters of the image pickup apparatus, which has been impossible with the conventional three-dimensional position information restoration method.

Further, the parameter error of the image pickup device can be easily estimated without performing the highly accurate estimation of the parameter. Further, in order to maintain the error of the estimation parameter within a certain range, it is not necessary to fix the image pickup device and take measures against image stabilization, so that it is possible to use the image pickup device that involves changes in the shooting posture and position.

In the conventional method shown in FIG. 17, the determination result varies depending on the processing order of the feature polyhedron, but it is pointed out that the above embodiment does not depend on the processing order. That is, in the above-described embodiment, the shape restoration processing is performed after the feature polyhedron and the visual field polyhedron are all arranged on the three-dimensional map, so that the restoration result does not depend on the order of the arrangement processing of the feature polyhedron, and the reliability is high. Some final result is obtained. Furthermore, after arranging all the polyhedra, in order to calculate the existence probability of the feature polyhedron and its reliability, a final result reflecting the imaging situation is obtained.

That is, since a region in which a point corresponding to a pixel on the projection plane may exist in a three-dimensional space is expressed by a polyhedron using point projection, even if an error in image pickup is taken into consideration, It is possible to accurately generate three-dimensional shape data.

[0080]

As described above, according to the present invention, since three-dimensional shape data is generated based on the feature polyhedron data and the visual field polyhedron data, the three-dimensional shape can be more accurately taken into consideration by taking an error in imaging into consideration. A three-dimensional object recognition device capable of generating shape data has been obtained.

[Brief description of drawings]

FIG. 1 is a block diagram of an arithmetic processing unit shown in FIG.

FIG. 2 is a block diagram of a three-dimensional object recognition device showing an embodiment of the present invention.

FIG. 3 is a schematic diagram briefly explaining an operation of the three-dimensional object recognition device shown in FIG.

FIG. 4 is a flowchart of processing in the arithmetic processing device shown in FIG.

5 is a flowchart of processing in the arithmetic processing apparatus shown in FIG.

6 is a flowchart of the feature point detection process shown in FIG.

FIG. 7 is a parameter diagram showing an example of parameters in the stereo camera device.

FIG. 8 is a schematic diagram showing a principle of generating a characteristic polyhedron.

FIG. 9 is a schematic diagram showing generation of a characteristic polyhedron when the error of the focal length is taken into consideration.

FIG. 10 is a schematic diagram showing the principle of generating a visual field polyhedron.

FIG. 11 is a schematic diagram showing generation of a visual field polyhedron when the error of the focal length is taken into consideration.

FIG. 12 is a schematic diagram showing an example of a relationship between two polyhedra in 8-branch tree description processing.

FIG. 13 is a schematic diagram showing another example of the relationship between two polyhedra in 8-branch tree description processing.

FIG. 14 is a schematic diagram showing still another example of the relationship between two polyhedra in the 8-branch tree description processing.

FIG. 15 is a schematic diagram showing an example when a partial intersection occurs between two polyhedra in the 8-branch tree description processing.

FIG. 16: There may be points corresponding to pixels on the projection plane 3
It is a schematic diagram which shows the area | region in dimensional space.

FIG. 17 is a schematic diagram showing that different restoration results can be obtained depending on the order of crossing processing in the conventional technique.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 stereo camera device 3 arithmetic processing device 4 3D map memory 31 feature point detection circuit 32 feature polyhedron generation circuit 33 feature polyhedron integration circuit 34 parameter setting error setting circuit 35 view polyhedron integration circuit 37 view polyhedron integration circuit 37 3D shape restoration circuit

Claims (1)

Claim: What is claimed is: 1. An image pickup means for picking up an image of the same three-dimensional subject from a plurality of first different positions to generate a first plurality of image signals, wherein the image pickup means is an image pickup device. An error setting unit that includes a second plurality of image pickup pixels arranged on the surface, and sets error data that defines an error of the image pickup unit; and in response to the first plurality of image signals. A characteristic point detecting means for detecting a corresponding third characteristic point among the first plurality of images; and a characteristic point detecting means for detecting the characteristic data in response to the error data set by the error setting means. A feature polyhedron generating unit that generates feature polyhedron data for defining a region in which the corresponding feature points may exist in a three-dimensional space; and the first based on the size of the image pickup pixel region on the image pickup surface. In several different positions In response to the characteristic polyhedron data and the visual field polyhedron data, the visual field polyhedron generating means for generating the visual field polyhedron data for defining the area of the visual field that can be commonly imaged by the imaging means,
A three-dimensional object recognition device, comprising: three-dimensional shape data generation means for generating three-dimensional shape data of the subject.