JP4835201B2 - 3D shape detector - Google Patents

Description

  The present invention relates to a three-dimensional shape detection apparatus using an image that can be used for detecting an obstacle around a vehicle.

  Various driving support systems that support driving by grasping the surrounding situation of a car and providing it to a driver are being developed. As one of such driving support systems, when it recognizes obstacles such as guardrails, side walls, parked vehicles, etc., evaluates the possibility of contact with the host vehicle, and determines that there is a possibility of contact, There is a system that performs notification and avoidance control (see, for example, Patent Document 1).

In this document, a CCD camera or an ultrasonic sensor is used to grasp the three-dimensional distance distribution in the direction of the vehicle, grasp the road shape and the obstacle, and determine the possibility of contact by comparing with the shape of the host vehicle. The technology is described.
Japanese Patent Laid-Open No. 10-283592

  In this technology, for obstacles on the road, such as utility poles, parked vehicles, oncoming vehicles, etc., a technique for obtaining the three-dimensional information by stereo vision technology is used, but obstacles below the road surface such as side grooves An object is grasped by an ultrasonic sensor. Although the reason is not described in detail, the spatial position calculation in the stereo view is generally performed based on the correspondence between the pixel positions of the feature points in the corresponding image. Technology is disclosed. However, for example, in the case of a side groove, the edge of the bottom may not be present in the captured image, and the edge position itself is located at the boundary position with the road surface at the top of the groove, and the pattern on the road surface ( It is difficult to distinguish from white lines. These apply not only to the side grooves but also to steps on the road surface.

  Therefore, an object of the present invention is to provide a three-dimensional shape detection device that can grasp the shape of a three-dimensional object such as a road surface step based on image data.

In order to solve the above-described problems, a three-dimensional shape detection apparatus according to the present invention includes (1) an imaging unit that captures a pair of images from different positions in the height direction, and (2) left and right in each of the acquired paired images. Edge extraction means for extracting edges extending in the direction; (3) corresponding point search means for searching for corresponding points for each of the upper and lower areas divided by the extracted edges in the image; ) Based on the relationship between the similarity calculation means for calculating the similarity of the corresponding point searched, and (5) the similarity of the corresponding point searched in the upper region and the similarity of the corresponding point searched in the lower region, And shape estimation means for estimating whether or not the edge is a horizontal edge of a downward step as viewed from the imaging means side .

  By acquiring a pair of images from different positions in the height direction, it is possible to acquire images from different viewpoints for the three-dimensional object. Considering the case of acquiring the image of the side groove as an example, the range of the side wall portion reflected in each of the image acquired from the upper side (upper camera image) and the image acquired from the lower side (lower camera image) is the upper camera. The image reaches a deeper position. In other words, the upper region of the boundary edge between the side wall image and the road surface close to the imaging device captures a different range between the upper and lower camera images. On the other hand, the lower area is obtained by imaging the same area at different angles. As for the boundary edge farther from the road surface, the same region is imaged on both the upper side and the lower side. Therefore, it can be determined from the degree of correlation between the upper region and the lower region whether the edge is closer to the side groove or the like.

Specifically, the shape estimation unit looks at the edge from the imaging unit side when the similarity of the corresponding point searched in the upper region is smaller than a predetermined reference by the similarity of the corresponding point searched in the lower region . If the similarity of the corresponding point searched in the upper area is estimated to be a horizontal edge of the descending step, or if the similarity of the corresponding point searched in the lower area is greater than a predetermined reference, the edge is on the imaging means side It is good to estimate that it is not the horizontal edge of the descending step as seen from .

  It is preferable to further include a light projecting unit that projects light toward an imaging target from a position between different positions in the height direction of the image capturing unit. The light projected by the light projecting unit is not limited to visible light. For example, when the imaging unit can acquire an infrared image, infrared light may be projected.

  Taking a side groove as an example, the light from the light projecting means reaches the upper side of the side wall facing the light projecting means, but the lower region (the bottom portion depending on the shape of the side wall) is the light from the light projecting means. Is blocked by the road surface on the light projecting means side from the edge of the opposite side wall and does not reach and becomes a shadow. When the image is taken from below the light projecting means, this shadow portion is not reflected, but when the image is taken from above the light projecting means, this shadow portion is reflected. That is, a three-dimensional object can be determined from the presence or absence of a shadow.

  According to the three-dimensional shape detection device according to the present invention, when there is a road surface step such as a side groove, the step and the plane are accurately determined by referring not only to the edge position but also to the image correlation degree above and below the edge. It becomes possible to do. At this time, when the object is projected from between the upper and lower imaging means, the three-dimensional shape can be determined with higher accuracy from the shadow formed on the surface of the object.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the description, the same reference numerals are given to the same components in the drawings as much as possible, and duplicate descriptions are omitted.

  FIG. 1 is a block diagram showing a configuration of a first embodiment of a three-dimensional shape detection apparatus according to the present invention. The three-dimensional shape detection device is mounted on a vehicle and used for detecting an obstacle, and includes a stereo camera 3 that captures an image of the imaging target 6 from different viewpoints, and a light projecting unit 4 that illuminates the imaging target 6. The A / D converter 2 for digitally converting the output signal of the stereo camera 3 and the image processing means 1 for detecting the three-dimensional shape by image processing are provided, and the estimated shape information is output to the vehicle behavior control device 5.

  The image processing means 1 is composed of a CPU, ROM, RAM, and the like. Each component in the image processing means 1 described below may be divided by hardware, but may be realized by software operating on common hardware. These software may be constituted by separate programs, subroutines, etc., but some or most of the routines may be shared.

  The image processing means 1 includes an edge extracting means 10 for detecting and extracting the contour line of the object in the image, a corresponding point searching means 11 for searching for a corresponding point between the edges in the corresponding image, and a predetermined adjacent to the edge. Similarity calculation means 12 for calculating the similarity between these areas, and shape estimation means 13 for estimating the three-dimensional shape of the imaging object 6 based on these pieces of information.

  The vehicle behavior control device 5 determines the possibility of contact between the detected obstacle and the own vehicle, and follows a device that performs an alarm or avoidance operation support, a device that maintains traveling in a lane, or a preceding vehicle. And a device that travels.

  FIG. 2 shows an arrangement example of the stereo camera 3, FIG. 3 shows a configuration example thereof, and FIG. 4 shows an example of an acquired image. The stereo camera 3 images the imaging target 6 from different height positions by arranging the upper camera 30u and the lower camera 30d vertically, and images acquired by the upper camera 30u and the lower camera 30d are captured. Sent to the unit 31 for processing. Each of the cameras 30u and 30d is a wide viewing angle camera, and may include, for example, a circumferential fisheye lens or a diagonal fisheye lens. The stereo camera 3 is disposed on the side surface, front, rear, or the like of the vehicle 8. In FIG. 2, the example arrange | positioned on the left side surface is shown. The imaging target is an obstacle or the like around the vehicle 8 and acquires an image of the side groove 90 and the curb 91 provided along the road surface 9. When the image of the side groove 90 as shown in FIG. 4A is acquired by such a stereo camera 3, the images Iu and Id acquired by the upper camera 30u and the lower camera 30d are respectively shown in FIG. 4 (c), respectively (hereinafter, the case where the light projecting means 4 is not used will be described as an example). If the bottom edge of the side groove 90 can be confirmed in both images, the presence of the side groove 90 can be determined from the bottom position, but only the upper edge of the side groove 90 can be seen as shown in FIGS. If not, it is impossible to determine whether the space is a plane or a three-dimensional object only from the position information of the edge portion.

  The three-dimensional shape detection apparatus of the present embodiment can detect the presence of the side groove 90 even from an acquired image as shown in FIGS. 4B and 4C, and the specific operation will be described below. . FIG. 5 is the first half of the main flowchart of the process, and FIG. 6 is the second half. This processing is performed from when the vehicle power is turned on to when it is turned off (if the image processing means has an on / off switch, it is only while the switch is turned on). It is repeatedly executed at the timing.

First, an image signal acquired by the stereo camera 3 and converted into a digital signal by the A / D converter 2 is acquired (see step S1, FIGS. 4B and 4C). Next, the edge extraction means 10 extracts a lateral edge line (step S3). Specifically, the edge points of the outline portions of the side groove corresponding portions P _u and P _d of the image shown in FIG. 7A are extracted by a method such as a Sobel filter or a vertical difference (see FIG. 7B). Then, edge lines L _u1 , L _u2 , L _d1 , and L _d2 as straight lines are obtained from the extracted edge point sequence by Hough transform or least square method (see FIG. 7C). The detected horizontal edge lines are numbered (indexed).

Next, an initial value 0 is substituted for the variable i (step S5), and matching between images is examined for the region below the horizontal edge straight line (step S7). Specifically, first, as shown in FIG. 8A, a small area (for example, 3 × 3 pixel area) a _u1, just below the horizontal edge curve L _u1 in the extracted image Iu of the upper camera 30u _. for _d1, located on epipolar polar _{e 1} corresponding in the image Id of the lower camera 30d by the corresponding point searching unit 11, and the lateral edge straight _{L d1, L d2} immediately under the small regions _{a d1, d1, a d2, d1} is searched, and the degree-of-similarity calculation means 12 obtains a correlation value between the small areas a _{u1, d1} and a _{d1, d1} and a _{d2, d1} . As a method for calculating the correlation value, a pattern recognition technique such as a normalized correlation method or a sum of differences may be used. Then, the small area with the highest correlation value is set as a corresponding area candidate. Let S _L be the correlation value (maximum correlation value) of this corresponding region candidate.

Then, the _{S L} obtained is compared with a threshold value THs (step S9). If S _L is equal to or lower than THs, it is determined that the corresponding area candidate does not correspond to the corresponding area, and the process proceeds to step S27 described later. If S _L exceeds THs, the corresponding area candidate is determined to be a corresponding area. In the example of FIG. 8B, the small areas a _{d1 and d1} are the corresponding areas of the small areas a _{u1 and d1} . The 3D conversion technique by Stereo Vision When the corresponding region can be confirmed edge lower subregion a _{d1, d1} from the positional relationship in the sub-region a _{u1, d1} image to calculate a three-dimensional position P _L of the region ( Step S11). This can be done using the principle of triangulation.

Next, the matching between images is examined for the upper region of the horizontal edge straight line (step S13). The contents of the process are the same as in step S7, and as shown in FIG. 9A, the small area (the size of the area is just above the horizontal edge curve L _u1 in the extracted image Iu of the upper camera 30u, as shown in FIG. 9A. the small area the same size of the step _{S7.) a u1,} for _u1, located on epipolar polar _{e 1} corresponding in the image Id of the lower camera 30d by the corresponding point searching unit 11, and the lateral edge straight line _{L d1} , L _d2 is searched for small areas a _{d1, u1} , a _{d2, u1} , and the similarity calculation means 12 obtains a correlation value. Then, the small area with the highest correlation value is set as a corresponding area candidate. Correlation value of the corresponding region candidates (maximum correlation value) and S _U.

Then, the _{S U} obtained is compared with a threshold value THs (step S15). If S _L is equal to or lower than THs, it is determined that the corresponding area candidate does not correspond to the corresponding area, and the process proceeds to step S19 described later. If the S _U is greater than THs determines the corresponding region candidate and the corresponding region. In the example of FIG. 9B, the small areas a _{d1 and u1} are the corresponding areas of the small areas a _{u1 and u1} . Corresponding region is calculated three-dimensional position _{P U} of the area and the edge upper subregion _{a d1, u1} After confirming the positional relationship in the sub-region _{a u1, u1} image (step S17).

Subsequently shape estimation unit 13 calculates the correlation value ratio RS is the ratio of _{S U} and _{S L} (step S19), the obtained RS is compared with a threshold value TH _R (step S21). The TH _R is less than 1 is value close to 1, for example, is set to 0.8 or the like. Depending on the shape of the three-dimensional object, the area reflected directly above and directly below the corresponding edge in the image Iu acquired by the upper camera 30u and the image Id acquired by the lower camera 30d may be different. Region is to try another when S _U and S _L are different values, RS deviates from 1. Hereinafter, a specific example of a three-dimensional shape will be described.

FIGS. 10 and 11 show a case where both edges E _d1 and E _d2 at the upper end of the side groove are within the imaging range of both cameras 30u and 30d. In this case, in the image Iu acquired by the upper camera 30u and the image Id acquired by the lower camera 30d, each of the small areas reflected immediately below the front edge E _d1 is an area a _d1 adjacent to E _d1. The small region reflected directly above the front edge E _d1 is the wall surface of the side groove on the opposite bank, and in Id, it becomes the upper a _u2 by the region a _u1 in Iu. As a result, the RS becomes smaller than 1.

On the other hand, as shown in FIGS. 12 and 13, the area reflected on the opposite bank, that is, immediately above and immediately below the back edge E _d1 , is the same in the combination of a _u3 and a _d2 in both the image Iu and the image Id. To do. Therefore, the RS in this case is approximately equal to 1.

Also, as in the case of the other side without the down step shown in FIG. 14, FIG. 15, the region being reflected on the lower side of the edge _{E d3} is image Iu, consistent with Id, the area _{a d3} adjacent the edge _{E d3} It becomes. On the other hand, a region where reflected on the immediately above, the image _{Iu, a u4,} the image _Id, becomes _{a u5,} towards the image Iu is, a region close to the camera 3 than the case of the image Id is being reflected. As a result, the RS in this case is also smaller than 1.

On the contrary, in the up step, as shown in FIGS. 16 and 17, for the front edge E _d4 , the areas reflected immediately above and immediately below coincide with a _u6 and a _d4 , and the back edge As for E _d5 , as shown in FIGS. 18 and 19, the areas reflected immediately above and directly below are the same in a _u7 and a _d5 . Again, the RS is approximately equal to 1.

  That is, RS is approximately equal to 1 when the edge is present on a plane or an edge of an ascending step, but when it is an edge of a descending step (including the front edge of a side groove), Less than 1. By utilizing this, it is possible to determine whether or not it is an edge of a downward step by determining RS.

When the RS determines that the threshold value TH _R is smaller than in step S21, it is determined that an edge of the downstream step, marked as gutter edge candidate (step S23). Then, it records the position information P _L obtained in step S11 as the position information of the edge. On the other hand, if the RS is determined to be the threshold value TH _R above, and determined not to be the edge of the downstream step, marked as an edge candidate other than gutters (step S25). In this case, it records the position information P _U obtained in step S17 as the position information of the edge.

  After steps S23 and S25 are completed, the process proceeds to step S27, and it is determined whether or not all constituent points of the horizontal edge line Li have been examined. If the determination has not been completed for all of the constituent points, the process returns to step S7, and the processes of steps S7 to S25 are executed for the next constituent point. As a result, as shown in FIGS. 8C and 9C, the areas corresponding to the area immediately above all the constituent points from the left end to the right end of the edge Lu1 and the small areas of the areas immediately below are detected, and the down step Whether or not it is an edge and its position information are calculated.

When the determination is completed for all the constituent points of the horizontal edge line L _i, the operation proceeds to step S29 shown in FIG. 6, the horizontal edge line Li, the number Ng of marks correlated with the structures point as gutter edge candidate, except gutter The number Nc of component points marked as edge candidates is counted. In FIG. 20, the side groove edge candidates are indicated by black circles, and the edge candidates other than the side grooves are indicated by black triangles. Depending on the conditions at the time of image acquisition (brightness, shadow state, angle with the camera, etc.) The determination may be reversed. In order to reduce the influence of such an erroneous determination, the accuracy of determination is improved by examining the ratio of the number Ng of side groove edge candidates to the number Nc of other edge candidates for the entire edge.

Specifically, comparing the ratio with a threshold TH _N of Ng and Nc (step S31). When Ng / Nc is greater than the TH _N is to association marks the edge Li as gutter edge (step S33), and derives the three-dimensional position and orientation of the edge line (step S35). In the example of FIG. 20, the near side edge where black circles occupy most is marked as a gutter edge. Using the three-dimensional information on the points constituting the edge line, or just under or immediately above the edge point, a three-dimensional linear expression of the edge can be derived by using a method such as Hough transform or least square method. That's fine. This linear formula can be expressed by the following formula (1), for example.

Note that x ₁ , y ₁ , z ₁ , a ₁ , b ₁ , and c ₁ are all constants.

On the other hand, when Ng / Nc is less than TH _N is to association marks the edge Li as an edge other than gutters (step S37), and derives the three-dimensional position and orientation of the edge line (step S39). In the example of FIG. 20, the far side edge in which the black triangle occupies most is marked as an edge other than the side groove. The calculation method of the spatial position may be the same method as in the case of the side groove edge described above.

  After steps S35 and S39 are completed, the process proceeds to step S41, and it is checked whether or not the determination has been completed for all the horizontal edge lines in the image. If not completed, the process proceeds to step S43, 1 is added to the variable i, and the process returns to step S7 to investigate the next edge line.

  When the investigation is completed for all the edge lines, the process proceeds to step S45, and it is examined whether or not a side groove edge line exists in the screen. When the side groove edge line does not exist, the subsequent process is skipped and the process is terminated. If there is a side groove edge line, the variable j is set to the initial value 0 (step S47), and the back edge corresponding to the side groove edge line Lgi (front edge) is searched (step S49). Specifically, the horizontal edge line Lo that is parallel to the side groove edge line Lgi and located above (back side) and closest to the side groove edge line Lgi is searched.

As an example, consider a case where the side groove edge line is expressed by the above formula (1) and another edge line is expressed by the following formula (2). _{(Where, x 2, y 2, z} 2, a 2, b 2, c 2 is also constant.

Angle theta ₁₂ in which the two straight lines forms can be obtained by the following equation (3).

When the absolute value of theta ₁₂ is smaller than the threshold value TH _theta (e.g., less than 5 °) in the case of, may be determined that the two lines are substantially parallel.

It is determined whether or not the search for the horizontal edge line Lo is successful (step S51). If the search is successful, that is, if Lo exists, the process proceeds to step S53, where Lo is the opposite bank edge, the width of the side groove, The apparent depth is calculated. The width of groove D ₁₂ can be calculated as the distance between the two straight lines can be expressed by the following equation (4).

  Further, when the positional relationship between the vehicle 8 and the side groove 90 is as shown in FIG. 21, the angle between the distance d to the side groove and the extension direction of the side groove and the extension line extending laterally from the vehicle 8. φ is expressed by the following formulas (5) and (6).

  On the other hand, when Lo does not exist, it means a case where the opposite bank does not exist in the imaging region or is a simple descending step having no opposite bank. In this case, the process proceeds to step S55, where it is determined that the step or the opposite bank is a wall, and information indicating that is recorded.

  After completion of steps S53 and S55, the obtained side groove information is output (step S57). Then, it is determined whether or not the search for the corresponding horizontal edge line for all the side groove edge lines has been completed (step S59). If the determination is not completed, the process proceeds to step S61, j is incremented by 1, and the corresponding lateral edge line search process is performed for the next gutter edge line. On the other hand, when the determination is finished, the entire process is finished.

  As described above, according to the present invention, it is possible to determine the presence of a three-dimensional object such as a gutter by determining not only the contour position of an object detected by edge extraction but also the similarity of the region immediately below the edge. Can do. As a result, the side groove can be correctly recognized as an obstacle, and appropriate vehicle behavior control (for example, obstacle avoidance control) can be performed in the vehicle behavior control device 5, thereby further improving safety.

  Furthermore, detection can be facilitated by utilizing the light projecting means 4. FIG. 22A shows a relationship diagram when the illumination light is emitted toward the subject by the light projecting means 4, and FIGS. 22B and 22C show the upper camera image Iu acquired in this case and the lower camera. An image Id is shown. Here, the illumination light irradiated by the light projecting means 4 is not limited to visible light such as infrared light and ultraviolet light as long as it can be detected by the upper camera 30u and the lower camera 30d.

Light projection area 40 which is projected by the light projecting means 4 arranged between the upper camera 30u and lower camera 30d will become as shown in FIG. 22 (a), an upper wall opposite to the front edge E _d1 Is located in the projection area 40, but the lower side is a shadow of the front edge E _d1 . Therefore, the area _au2 reflected in the image Id immediately above the front edge E _d1 is located in the light projecting area 40 and has a relatively high luminance value, whereas in the image Iu, immediately above the front edge E _d1 . The reflected area a _u1 is located in the shadow S1 outside the projection area 40 and has a relatively low luminance value. For this reason, the luminance difference becomes large and the correlation value becomes low. As a result, the front edge can be easily detected, and the side groove and the like can be determined with higher accuracy.

  In the above description, an example in which the upper and lower stereo images are captured by the upper camera 30u and the lower camera 30d has been described. However, acquisition of a stereo image does not necessarily require two cameras. FIG. 23 is a schematic diagram illustrating a configuration of a stereo camera 3a that can capture an upper and lower stereo image with one camera. In the camera 3a, symmetrical lenses 35u and 35d and mirrors 36u, 37u, 36d, and 37d are arranged on the entire surface of an optical system 33 of a camera body 34 that includes a solid-state imaging device such as a CCD or CMOS, an imaging tube, or the like. . By arranging the optical axes of the lens 35u and the lens 35d to be parallel and separated by a predetermined distance (about 50 mm in the figure), the upper half of the acquired one screen is the upper camera image Iu, and the lower half is the lower camera image Id. It is equivalent to. When such a camera is used, when installing an upper and lower stereo camera on the vehicle 8, it is not necessary to adjust the optical axis and the arrangement interval between the upper and lower cameras, and the installation is facilitated and the arrangement accuracy is improved. As a result, the recognition accuracy of the three-dimensional object can be increased.

  Further, in the above description, the case of the top and bottom stereo view has been described as an example. However, even in the case of the left and right stereo view or the stereo camera arranged at an angle, it is difficult to make a determination only by the normal stereo view processing by the same method. It is possible to determine a solid object shape.

  Next, a second embodiment of the three-dimensional shape detection apparatus according to the present invention will be described. FIG. 24 is a block diagram showing the configuration of the second embodiment. Here, a case where the above-described stereo camera 3a is used will be described as an example. The output of the camera 3a is converted by the A / D converter 2 and sent to the image processing means 1a. The image processing unit 1a is configured by a CPU, a ROM, a RAM, and the like similarly to the image processing unit of the first embodiment shown in FIG. 1, and each component is realized by hardware or software.

  The image processing unit 1a includes an edge extraction unit 10 that extracts a horizontal edge in the image, a first spatial position detection unit 14 that detects a three-dimensional position of the extracted edge, and sets a detection region based on the result. Detection area setting means 15 for performing, second spatial position detection means 16 for detecting the three-dimensional position of the set detection area, and shape estimation means 17 for estimating the three-dimensional shape of the object according to the detection result. Yes.

  The image processing means 1a receives the output of the navigation device 51 that detects the current position of the vehicle and reads the map information stored in the map DB 50, and controls the vehicle behavior as in the first embodiment. Information about the estimated three-dimensional shape is output to the device 5.

  FIG. 25 shows a flowchart of the three-dimensional object detection process that is the operation of this embodiment. First, in step S101, an image as shown in FIG. 26A is acquired by the camera 3a. Next, the edge extraction means 10 extracts a lateral edge from the acquired image (see step S103, FIG. 26B). The edge extraction process itself may be performed using the same method as the processing method in step S3 in the first embodiment described above.

  Next, vehicle position / posture / envelopment information is acquired from the navigation device 51 (step S105). Then, the position information is read from the map DB 50, and the traveling locus of the vehicle is associated with the current vehicle position information (step S107). Then, the relative position information of the road structure is acquired from the map DB 50 (step S109). A position in the image of the structure is predicted from the acquired relative position information of the road structure, and a search range is set (step S111).

  Based on the set search range, the corresponding point searching means 11 searches for corresponding points of the edge between the upper and lower images (see step S113, FIG. 26C). Here, in the upper and lower images, the corresponding point of the edge is located on the same epipolar polar line indicated by the symbol e in FIG. Corresponding points are detected using this relationship and the similarity in the image.

  When the corresponding point is detected, the first spatial position detecting means 14 derives the three-dimensional position by a stereo vision technique (step S115). An example of the detected three-dimensional position of the lateral edge is shown in FIGS. 27 (a), 27 (b), and 28. FIG. Fig.27 (a) is a perspective view, FIG.27 (b) is sectional drawing which looked at it from the vehicle forward direction. FIG. 28 is a diagram showing a diagram corresponding to FIG.

Based on the three-dimensional position detection result, the detection region setting means 15 extracts a point (edge) whose distance from the plane 9 on which the host vehicle 8 is on is within the threshold Th1 (step S117). In the example shown in FIG. 28, Ed-c to Ed-e correspond. FIG. 29 shows the positions of the edges Ed-c to Ed-e in the actual images Iu and Id. The detection area setting means 15 further sets a small area Ri between the edges located on the extracted plane (including edges located close to it), and the second spatial position detection means 16 sets the spatial position. Derived (step S119). For example, as illustrated in FIGS. 29 and 30, small regions a _{A to} a _C are arranged between Ed-d and Ed-e in the lower image Id. Then, they searched the area corresponding to the small region a _A ~a _C between the upper image Ed-d and Ed-e, to derive the three-dimensional positions of the techniques of stereo vision.

The shape estimating means 17 determines the spatial positional relationship between the obtained small region Ri and the edge sandwiching the small region Ri (step S121). For example, as shown in FIG. 31 (a), the small region _a A ~a _C is located in the same vertical plane on the rear side edge Ed-d, if the plane 9 lower than the threshold value Th2 position, step S123 It shifts to and is judged as a side groove. On the other hand, as shown in FIG. 31 (b), if the small region _a A ~a _C is positioned at substantially the same horizontal plane as the edge Ed-c and Ed-d, unless a ditch the operation proceeds to step S125 judge. After steps S123 and S125 are completed, the process proceeds to step S127, and the determination result is output. The vehicle behavior control device 5 performs desired vehicle behavior control according to the determination result.

  In this way, not only the edge but also the small region sandwiched between the edges is taken, and by calculating its three-dimensional spatial position, for example, from the edge like the side groove where the bottom image cannot be acquired, It becomes possible to grasp the shape of a three-dimensional object that cannot accurately grasp the three-dimensional shape with high accuracy.

  Here, although the determination is made for the edge on the plane on which the vehicle is riding, the determination may be performed for other planes, for example, the plane including Ed-a and Ed-b in FIG. Moreover, although the case where the stereo camera 3a was used was demonstrated in the above example, you may perform a three-dimensional position detection with a motion-based stereo method using a time-sequential image, for example. Alternatively, the determination may be made using the output of the laser scanner.

  In addition, here, an example has been described in which the position in the image is predicted from the position information of an object (such as a guardrail) searched in advance using the map DB 50 or the navigation device 51, and detection is performed based on the prediction result. The present invention can be suitably applied even when a three-dimensional object is detected only from an image without using such object position information.

  In the above description, the case where it is mounted on a vehicle has been described as an example. However, the three-dimensional shape detection apparatus according to the present invention is not limited to a vehicle-mounted device, for example, a robot camera, various inspection devices, and monitoring. It can be used for cameras.

It is a block diagram which shows the structure of 1st Embodiment of the three-dimensional shape detection apparatus concerning this invention. The example of arrangement | positioning of the stereo camera 3 of FIG. 1 is shown. The structural example of the stereo camera 3 of FIG. 2 is shown. An example of the image acquired with the stereo camera 3 of FIG. 3 is shown. It is the first half part of the flowchart of the solid shape detection process by embodiment of FIG. It is a latter half part of the flowchart of the solid shape detection process by embodiment of FIG. It is a figure explaining an edge extraction method. It is a figure explaining the example of a similarity calculation process of the area | region directly under an edge. It is a figure explaining the example of a similarity calculation process of the area | region directly above an edge. It is a figure which shows the positional relationship of the area | region reflected in the front edge when the both edges of the upper end of a side groove | channel are in the imaging range of a camera, and its directly upper and lower. In the case of FIG. 10, it is a figure which shows each image acquired with an upper camera and a lower camera. It is a figure which shows the positional relationship of the area | region reflected in the back side edge when the both edges of the upper end of a side groove | channel are in the imaging range of a camera, and its directly upper and lower. In the case of FIG. 12, it is a figure which shows each image acquired with an upper camera and a lower camera. It is a figure which shows the positional relationship of the edge at the time of imaging a downward level | step difference, and the area | region reflected just above and directly under it. In the case of FIG. 14, it is a figure which shows each image acquired with an upper camera and a lower camera. It is a figure which shows the positional relationship of the area | region reflected on the near side edge in case both edges of an uphill level | step difference exist in the imaging range of a camera, and its directly upper and lower right. In the case of FIG. 16, it is a figure which shows each image acquired with an upper camera and a lower camera. It is a figure which shows the positional relationship of the area | region reflected in the back side edge in case both edges of an uphill level | step difference exist in the imaging range of a camera, and its directly upper and lower. In the case of FIG. 18, it is a figure which shows each image acquired with an upper camera and a lower camera. It is a figure which shows the example of marking of the composing point as a side groove edge candidate and candidates other than a side groove edge. It is a figure explaining the positional relationship information of the side groove with respect to a vehicle. It is a figure explaining the usage example of the light projection means in 1st Embodiment. It is a figure explaining another embodiment of a stereo camera. It is a block diagram which shows the structure of 2nd Embodiment of the solid shape detection apparatus concerning this invention. It is a flowchart of the solid-object detection process by 2nd Embodiment shown in FIG. It is a figure explaining the edge extraction process in the process of FIG. It is a figure explaining an example of the three-dimensional position of the detected horizontal edge. It is a figure which shows the positional relationship of a vehicle and a horizontal edge corresponding to FIG. It is a figure which shows the positional relationship in the space position of an edge and an image. It is a figure which shows the example of a setting of the small area | region between edges. It is a figure explaining the example of determination of the spatial position of a small area.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1a ... Image processing means, 2 ... A / D converter, 3, 3a ... Stereo camera, 4 ... Light projection means, 5 ... Vehicle behavior control apparatus, 6 ... Imaging object, 8 ... Vehicle, 9 ... Plane, 10 ... edge extraction means, 11 ... corresponding point search means, 12 ... similarity calculation means, 13 ... shape estimation means, 14 ... first spatial position detection means, 15 ... detection region setting means, 16 ... second spatial position detection means, 17 ... Shape estimation means, 30d ... Lower camera, 30u ... Upper camera, 31 ... Imaging unit, 33 ... Optical system, 34 ... Camera body, 35u, 35d ... Lens, 36u, 36d, 37u, 37d ... Mirror, 40 ... Throw light area, 50 ... map DB, 51 ... navigation device, 90 ... groove, 91 ... curbs, e 1 _... epipolar polar, Ed ... edge, Id ... lower camera image, Iu ... upper camera image.

Claims (4)

Imaging means for capturing a pair of images from different positions in the height direction;
Edge extraction means for extracting an edge extending in the left-right direction in each of the acquired paired images;
Corresponding point search means for searching for corresponding points for each of the upper region and the lower region divided by the extracted edge in the paired image;
A similarity calculation means for calculating the similarity of the searched corresponding points;
Based on the relationship between the similarity of the corresponding point searched in the upper area and the similarity of the corresponding point searched in the lower area, it is determined whether or not the edge is a horizontal edge of the downward step as viewed from the imaging means side. Shape estimation means for estimation;
A three-dimensional shape detection apparatus comprising: The shape estimation unit, when the similarity of the corresponding points is searched in the upper region is smaller than a predetermined reference than the similarity of the corresponding points is searched in the lower region, next to the downstream step in which the edge is viewed from the imaging means side The three-dimensional shape detection apparatus according to claim 1, wherein the three-dimensional shape detection apparatus estimates an edge. The shape estimation unit, when the similarity of the corresponding points is searched in the upper region is larger than a predetermined reference than the similarity of the corresponding points is searched in the lower region, next to the downstream step in which the edge is viewed from the imaging means side The three-dimensional shape detection apparatus according to claim 1, wherein the three-dimensional shape detection apparatus estimates that the edge is not an edge.   The three-dimensional shape detection according to claim 1, further comprising a light projecting unit that projects light toward a target to be imaged from a position between different positions in the height direction of the image capturing unit. apparatus.

Images