JP2013161190A - Object recognition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an object recognition device capable of enhancing object recognition accuracy in a distant area.SOLUTION: An object recognition device 1 includes a stereo camera 17, and a processing device 20 that processes images acquired by the stereo camera. The processing device calculates parallax from the two images acquired by the stereo camera, detects a three-dimensional object on the basis of the calculated parallax, generates a luminance change pattern of the three-dimensional object along a horizontal line in one image of the two images acquired by the stereo camera, and executes the search of the luminance change pattern toward a more distant side than a detection position of the three-dimensional object.

Description

  The present invention relates to an object recognition apparatus using a stereo camera.

  Conventionally, a lane candidate area is detected from a stereo camera, and the size of the three-dimensional object in real space calculated from the parallax is within a preset threshold value, and the luminance change in the vertical direction of the three-dimensional object is constant. When it has, the technique which detects the solid object as a driving | running | working guidance obstacle is known (for example, refer patent document 1).

JP 2007-280132 A

  By the way, there is a case where sufficient disparity cannot be obtained even in the case of a three-dimensional object in a far region in an image acquired by a stereo camera. In particular, for a three-dimensional object such as a curb or a sidewalk, the detection based on parallax becomes difficult as the distance increases.

  Accordingly, an object of the present invention is to provide an object recognition apparatus that can improve the object recognition accuracy in a far region.

In order to achieve the above object, according to one aspect of the present invention, a stereo camera;
A processing device for processing an image acquired by the stereo camera,
The processing device calculates a parallax from two images acquired by the stereo camera, detects a three-dimensional object based on the calculated parallax, and in one of the two images acquired by the stereo camera. An object recognition device is provided, wherein a brightness change pattern of the three-dimensional object along a horizontal line is generated, and the search of the brightness change pattern is performed farther than a detection position of the three-dimensional object. The

  ADVANTAGE OF THE INVENTION According to this invention, the object recognition apparatus which can improve the object recognition precision in a distant area | region is obtained.

It is the figure which showed the principal part structure of the object recognition apparatus 1 by one Example of this invention. It is a flowchart which shows an example of the main processes implement | achieved by stereo ECU20 of a present Example. It is a figure which shows an example of an edge detection result and a solid-object detection point. It is a figure which shows the example of a setting of a curb candidate area | region. It is a figure which shows the example of a production | generation of a luminance change pattern. It is a figure which shows an example of the search area | region of a luminance change pattern.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram showing a main configuration of an object recognition apparatus 1 according to an embodiment of the present invention. The object recognition device 1 includes an imaging device 10 and a stereo ECU (Electric Control Unit) 20. The ECU includes a plurality of circuit elements such as a central processing unit (CPU), a ROM that stores programs, a RAM that temporarily stores data, an input interface, and an output interface as a unit.

  The imaging apparatus 10 includes a stereo camera 17 that includes imaging elements 11 and 12 and lenses 15 and 16. The imaging device 10 includes a camera CPU 13 and an image output unit 14. The image sensors 11 and 12 are constituted by, for example, a CCD (Charge Coupled Device). The camera CPU 13 controls the exposure and the like on the camera side based on a control signal from the stereo ECU 20. The camera CPU 13 transmits images captured by the imaging elements 11 and 12 to the stereo ECU 20 as image signals via the image output unit 14 that is an output interface. Note that the number of image pickup devices is not limited to two, and may be greater than that. Further, the imaging elements 11 and 12 and the lenses 15 and 16 are not necessarily arranged in the same camera unit, and may be arranged apart from each other in the vehicle lateral direction.

  The stereo ECU 20 includes an image input unit 21, which is an input interface, a geometric conversion LSI 22, an image processing LSI 24, an SV-CPU 23 that supervises each processing unit, and the like.

  The image signal output from the image output unit 14 of the imaging device 10 is sent to an image input unit 21 that is an input interface of the stereo ECU 20. The image output unit 14 and the image input unit 21 are predetermined digital transmission system interfaces.

  Upon receiving the image signal, the image input unit 21 sends image data of the image signal to the geometric transformation LSI 22. The geometric conversion LSI 22 uses the calibration data, and from the captured image of the stereo camera 17 used for the stereo calculation processing or the like, causes a hardware internal error factor (lens distortion, A known process for removing the influence of optical axis deviation, focal distance deviation, image pickup element distortion, etc.) and aligning the epipolar line with the image horizontal line is performed. The geometric conversion LSI 22 converts an input image based on a geometric conversion LUT (Look Up Table) stored in the memory 25.

  The image processing LSI 24 performs image recognition processing based on the geometrically converted image data from the geometric conversion LSI 22. The image processing program and the image to be processed are recorded in the memory 26, and the image processing LSI 24 reads them and performs image recognition processing. The image recognition process includes an object recognition process based on a parallax image (parallax information) and a luminance change pattern search process.

  In the object recognition processing based on the parallax image, for example, a correlation between a pair of images captured by the imaging elements 11 and 12 arranged on the left and right is obtained, and the distance to the object is calculated in the manner of triangulation based on the parallax with respect to the same object. Is to be calculated. That is, the image processing LSI 24 extracts a portion in which the same imaging object is captured from a set of stereo images captured by the imaging elements 11 and 12, and the same point of the imaging object is detected between the pair of stereo images. Are calculated, and the distance to the imaging target is calculated by obtaining the shift amount (parallax) of the associated point (corresponding point). When the imaging object is in front, when the image by the imaging element 11 and the image by the imaging element 12 are overlapped, the imaging object is shifted in the horizontal direction. Then, the most overlapping position is obtained while shifting one image pixel by pixel. The number of pixels shifted at this time is n. Assuming that the focal length of the lens is f, the distance between the optical axes is m, and the pixel pitch is d, the distance L to the object to be imaged is L = (f · m) / (n · d). . This (n · d) term is parallax. The luminance change pattern search process will be described later.

  The SV-CPU 23 is a CPU that supervises each processing unit. The image processing LSI 24 may be used as well. The SV-CPU 23 transmits and instructs a camera control value (for example, shutter speed information that is a parameter for exposure adjustment) calculated based on the processed image of the image processing LSI 24 to the camera CPU 13 in the imaging apparatus 10.

  By mounting such an imaging device 10 or stereo ECU 20 on a vehicle, it can be used for control using image recognition information such as obstacles on the road. For example, the SV-CPU 23 may transmit the result to another ECU that requires the image recognition processing result (obstacle recognition result) via the in-vehicle LAN. Other ECUs include, for example, a preceding vehicle following system such as ACC (Auto Cruise Control), an ECU that controls a collision avoidance / collision reduction system, an ECU that controls a lane keeping support system and a lane departure warning system, an inter-vehicle ECU, a brake ECU etc.

  FIG. 2 is a flowchart showing an example of main processing realized by the stereo ECU 20 of the present embodiment. The processing routine shown in FIG. 2 may be repeatedly executed at predetermined intervals while the ignition switch of the vehicle is on.

  In step 200, edge detection is performed. The edge detection may be performed on an image by any one of the imaging elements 11 and 12 of the stereo camera 17. Here, the image by the imaging device 11 of the stereo camera 17 is the right image (right image), the image by the imaging device 12 is the left image (left image), and the right image is the reference image. In this case, edge detection may be performed on the right image. FIG. 3A shows an example of the edge detection result. In the figure, thick line portions E1-E6 indicate detected edges. In the example shown in FIG. 3, the edge of the roadside curb is detected along with the white line of the road. That is, in FIG. 3, the edges related to the white line of the road are indicated by symbols E5 and E6, and the edges related to the curb are indicated by symbols E1, E2 and E3. The curb has an upper edge related to the edges E1 and E2, and is connected to the road at the edge related to the edge E3. In other words, the curb is erected on the road with a width between the upper edges of the edges E1 and E2.

  In step 202, a horizontal shift amount (that is, parallax) between the left and right pixels of the stereo camera 17 is calculated. That is, the parallax of each corresponding point between each image by the imaging elements 11 and 12 of the stereo camera 17 is calculated, and a parallax image is generated. Specifically, the pixel corresponding to the pixel whose edge is detected in step 200 is searched on the left image, and the parallax between the left and right pixels is calculated. Next, road surface points are removed using parallax statistics (parallax distribution in a local region, etc.), and only points (three-dimensional object detection points) that become solid objects are detected (extracted). FIG. 3B shows an example of a three-dimensional object detection point. Each white point in the figure indicates a three-dimensional object detection point. In the example shown in FIG. 3, a curb on the roadside is detected as a three-dimensional object detection point. That is, solid object detection points appear along the upper edge (both sides) of the curb.

  Here, the three-dimensional object detection process in step 202 is preferably executed only in a pixel region where the amount of parallax is sufficiently large. For example, the three-dimensional object detection process is executed only in a pixel area (that is, a short distance area) where the parallax calculated in step 200 is a certain value or more. The constant value may correspond to a lower limit value of parallax that can accurately recognize an obstacle (a three-dimensional object) based on parallax information. This constant value changes according to the performance of the image pickup devices 11 and 12 of the stereo camera 17 and the required accuracy, and may be adapted by experiments or the like. In the example shown in FIG. 3, the curb on the roadside is detected as a three-dimensional object detection point only in the short distance region. It should be noted that it is difficult to detect a three-dimensional object having a low height, such as a curb on the side of a road, in a pixel region (a long distance region) where the parallax is less than a certain value.

  In step 204, a curb candidate area extending left and right around the line connecting the three-dimensional object detection points (one-dot chain line) is set. The width of the curb candidate area (the number of pixels extending to the left and right) may be a fixed value or may be varied according to the distance between the three-dimensional object detection points. In the latter case, the width of the curb candidate area may be reduced as the distance between the three-dimensional object detection points increases. FIG. 4 is a diagram illustrating a setting example of a curb candidate area. In the example shown in FIG. 4, the curb candidate area is set within a frame with a reference numeral 70. In the example shown in FIG. 4, the line connecting the three-dimensional object detection points (one-dot chain line) is set at the center in the width direction of the upper edge (both sides) of the curb. In this case, the width of the curb candidate area (the number of pixels spread to the left and right) may be a value corresponding to the width of the curb, and similarly may be varied according to the distance of each three-dimensional object detection point. The method for determining the center line (one-dot chain line) of the curb candidate area is arbitrary, and the curb candidate area only needs to be set so as to include the three-dimensional object detection point that becomes the curb candidate in the vicinity of the center in the width direction.

  In step 206, the luminance value along the horizontal line in the curb candidate area set in step 204 is measured, and the luminance change pattern along the horizontal line in the curb candidate area set in step 204 is generated and recorded. The The number of horizontal lines for measuring the luminance value may be arbitrary. For example, the luminance change pattern may be based on a horizontal line positioned at the center in the vertical direction of the curb candidate area set in step 204 above, or a plurality of representative horizontal lines are determined and their statistics are calculated. It may be based on values (average, variance, etc.). FIG. 5 is a diagram illustrating an example of generation of a brightness change pattern, FIG. 5A illustrates an example of a horizontal line, and FIG. 5B illustrates an example of a brightness change pattern. In the example shown in FIG. 5A, a horizontal line for generating a luminance change pattern is indicated by an arrow with a reference numeral 90. In FIG. 5B, the horizontal axis represents the X coordinate of each pixel on the horizontal line (X coordinate on the image plane), and the vertical axis represents the luminance value of each pixel on the horizontal line.

  In step 208, based on the luminance change pattern generated in step 206, the presence of a similar luminance change pattern is searched for far away. This is because when the curb extends far, a similar luminance change pattern due to the curb should extend far. Thereby, it is possible to detect a distant curb that is difficult to detect only by the parallax information. Since the number of pixels in the horizontal direction of the curb decreases with increasing distance, the brightness change pattern to be verified may be reduced in the horizontal direction. It should be noted that when searching for a luminance change pattern similar to the luminance change pattern generated in step 206, the pattern near the center line (the dashed line in FIG. 5) in the luminance change pattern generated in step 206 is particularly important. May be. That is, the search for the luminance change pattern may be performed in a manner in which a heavy weight is given to the luminance change pattern near the three-dimensional object detection point.

  In step 208, the search for the luminance change pattern may be executed only in the search region for the luminance change pattern. FIG. 6 is a diagram illustrating an example of a search region for luminance change patterns. In the example shown in FIG. 6, the search area for the luminance change pattern is set within a frame marked with reference numeral 80. The luminance change pattern search region 80 may be set with the curb candidate region 70 as a reference. The luminance change pattern search region 80 is set to a region farther than the curb candidate region 70 (a pixel region where the parallax is less than a certain value) in a manner that continues from the curb candidate region 70. At this time, the luminance change pattern search region 80 may extend within a range including an extension line of a line connecting the three-dimensional object detection points (refer to a one-dot chain line in FIG. 5A).

  In Step 210, the solid object detected in Step 202 (in this example, curbstone) and the solid object detected in the search in Step 208 are determined as a curb area, and the result is output. That is, a pixel area having a luminance change pattern similar to the luminance change pattern generated in step 206 is determined as a curb area together with the three-dimensional object detection point area detected in step 202.

  According to the object recognition apparatus 1 of the present embodiment described above, the following excellent effects can be obtained.

  According to the present embodiment, as described above, in a distant region where a sufficient amount of parallax is not obtained, the same three-dimensional object (accurately based on the luminance change pattern of the three-dimensional object detected based on the disparity information in the short-distance region. In this case, the three-dimensional object can be detected with high accuracy even in a remote area where a sufficient amount of parallax is not obtained. Especially for solid objects such as curbs and sidewalks that are low in height, it becomes difficult to detect based on parallax as the distance increases. It can be detected with high accuracy. In addition, the luminance change pattern used for detecting the three-dimensional object in the far region is generated based on the luminance measurement value of the three-dimensional object detected based on the parallax information in the short-distance region as described above, and thus represents the three-dimensional object. The reliability as a luminance change pattern is high, and the same three-dimensional object can be detected with high reliability in a distant region.

  The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the three-dimensional object detected by collating the brightness change pattern is a curb, but the same detection method should be used effectively to detect a three-dimensional object other than the curb. Can do. In particular, the detection method according to the above-described embodiment is suitable for detecting a three-dimensional object continuously extending along a road from a short distance to a long distance. As such a three-dimensional object, there are a guard rail, a side wall, a sidewalk (step) and the like other than the curb. Note that, in the case where the three-dimensional object is a sidewalk, the three-dimensional object detection point may appear in a mode that extends far by one, unlike a curb that extends far by two (see FIG. 3 and the like). In this case, a region corresponding to the curb candidate region 70 may be set with a predetermined width around the one three-dimensional object detection point.

  Moreover, in the Example mentioned above, in order to raise processing efficiency, based on the detected solid object detection point, the curb candidate area 70 which limited the range so that these solid object detection points are mainly included is generated, Although the luminance change pattern is generated on the horizontal line in the curb candidate area 70, the luminance change pattern may be generated on the horizontal line from the end to the end of the image.

  Further, in the above-described embodiment, in order to increase the processing efficiency, the luminance change pattern search region 80 is set in a region having a high probability that they exist in the far region based on the detected three-dimensional object detection points, and the luminance The search is performed in the change pattern search area 80. However, for example, the search may be performed over the entire region where the parallax is less than a certain value. That is, the luminance change pattern search area 80 may be set in an arbitrary manner.

  Moreover, although the object recognition apparatus 1 provided with a specific hardware configuration is disclosed in FIG. 1, the hardware configuration of the object recognition apparatus 1 may be arbitrary. In the hardware configuration shown in FIG. 1, the function sharing between the camera CPU 13 and the stereo ECU 20 is also arbitrary, and part or all of one function may be realized by the other.

DESCRIPTION OF SYMBOLS 1 Object recognition apparatus 10 Imaging device 11, 12 Image sensor 13 Camera CPU
14 Image output unit 15, 16 Lens 17 Stereo camera 20 Stereo ECU
21 Image Input Unit 22 Geometric Transformation LSI
23 SV-CPU
24 Image processing LSI
25, 26 Memory 28 EEPROM
70 Curb candidate area 80 Brightness change pattern search area 90 Horizontal line

Claims (4)

A stereo camera,
A processing device for processing an image acquired by the stereo camera,
The processing device calculates a parallax from two images acquired by the stereo camera, detects a three-dimensional object based on the calculated parallax, and in one of the two images acquired by the stereo camera. An object recognition apparatus characterized by generating a luminance change pattern of the three-dimensional object along a horizontal line, and executing the search for the luminance change pattern farther than the detection position of the three-dimensional object. The processing device detects the three-dimensional object in a short-distance pixel region where the calculated parallax is a predetermined value or more,
The object recognition device according to claim 1, wherein the processing device executes the search for the luminance change pattern in a pixel region where the calculated parallax is less than a predetermined value.   The object recognition device according to claim 1, wherein the processing device detects a portion having a luminance change pattern similar to the luminance change pattern as a continuous portion of the three-dimensional object by searching for the luminance change pattern.   The object recognition apparatus according to any one of claims 1 to 3, wherein the three-dimensional object is a three-dimensional object that continuously extends far along a road.

Images