JP2014119859A - Image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which performs ranging using corresponding point search based on a stereoscopic image, and detects and recognizes an object in the image, while reducing the total processing load from corresponding point search to object recognition.SOLUTION: When an object (short-range object) is detected within a predetermined distance by object detection based on small-size image data, corresponding point search using a predetermined search range and corresponding point search using a search range modified from the predetermined search range are performed in object detection based on large-size image data. Only object detection can be performed based on the small-size image data, thereby reducing processing load as compared with a conventional case where a result of object recognition for a small-size image is used for object detection and object recognition based on a large-size image.

Description

  The present invention relates to an image processing apparatus that performs distance measurement by a corresponding point search method based on a stereo image, and performs detection processing and recognition processing of an object present in the image.

JP 2008-39491 A

  For example, driving assistance technologies for assisting the driver, such as collision avoidance such as detecting the approach of a preceding vehicle and other obstacles to brake the vehicle, and auto-cruising, are known. In such driving support technology, vehicle control such as vehicle braking and throttle control is performed based on the result of detecting the distance to an object existing in front of the vehicle and recognizing the type of the object (preceding vehicle, pedestrian, etc.). Is done.

As a technique for obtaining information on the distance to the imaged object, a technique using a so-called corresponding point search method is known.
Conventionally, distance information for each corresponding point calculated by the corresponding point search method is stored as a distance image in association with the coordinates of each corresponding point (coordinates in a two-dimensional space as an image plane). Is used to perform processing (object detection processing) for grouping pixel ranges regarded as the same object existing in the image. And the process (object recognition process) which recognizes the classification of each object based on a captured image is performed using the information of the pixel range of each object obtained by this object detection process.
Corresponding point search corresponds to a certain pixel block (for example, a predetermined size such as 4 pixels or 4 pixels) on a reference image when two captured images obtained by stereo imaging are used as a reference image and a comparison image, respectively. This is a process of searching for a pixel block to be performed (a pixel block in which the same subject is projected: a corresponding point) on the comparison image. Specifically, this is a process of searching the comparison image for a pixel block having the largest correlation value with respect to the pixel block on the reference image.
Thus, the corresponding point search is relatively heavy because the correlation value is obtained while shifting the pixel block pixel by pixel on the comparative image.

  Here, in the case of distance measurement by the corresponding point search method, the parallax becomes larger as the object is closer. For this reason, in order to calculate the distance of a short-distance object, it is necessary to perform corresponding point search processing (correlation value calculation) in a wider range on the comparison image than in the case of calculating the distance of a long-distance object. is there. In consideration of this point, in order to measure a distance from a long distance to a short distance, it is desirable that the search width of the corresponding point is set wide enough to detect a short distance object.

However, if the search width is widened, the amount of calculation of correlation values increases, and the processing load required for corresponding point search increases.
Therefore, the present applicant has previously proposed a method of executing corresponding point search for a short-distance object based on a reduced image (see Patent Document 1 above). Specifically, in the method described in Patent Document 1, a corresponding point search for a long-distance object is performed with a normal search width based on a normal-size captured image (hereinafter referred to as “normal image”), and a short-distance object is detected. The corresponding point search is performed based on the 1/4 reduced image in which the number of vertical and horizontal pixels of the normal image is ½, respectively, with the normal search width.
Since the size of the subject is also reduced in the reduced image, when the corresponding point search is performed on the reduced image with the normal search width as described above, an effect equivalent to the expansion of the search range can be obtained, and the near-distance object can be obtained. Corresponding points can be detected. Since the size of the pixel block, which is the minimum unit of corresponding point search, is also reduced in accordance with the image reduction rate, the processing load for corresponding point search based on the reduced image is performed based on the normal image. The processing burden for a long-distance object can be reduced by the image reduction rate. Specifically, when the reduction ratio of the image is 1/2 · 1/2 = 1/4 as described above, the processing burden related to the corresponding point search for the short-distance object is the far distance based on the normal image. If the processing load related to the corresponding point search for the distance object is “1”, it can be suppressed to “¼”. In other words, the processing load related to the corresponding point search for detecting from a long-distance object to a short-distance object is “1.25”.
For example, in order to enable detection of an object within the same distance range as the method described in Patent Document 1 based only on a normal image, the search width is increased to twice the normal search width. When the search width is doubled, the processing load of the corresponding point search is “2”. From this point, it can be seen that according to the method described in Patent Document 1, the processing load related to the corresponding point search is greatly reduced (approximately 62% reduction from 1.25 / 2).

In a conventional image processing apparatus that performs object recognition processing, a corresponding point search for a long-distance object and a short-distance object is performed by the method described in Patent Document 1 described above. Mitigation is planned.
However, in the conventional image processing apparatus, as the processing based on the reduced image, the corresponding point search processing as described above, the object detection processing based on the processing result of the corresponding point search, and the object detected by the object detection processing Object recognition processing for recognizing the type is performed. Then, by using a result of a series of processing on the reduced image, a series of processing of a corresponding point search process based on a normal image, an object detection process, and an object recognition process is performed, and a short distance object and a long distance object are processed. Final distance information and object recognition result information are obtained.
As described above, in the conventional image processing apparatus, the same processing is performed in parallel for the normal image and the reduced image, and processing is wasted.
The present invention has been made in view of the above problems, and an object thereof is to reduce the total processing load required from the corresponding point search to the object recognition processing.

The image processing apparatus of the present invention includes an imaging unit that obtains a pair of image data by stereo imaging, and the pair of image data having a large image size and the image size based on the pair of image data obtained by the imaging unit. A large and small image acquisition unit that obtains the pair of small-size image data and a distance image by performing distance measurement by a corresponding point search method based on the pair of large-size image data, and based on the distance image A large object detection unit that groups pixel ranges that are regarded as the same object in the image, an object detection processing result by the large object detection unit, and the large-size image data acquired by the large and small image acquisition unit, Based on the object recognition unit for recognizing the type of the object detected by the large image object detection unit, and the distance image by performing distance measurement by the corresponding point search method based on the pair of image data of the small size And a small object detection unit that groups pixel ranges that are regarded as the same object in the image based on the distance image, and the large object detection unit is configured to perform a predetermined distance by the small object detection unit. When a corresponding object is detected, a corresponding point search by a predetermined search range and a corresponding point search by a search range changed from the predetermined search range are executed, and each corresponding point obtained by these corresponding point searches Based on the information, the generation of the distance image and the grouping based on the distance image are performed.
According to the present invention, the processing based on the small-size image data is stopped by the object detection processing as compared with the conventional image processing apparatus, so that the object recognition processing based on the small-size image data is omitted. The processing burden is reduced.

  According to the present invention, the total processing load required from the corresponding point search to the object recognition process can be reduced.

It is the figure which showed typically the system configuration for performing vehicle control. It is the figure which extracted and showed the structure of the part which concerns on the image process until object recognition in the system configuration | structure shown in FIG. It is explanatory drawing about a large size image and a small size image. It is the figure which showed typically the mode of the corresponding point search based on a large size image and a small size image. It is explanatory drawing of an object detection process. It is explanatory drawing about the change method of a search range. It is the flowchart which showed a series of processing procedures of image processing.

<1. Overall system configuration>
FIG. 1 schematically shows a system configuration that includes an image processing apparatus according to an embodiment of the present invention and performs vehicle control of a vehicle 1. This system includes an imaging device 2, an outside environment recognition device 3, a vehicle control device 4, a steering angle sensor 5, a vehicle speed sensor 6, an actuator 7, and a display 8 provided for the vehicle 1.
In the present system, the vehicle environment recognition device 3 recognizes an object existing in front of the vehicle based on the captured image obtained by the imaging device 2. In this system, the vehicle control device 4 avoids a collision with an object recognized by the vehicle exterior environment recognition device 3, and keeps a separation distance (inter-vehicle distance) from the object as the preceding vehicle at a safe distance. Execute control etc.

Specifically, the vehicle control device 4 includes information on the current traveling state of the vehicle 1 acquired through the steering angle sensor 5 that detects the steering angle and the vehicle speed sensor 6 that detects the traveling speed of the vehicle 1, and the outside environment recognition device. Based on the information about the object existing in front of the vehicle 1 detected in 3, the vehicle 1 is automatically maintained when the distance between the preceding vehicle and the preceding vehicle is kept safe or when a collision with the object is assumed. Control for causing the actuator 7 to brake is executed. The actuator 7 comprehensively represents an actuator for controlling a brake, a throttle valve, a steering angle, and the like for braking the vehicle 1.
Further, when a collision with an object is assumed, the vehicle control device 4 performs control so that a warning display to that effect is displayed on the display 8 disposed in front of the driver.
The vehicle control device 4 can also be formed integrally with the outside environment recognition device 3.

  The image processing apparatus according to the present embodiment is based on the captured image obtained by the imaging apparatus 2 and performs object detection processing for detecting the range of an object existing in the image (present in front of the vehicle 1) and the type of detected object. The object recognition process for recognizing the object is executed, and in the above-described system configuration, the vehicle environment recognition device 3 is included.

<2. Configuration related to image processing up to object recognition>
FIG. 2 shows an extracted internal configuration of the imaging device 2 shown in FIG. 1 and a configuration of a main part related to image processing up to object recognition out of the internal configuration of the outside-vehicle environment recognition device 3. In the figure, the vehicle control device 4 shown in FIG. 1 is also shown.
As illustrated, the imaging apparatus 2 includes a first camera unit 20-1, a second camera unit 20-2, an A / D converter 21-1, an A / D converter 21-2, an image correction unit 22, and an interface. A part (I / F part) 23 is provided.
Each of the first camera unit 20-1 and the second camera unit 20-2 includes a camera optical system and an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). A subject image is formed on the imaging surface of the imaging device by the optical system, and an electrical signal corresponding to the amount of received light is obtained in pixel units by the imaging device.

The first camera unit 20-1 and the second camera unit 20-2 are installed so as to enable distance measurement by a so-called stereo method. The first camera unit 20-1 and the second camera unit 20-2 in this example are arranged at a predetermined interval in the vehicle width direction in the vicinity of the upper part of the windshield of the vehicle 1. The optical axes of the first camera unit 20-1 and the second camera unit 20-2 are parallel.
In addition, the focal lengths of the first camera unit 20-1 and the second camera unit 20-2 are the same, and the frame periods are synchronized. The frame rate is 60 fps, for example.
The electrical signal obtained by the image sensor of the first camera unit 20-1 is sent to the A / D converter 21-1, and the electrical signal obtained by the image sensor of the second camera unit 20-2 is sent to the A / D converter 21. -2 and A / D conversion is performed respectively. As a result, a digital image signal (image data) representing a luminance value with a predetermined gradation in pixel units is obtained.

The image correction unit 22 includes image data based on an image captured by the first camera unit 20-1 obtained through the A / D converter 21-1 (hereinafter referred to as “first captured image data”), A Image data (hereinafter referred to as “second captured image data”) based on an image captured by the second camera unit 20-2 obtained via the / D converter 21-2 is input.
For example, the image correction unit 22 corrects a deviation caused by an error in the attachment positions of the first camera unit 20-1 and the second camera unit 20-2 for each of the first captured image data and the second captured image data. Use affine transformation or the like. The image correction unit 22 also corrects the luminance value including noise removal for each of the first captured image data and the second captured image data.

  The interface unit 23 is provided for exchanging various data with an external device. The first captured image data and the second captured image data corrected by the image correction unit 22 are transferred to the vehicle environment recognition apparatus 3 via the interface unit 23.

The outside environment recognition device 3 includes an interface unit 31, a bus 32, an image processing unit 33, and a memory unit 34. The interface unit 31, the image processing unit 33, and the memory unit 34 are respectively connected via the bus 32, and mutual data communication via the bus 32 is enabled.
The interface unit 31 is provided for exchanging various data with an external device. The first captured image data and the second captured image data transferred from the imaging device 2 side by the interface unit 23 are received by the interface unit 31. The first captured image data and the second captured image data received by the interface unit 31 in this way are held in the memory unit 34 via the bus 32 and are used for processing of the image processing unit 33.
In addition, information on the distance to the object and information on the type of the object obtained by the object detection process and object recognition process described later performed by the image processing unit 33 are transferred to the vehicle control device 4 via the interface unit 31. As can be understood from the description of FIG. 1, the vehicle control device 4 executes the above-described vehicle control based on the distance and object type information thus transferred.

The image processing unit 33 is configured by a semiconductor integrated circuit including, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory) as a work area, and follows a program stored in the ROM. Perform various processes.
In the case of the present embodiment, the image processing unit 33 performs the following processing based on the first captured image data and the second captured image data held in the memory unit 34 via the interface unit 31.
First, image reduction processing is performed to reduce the first captured image data and the second captured image data held in the memory unit 34 at a predetermined reduction ratio to obtain small-sized first captured image data and second captured image data. Run. By this image reduction processing, the memory unit 34 has the first captured image data and the second captured image data having the normal (before reduction) image size, and the first captured image data and the second captured image data having a small size. And are held. Hereinafter, the first captured image data and the second captured image data with the normal image size are also referred to as “large size images”, and the small captured first captured image data and the second captured image data are “small size”. Also referred to as “image”.

The image processing unit 33 performs the following processing based on the large size image and the following processing based on the small size image in parallel.
That is, a pixel range that is regarded as the same object on the basis of the distance image obtained by the large image distance image generation processing and the large image distance image generation processing by performing distance measurement based on the corresponding point search method based on the large size image Large image object detection processing for grouping and large image object recognition processing for recognizing the type of object detected in the large image object detection processing.
In addition, a pixel range that is regarded as the same object based on a small image distance image generation process that performs distance measurement by a corresponding point search method based on a small image and a distance image obtained by the small image distance image generation process And small object detection processing for grouping.

  In the case of this example, the above image reduction processing, large image distance image generation processing, large image object detection processing, large image object recognition processing, small image distance image generation processing, and small image object detection processing are the programs stored in the ROM. It is realized by software processing by the CPU according to the above. In FIG. 2, for the sake of convenience, it is assumed that there is hardware for executing these processes, and an image reduction processing unit 33A, a large image distance image generation processing unit 33B, a large image object detection processing unit 33C, The image object recognition processing unit 33D, the small image distance image generation processing unit 33E, and the small image object detection processing unit 33F are illustrated.

<3. Large image and small image>
First, a large size image and a small size image will be described with reference to FIG. FIG. 3A shows an example of first captured image data and second captured image data as large-size images, and FIG. 3B shows first captured image data and second captured images as small-size images obtained by image reduction processing. An example of data is shown.

First, as a premise, the corresponding point search method compares the corresponding points for each pixel block on the reference image T, with one of the pair of captured image data obtained by stereo imaging as the reference image T and the other as the comparison image C. This is a method of searching from each of the images C. In the case of this example, it is assumed that the first captured image data obtained by the first camera unit 20-1 is the reference image T, and the second captured image data obtained by the second camera unit 20-2 is the comparison image C. . The large size reference image T is denoted as a reference image T1, and the large size comparison image C is denoted as a comparison image C1. On the other hand, the reference image T obtained by the image reduction process is denoted as a reference image T2, and the comparison image C obtained by the image reduction process is denoted as a comparison image C2.
In the case of this example, the first camera unit 20-1 is arranged on the right side when the left and right are defined in the state of facing the traveling direction of the vehicle 1, and the second camera unit 20-2 is the left side according to the definition. Is arranged. Therefore, the reference image T can be rephrased as a right eye image and the comparative image C as a left eye image.

The comparison image C has a larger number of pixels in the horizontal direction than the reference image T, and the corresponding points of the pixel blocks located at the horizontal ends of the reference image T can be properly detected from the comparison image C. . As will be described later, if the maximum value of the parallax dp of the short-distance object is, for example, 256 pixels, the comparison image C uses an image in which the number of horizontal pixels is 256 pixels larger than that of the reference image T.
Although the image reduction ratio in the image reduction process is arbitrary, in this example, the reference image T2 and the comparison image C2 are reduced by reducing the number of horizontal pixels and the number of vertical pixels of the reference image T1 and the comparison image C1 to ½. Get. In this case, the image size is reduced to ¼ (reduction ratio = 1/4).
Note that a specific method of image reduction is not particularly limited, but an example is a thinning process.
The reference image T2 and the comparison image C2 obtained by the image reduction process are stored in the memory unit 34.

<4. Large stroke distance image generation processing and small stroke distance image generation processing>
Next, the large image distance image generation processing executed by the large image distance image generation processing unit 33B and the small image distance image generation processing executed by the small image distance image generation processing unit 33E will be described.
First, a corresponding point search process and a distance image generation process performed in common in the large image distance image generation process and the small image distance image generation process will be described.

The outline of the method for obtaining the parallax by the corresponding point search method is described below.
In the following description, the position of each pixel in the reference image T is represented by coordinates (i, j). The coordinates (i, j) represent the i and j coordinates of the pixel when the lower left corner of the reference image T is the origin, the horizontal direction (lateral direction) is the i coordinate axis, and the vertical direction (vertical direction) is the j coordinate axis. . For the comparison image C, the i-coordinate and the j-coordinate are similarly taken with the pixel previously associated with the origin of the reference image T as the origin.
In the following description, the luminance value for each pixel is represented by the luminance value p, the luminance value p of the pixel specified by the coordinates (i, j) in the reference image T is the luminance value pTij, and the coordinates (i in the comparative image C are , J) represents the luminance value p of the pixel specified by the luminance value pCij.

In the corresponding point search process, the reference image T is divided in units of pixel blocks composed of a predetermined plurality of pixels, for example, 4 or 4 pixels, and pattern matching processing with the comparison image C is performed in units of pixel blocks, so that the reference image A point corresponding to T and the comparison image C (a point where the same subject is considered to be projected: a corresponding point) is detected.
Specifically, the comparison image C is divided into horizontal lines of four pixels width extending in the horizontal direction, one pixel block of the reference image T is taken out, and the corresponding comparison image C is on the horizontal line (on the epipolar line). Are sequentially shifted in the horizontal direction (i direction) in units of one pixel, and the luminance values pTij of the 16 pixels in the pixel block of the reference image T and the luminance values pCij of the 16 pixels in the comparison image C corresponding thereto are obtained. get. Then, a pixel block on the horizontal line that minimizes the city block distance CB obtained by the following [Equation 1] obtained by summing the absolute values of the differences between the luminance value pTij of the reference image T and the luminance value pCij of the comparative image C is obtained. The pixel block (corresponding point) on the comparison image C having the luminance value characteristic closest to the pixel block of the reference image T is specified.
CB = Σ | pTij−pCij | [Formula 1]
The coordinate shift amount between the pixel block on the comparison image C specified in this way and the pixel block on the original reference image T is calculated, and the shift amount is obtained as the parallax dp. The value of the parallax dp is obtained for each detected corresponding point, and in the distance image generation process, the value of the parallax dp is held in correspondence with the position information of the corresponding point where the parallax dp is detected.

  The parallax dp determined as described above is the horizontal direction related to the mapping position of the same object in the reference image T and the comparison image C derived from the separation distance between the first camera unit 20-1 and the second camera unit 20-2. Represents the relative shift amount. Using such a value of the parallax dp, the distance from the center position of the first camera unit 20-1 and the second camera unit 20-2 to the subject as a corresponding point (distance in real space) is triangulated. Calculate based on the principle.

The calculation of the distance in the real space to the subject as the corresponding point can be performed as follows, for example.
First, as a premise, the vehicle width direction (horizontal direction) of the vehicle 1 is set to X with the point on the road surface located directly below the center position of the first camera unit 20-1 and the second camera unit 20-2 as the origin. A three-dimensional space is defined with the axis, the vehicle height direction as the Y axis, and the vehicle length direction (distance direction) as the Z axis.
In the three-dimensional space, the coordinates (X, Y, Z) of the position of the subject as the corresponding point for which the parallax dp is obtained are the coordinates on the image of the corresponding point for which the parallax dp is obtained (ic, jc), it is obtained by the following [Expression 2] to [Expression 4] using the calculated value of the parallax dp.
X = CD / 2 + Z · PW · (ic−IV) (Formula 2)
Y = CH + Z · PW · (jc−JV) (Formula 3)
Z = CD / (PW · (dp−DP)) (Formula 4)
However, CD is the arrangement interval between the first camera unit 20-1 and the second camera unit 20-2, PW is the viewing angle per pixel, and CH is the first camera unit 20-1 and the second camera unit 20-2. , IV and JV are the i-coordinate and j-coordinate on the infinity point image in front of the vehicle 1, and DP is the vanishing point parallax.
The distance to the subject as the corresponding point for which the parallax dp is calculated (hereinafter referred to as “distance L”) uses the calculated parallax dp value and the CD, PW, and DP values according to [Equation 4]. Can be calculated.

At this time, in the distance image generation process, a filtering process for the obtained parallax dp is performed for the purpose of improving the reliability of the parallax dp. For example, when a 4/4 pixel pixel block having only a feature image of a roadway is scanned on a horizontal line having a width of 4 pixels in the comparison image C, all the portions of the comparison image C where the roadway is imaged are correlated. Even if the corresponding pixel block is specified and the parallax dp is calculated, the reliability of the parallax dp is low. For this reason, a filtering process for invalidating such parallax dp is performed. That is, 0 (invalid value) is assigned as the value of the parallax dp with low reliability.
As a result of performing such filtering processing, the parallax dp is normally data having an effective value only for a so-called edge portion where the difference in luminance value pTij is large between pixels adjacent in the horizontal direction of the reference image T. Accordingly, the distance image obtained from the calculation result of the parallax dp is also data having an effective value only for the edge portion.

In the distance image generation process, for the target pixel block, the distance L is calculated from the calculated parallax dp value according to the previous [Expression 4], and the calculated distance L value is used as information on the position of the pixel block (coordinate ( i, j)) and stored in the memory unit 34.
In this way, when the information in which the value of the distance L is associated with the position information on the image is developed on the two-dimensional plane by the i-axis and the j-axis, the distance L in the real space to each corresponding point (each subject) is obtained. It will be represented on the image. In this sense, the information in which the value of the distance L is associated with the position information on the image as described above is called a “distance image”.

In the present embodiment, the corresponding point search process described above is performed with the search width fixed regardless of the large image / small image.
FIG. 4 shows a state of corresponding point search based on a large image performed in the large image distance image generation process (FIG. 4A) and a state of corresponding point search based on the small image performed in the small image distance image generation process (FIG. 4B). Is schematically shown.
As can be seen with reference to FIGS. 4A and 4B, the search width SW is the same in the corresponding point search processing based on the large size image and the corresponding point search processing based on the small size image. The search width SW means a width from the start position to the end position of the corresponding point search for a certain pixel block on the reference image T performed on the comparison image C.

By making the search width SW the same for the corresponding point search based on the large image and the corresponding point search based on the small image, the corresponding point search process based on the small image is more effective than the corresponding point search based on the large image. Corresponding points can be detected up to an object at a short distance (an object having a larger parallax dp). If the search width SW is increased, corresponding points can be detected up to an object closer to that distance. However, the processing load increases, so it is desirable that the search width SW is smaller. As an example, in this example, the camera parameters in the imaging device 2 are the arrangement interval between the first camera unit 20-1 and the second camera unit 20-2 = 350 mm, the first camera unit 20-1 and the second camera unit 20 When the focal length of -2 is 8 mm and the amount of deviation per pixel is 0.006 mm, the search width SW is set to 128 pixels. When the corresponding point search based on the large size image is performed with the search width SW = 128 pixels under the above camera parameters, the detectable distance L is 350... 8 mm / 128 pixels / 0.006 mm = 3646. ≒ 3.6m or more.
Therefore, when a corresponding point search is performed with the same search width SW for a small-size image, it is possible to detect corresponding points for a short-distance object of less than 3.6 m. For example, the image reduction ratio is 1/4 as described above. If there is, a corresponding point can be detected up to a short distance object of 3.6 m / 2 = 1.8 m.

At this time, in the corresponding point search performed in the small image distance image generation process, the corresponding point is searched in units of pixel blocks reduced in accordance with the image reduction rate of the small size image. For example, when the image reduction ratio is ¼ as described above and the size of the pixel block corresponding to the large-size image is the aforementioned 4 · 4 = 16 pixels, the size of 2 · 2 = 4 pixels. The corresponding point search is performed in units of pixel blocks.
In this respect, the processing load of the corresponding point search for the short-distance object performed based on the small size image can be suppressed to ¼ in comparison with the corresponding point search performed based on the large size image.

<5. Object detection processing>
Next, the large image object detection process executed by the large image object detection processing unit 33C and the small image object detection process executed by the small image object detection processing unit 33F will be described.
With reference to FIG. 5, the object detection process performed in each of the large image object detection process and the small image object detection process will be described. In the large object detection process, the object detection process described below is executed for the distance image generated in the large image distance image generation process, and in the small object detection process, the distance generated in the small image distance image generation process. The object detection process described below is executed for the image as a processing target.

First, in the object detection process, the distance image to be processed is divided into a plurality of areas (hereinafter referred to as “vertical area VR”) by dividing lines along the vertical direction of the image as shown in FIG. 5A. Then, for each vertical region VR, a distance histogram representing the distance distribution in the depth direction is created from the distance data existing in the vertical region VR, and the distance of the position (corresponding point) at which the frequency is maximum is indicated in the vertical region VR. The representative distance of an object existing in FIG. 5B schematically represents the representative distance obtained by such processing for each vertical region VR by a black circle.
Next, for each corresponding point with the maximum frequency for which the representative distance is obtained, pixel ranges that are considered to be the same object are grouped from the relationship such as the distance to each corresponding point and the direction. FIG. 5C schematically shows that the three preceding vehicles existing in the image shown in FIG. 5A are grouped as groups G1, G2, and G3, respectively.
In this way, each object existing in the captured image is detected including information on the distance to the object.

<6. Large object recognition processing>
Next, the large object recognition process executed by the large object recognition processing unit 33D will be described.
The large object recognition process is a process for recognizing the type of an object detected in the large object detection process based on the object detection process result by the large object detection process and the captured image data (imaging source image) as a large image. It is. Specifically, this is processing for recognizing the type of the detected object, such as whether the detected object is a preceding vehicle or an obstacle such as a pedestrian.
This object recognition processing can be performed, for example, based on the result of preparing a representative image pattern of the object for each type of object to be recognized and matching the image pattern. Alternatively, for example, a method of recognizing which object type corresponds to the result of determining whether or not a characteristic pattern exists in the object such as a brake lamp in a vehicle can be adopted.
Various methods for object recognition processing can be adopted, and should not be limited to specific methods.

<7. Correspondence processing when detecting short-range objects>
Next, a description will be given of the corresponding processing at the time of short-distance object detection executed by the large-field-distance image generation unit 33B.
In the case of the present embodiment, in the above-described large-field-distance image generation processing, corresponding point search processing based on a predetermined search range in response to the detection of a short-distance object in the previous small-image object detection processing, The corresponding point search process is executed using the search range changed from the search range.
For this reason, in the present embodiment, it is determined whether or not an object within a predetermined distance has been detected in the small object detection process executed by the small object detection processing unit 33F. In this example, it is determined whether or not a short-distance object having a distance L of less than 3.6 m is detected under the above-described camera parameter conditions.
When it is determined that a short-distance object has been detected, a predetermined search is performed as a corresponding point search process based on the reference image T1 and the comparison image C1 of the frame next to the current processing target frame in the large-range image generation process. Corresponding point search processing based on a range and corresponding point searching processing based on a search range changed so that a near-distance object can be detected are executed.

FIG. 6 is an explanatory diagram of the search range changing method, in which the reference image T1 is shown in the upper part and the comparison image C1 is shown in the lower part.
The pixel block Rp in the figure means a pixel block on the reference image T1 for which the corresponding point should be searched from the comparison image C1.
In a normal corresponding point search process, that is, a corresponding point search process executed when a short-distance object is not detected, a search start position (see FIG. 5) for a pixel block located at the same coordinate as the pixel block Rp on the comparison image C1. (Indicated as “Ss”), a corresponding point search is performed with a constant search width SW. According to this normal corresponding point search process, only the corresponding point for an object having a distance L of 3.6 m or more is correctly detected under the setting of the previous camera parameter and search width SW = 128 pixels.
On the other hand, when a short-distance object is detected, together with the normal corresponding point search process described above, from the position Ss ′ offset in the horizontal direction from the normal search start position Ss by the offset of as shown in the figure, Corresponding point search is performed with a constant search width SW. Thereby, it is possible to detect a short-range object that cannot be properly detected in the normal search range without increasing the search width SW. That is, it is possible to detect a short-distance object without increasing the processing load in comparison with a normal corresponding point search.
In the case of this example, the offset of is set to 128 pixels under the setting of the previous camera parameter and search width SW = 128 pixels. Thereby, the corresponding point about the object whose distance L is less than 3.6 m can be detected.

As described above, in the present embodiment, as the corresponding point search based on the large size image, the corresponding point search based on the large size image is performed by executing the corresponding point search based on the search range changed from the predetermined search range. Can detect points with high resolution. That is, as a result, the distance L for a short-distance object can be calculated more accurately than when it is obtained based on a small-size image.
Note that, for the sake of confirmation, the “search range” referred to here is the reference for searching for a corresponding point of a certain pixel block (denoted as “reference pixel block”) on the reference image T. It means a range on the comparison image C in which pattern matching with the pixel block is performed.

  In the large-field-distance image generation processing according to the present embodiment, a distance image is generated including the corresponding points for the short-distance objects detected by the corresponding processing at the time of short-distance object detection as described above. Then, in the large image object detection process executed after the large image distance image generation process, the object detection process is performed based on the distance image generated including the information on the distance L for the short distance object, In the large object recognition process, an object recognition process is performed on a short-distance object detected by the object detection process.

<8. Series of processing steps>
Next, a series of processing procedures of the image processing according to the embodiment described above will be described with reference to the flowchart of FIG.
In the figure, the process “reduced image processing” represents a process executed based on the reference image T2 and the comparison image C2 as a small-size image obtained by the previous image reduction process, and is denoted “rear-large image process”. The process represents a process to be executed based on the reference image T1 as a large size image and the comparison image C1. The “versus-reduced image processing” and “to-large image processing” are executed as parallel processes, and these processes are repeatedly executed for each frame.

First, “versus-reduced image processing” will be described.
As the “versus-reduced image processing”, first, the CPU (the CPU of the image processing unit 33) generates a distance image by the corresponding point search method based on the reference image T2 and the comparative image C2 as a small size image (step S101). That is, the corresponding point search based on the reference image T2 and the comparative image C2 is performed with the search width SW to detect the corresponding point and obtain the parallax dp for each corresponding point. From the result, the distance L in the real space for each corresponding point is obtained. A distance image is calculated and the distance L for each corresponding point is associated with the position on the image.
Next, pixel ranges regarded as the same object are grouped based on the generated distance image (step S102). In addition, since the specific processing content of the grouping has already been described, redundant description is avoided.

Next, it is determined whether or not a short distance object has been detected (step S103). That is, in this example, it is determined whether or not an object having a distance L of less than 3.6 m is detected. If it is determined in step S103 that a short-distance object has not been detected, the process proceeds to step S104 and flag = 0 is set.
On the other hand, if it is determined in step S103 that a short-distance object has been detected, the process proceeds to step S105, where flag = 1 is set.
The flag is information managed by the CPU to identify whether or not a near-distance object is detected. “0” indicates that no near-distance object is detected and “1” indicates that a near-distance object is detected. The initial value of flag is “0”.
After setting flag in step S104 or step S105, the process returns to the previous step S101.

Next, “large image processing” will be described.
First, a corresponding point search is performed based on the reference image T1 as a large size image and the comparison image C1 (step S201). That is, the corresponding point search is performed in the normal search range by the search width SW.
Next, it is determined whether or not flag = 1 (S202). If it is determined in step S202 that flag = 1, that is, if it is determined that a short-distance object has been detected, the process proceeds to step S203, the search range is changed, and a corresponding point search based on the large-size image is performed again. Proceed to step S204. As described above, the change of the search range in this example is realized by setting the search width SW to be the same as the normal time and offset the search start position in the horizontal direction by the offset of from the normal search start position. .

On the other hand, if it is determined in step S202 that flag = 0, that is, if it is determined that a short-distance object has not been detected, the process proceeds to step S204 without performing the process in step S203.
In step S204, a distance image is generated based on the result of calculating the parallax dp for each corresponding point. In subsequent step S205, pixel ranges that are regarded as the same object are grouped based on the generated distance image. Further, in the subsequent step S206, object recognition processing is performed based on the grouping result and the captured image, and the process returns to the previous step S201.
If a near-distance object is detected in the object detection process based on the small-size image by the above process, the distance detection of both the short-distance object and the long-distance object based on the large-size image, the object detection process, and the object recognition process Is done.

<9. Summary>
As described above, in the present embodiment, object detection processing based on small-size image data is performed. As a result, when a short-distance object within a predetermined distance is detected, it is based on large-size image data. In the object detection process, a corresponding point search using a predetermined search range and a corresponding point search using a search range changed from the predetermined search range are performed. Then, object detection processing and object recognition processing are performed based on information on corresponding points for both a long-distance object and a short-distance object obtained by these two corresponding point searches.
Since the processing based on the small-size image data has been stopped by the object detection processing, the object recognition processing that the conventional image processing apparatus has performed based on the small-size image data is omitted, and the processing burden is reduced accordingly. Is done. Therefore, it is possible to reduce the total processing load (total processing load required from the corresponding point search to the object recognition processing) required to obtain the final object recognition processing result based on the large-size image data.

In the present embodiment, the search range is changed when a short-distance object is detected by offsetting the search start position in the horizontal direction in comparison with the corresponding point search by the normal search range. .
Therefore, even when the same search width SW as that in the normal corresponding point search is set, an object at a closer distance can be detected. Further, even if a search width SW that is narrower than the originally required search width SW is set, the corresponding points can be detected, so that the processing load related to the corresponding point search can be reduced.

<10. Modification>
The present invention should not be limited to the specific examples described above, and various modifications can be considered.
For example, in the description so far, the example of changing the search range of the corresponding point search performed for the large-size image when a short-distance object is detected is performed by offsetting the search start position. May be performed by enlarging the search width SW.
In addition, regarding the method of offsetting the search start position, there is no need to fix the search width SW to the same width as in the normal time as exemplified above, and a search width SW different from the normal time may be set.

In addition, in the corresponding point search for a short-distance object based on a large-size image, in order to further reduce the processing burden, an object (see FIG. 5) that detects a pixel range to be searched by a small object detection process. It is also possible to limit the pixel range to the pixel range of group G). Specifically, assuming that the pixel range in which the close-range object is detected is “area N”, the corresponding point search for the close-range object based on the large-size image is performed only for the pixel range corresponding to the area N on the reference image T1. To target.
Normally, the corresponding point search is performed on the entire pixel range in the reference image T1, but if the pixel range on which the corresponding point search is performed in this way is limited to a range corresponding to the pixel range in which the short-range object is detected, processing is performed. The burden can be greatly reduced.

  1. Vehicle, 2. Imaging device, 3. External environment recognition device, 31. Interface (I / F) unit, 32. Bus, 33. Image processing unit, 33A. Image reduction processing unit, 33B. Large image distance image generation. Processing Unit, 33C / Large Object Detection Processing Unit, 33D / Large Object Recognition Processing Unit, 33E / Small Image Distance Image Generation Processing Unit, 33F / Small Object Detection Processing Unit

Claims (3)

An imaging unit for obtaining a pair of image data by stereo imaging;
Based on the pair of image data obtained by the imaging unit, a large and small image acquisition unit that obtains the pair of image data having a large image size and the pair of image data having a small image size;
Based on the pair of image data of the large size, a distance image is generated by performing a distance measurement by a corresponding point search method, and a large image grouping pixel ranges regarded as the same object in the image based on the distance image An object detection unit;
An object recognition unit for recognizing the type of the object detected by the large image object detection unit based on the object detection processing result by the large image object detection unit and the large-size image data acquired by the large and small image acquisition unit; ,
Based on the pair of image data of the small size, a distance image is generated by performing distance measurement by a corresponding point search method, and based on the distance image, a small image that groups pixel ranges regarded as the same object in the image An object detection unit,
The large image object detection unit is configured to search for corresponding points based on a predetermined search range and corresponding points based on a search range changed from the predetermined search range when an object within a predetermined distance is detected by the small image object detection unit. An image processing apparatus that performs a search and generates the distance image and performs the grouping based on the distance image based on information on each corresponding point obtained by the corresponding point search. The large object detection unit
The image processing apparatus according to claim 1, wherein as the corresponding point search by the search range changed from the predetermined search range, a search in which a search start position is offset in a horizontal direction compared to the corresponding point search by the predetermined search range is performed. The large object detection unit
The corresponding point search by the search range changed from the normal search range is limited to the pixel range of the object within the predetermined distance grouped by the small object detection unit. An image processing apparatus according to 1.

Images