JP6753134B2 - Image processing device, imaging device, mobile device control system, image processing method, and image processing program - Google Patents

Description

The present invention relates to an image processing device, an imaging device, a mobile device control system, an image processing method, and an image processing program.

Today, there is known a technique for detecting an object such as a person or an automobile at high speed by measuring a distance with a millimeter wave radar, a laser radar, or a stereo camera. For example, when detecting the three-dimensional position and size of an object such as a person or a preceding vehicle by distance measurement with a stereo camera, the parallax of the object is interpolated, the position of the road surface is detected, and the object is in contact with the road surface. Detects existing objects. The detection output of such an object is used for automatic brake control, automatic steering wheel control, and the like.

Patent Document 1 (Japanese Unexamined Patent Publication No. 2014-096005) discloses an object detection device that accurately groups and detects objects that are considered to be the same. This object detection device groups effective regions with each other using a distance image, and then divides the regions by focusing on the contour portion (edge) of the shaded image with respect to the grouped objects.

Patent Document 2 (International Publication No. 2012/017650) discloses an object detection device capable of accurately detecting an object and a road surface. After detecting the road surface region, this object detection device detects data having a height higher than the road surface as an object candidate region, and discriminates between the object and the road surface based on the shape feature.

However, the object detection device disclosed in Patent Document 1 has a problem that it is difficult to correctly separate objects grouped in one region when they have complicated edges in a grayscale image. Further, the object detection device disclosed in Patent Document 2 has a problem that adjacent objects are combined and detected in a region where an object above the road surface can exist. For this reason, the object detection device of Patent Document 1 and the object detection device of Patent Document 2 have a problem that it is difficult to detect an object with high accuracy.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing device, an imaging device, a mobile device control system, an image processing method, and an image processing program capable of accurately detecting an object. ..

In order to solve the above-mentioned problems and achieve the object, the present invention has a reference detection unit that detects a reference object as a reference for the height of an object from a distance image provided with distance information for each pixel, and a reference object. Distance information of each pixel with respect to the map of the height range corresponding to the height from the reference object in each pixel of the distance image among a plurality of maps in which the height range from is set to a different height range. A map generation unit that votes for, a set detection unit that detects the set area of distance information from each map, and a classification part that classifies the types of objects in the set area based on the distribution of the set area in each map. The map generation unit generates a two-dimensional histogram generated with the actual distance to the object on the distance image as the horizontal axis and the distance information of the distance image as the vertical axis as a map .

According to the present invention, there is an effect that an object can be detected with high accuracy.

FIG. 1 is a schematic diagram showing a schematic configuration of a mobile control system according to an embodiment. FIG. 2 is a block diagram showing a schematic configuration of an imaging unit and an analysis unit. FIG. 3 is a diagram showing the positional relationship between the subject and the imaging lens of each imaging unit on the XZ plane. FIG. 4 is a functional block diagram of various functions of the mobile control system of the embodiment. FIG. 5 is a diagram showing an captured image captured by the imaging unit. FIG. 6 is a diagram showing a V map corresponding to the captured image of FIG. FIG. 7 is a diagram showing an example of a captured image. FIG. 8 is a diagram showing a U-map of the frequency corresponding to the captured image of FIG. 7. FIG. 9 is a diagram showing a U map having a height corresponding to the captured image of FIG. 7. FIG. 10 is a diagram showing a real U map corresponding to the captured image of FIG. 7. FIG. 11 is a diagram showing an example of a captured image. FIG. 12 is a diagram showing a real U map corresponding to the captured image of FIG. FIG. 13 is a diagram showing a layer U map of a layer of an object having a height from the road surface of 60 cm to 200 cm, which corresponds to the captured image of FIG. FIG. 14 is a diagram showing a layer U map of a layer of an object having a height from the road surface of 150 cm to 200 cm, which corresponds to the captured image of FIG. FIG. 15 is a flowchart for explaining the layer U map generation operation. FIG. 16 is a diagram showing an example of a captured image. FIG. 17 is a diagram showing a real U map corresponding to the captured image of FIG. FIG. 18 is a diagram showing a layer U map of a layer of an object having a height from the road surface of 60 cm to 200 cm, which corresponds to the captured image of FIG. FIG. 19 is a diagram showing a layer U map of a layer of an object having a height from the road surface of 150 cm to 200 cm, which corresponds to the captured image of FIG. FIG. 20 is a diagram showing a layer U map of a layer of an object having a height of 200 cm or more from the road surface, which corresponds to the captured image of FIG. FIG. 21 is a flowchart showing the flow of the isolated region detection process. FIG. 22 is a schematic diagram for explaining the labeling method. FIG. 23 is another schematic diagram for explaining the labeling method. FIG. 24 is still another schematic diagram for explaining the labeling method. FIG. 25 is a diagram showing a layer U map in which a distant road surface is detected lower than the actual height and the road surface and the vehicle are detected as one object. FIG. 26 is a diagram showing a real U map having a height corresponding to FIG. 25. FIG. 27 is a flowchart showing the flow of the peripheral area removal process. FIG. 28 is a diagram for explaining a region removed by the peripheral region removal process. FIG. 29 is a flowchart showing the flow of the lateral separation process. FIG. 30 is a diagram for explaining the calculation process of the evaluation value in the horizontal direction. FIG. 31 is a diagram for explaining the binarization process of the evaluation value. FIG. 32 is a diagram showing a layer U map after the smoothing process. FIG. 33 is a flowchart showing the flow of the vertical separation process. FIG. 34 is a diagram for explaining the calculation operation of the evaluation value of each line. FIG. 35 is a diagram for explaining an operation of detecting the actual width of the object from the line having the maximum evaluation value. FIG. 36 is a diagram for explaining the calculation operation of the separation boundary of the object. FIG. 37 is a diagram for explaining the vertical separation position of the object on the real U map. FIG. 38 is a flowchart showing the flow of the isolated region detection process and the object type classification process. FIG. 39 is a diagram showing an example of a captured image. FIG. 40 is a diagram showing a layer U map of a layer of an object having a height from the road surface of 60 cm to 200 cm, which corresponds to the captured image of FIG. 39. FIG. 41 is a diagram showing a layer U map of a layer of an object having a height from the road surface of 150 cm to 200 cm, which corresponds to the captured image of FIG. 39. FIG. 42 is a diagram showing a layer U map of a layer of an object having a height of 200 cm or more from the road surface, which corresponds to the captured image of FIG. 39. FIG. 43 is a diagram showing a layer U map in a state in which the layers of FIGS. 40 to 42 are superimposed. FIG. 44 is a diagram showing a real U map of frequency. FIG. 45 is a diagram for explaining an operation of determining the circumscribed rectangle of the object line group as an object area in the parallax image. FIG. 46 is a flowchart showing the flow of the detection process of the object area (corresponding area) in the parallax image.

Hereinafter, the mobile control system of the embodiment will be described in detail with reference to the accompanying drawings. FIG. 1 is a schematic diagram showing a schematic configuration of a mobile control system according to an embodiment. As shown in FIG. 1, the mobile body control system is provided in a vehicle 1 such as an automobile, which is an example of a mobile body. The mobile control system includes an imaging unit 2, an analysis unit 3, a control unit 4, and a display unit 5.

The image pickup unit 2 is provided near the rearview mirror of the windshield 6 of the vehicle 1 and captures an image of the vehicle 1, for example, the traveling direction. Various data including the image data obtained by the imaging operation of the imaging unit 2 are supplied to the analysis unit 3. The analysis unit 3 analyzes the recognition target such as the road surface on which the vehicle 1 is traveling, the vehicle in front of the vehicle 1, pedestrians, obstacles, etc., based on various data supplied from the imaging unit 2. Based on the analysis result of the analysis unit 3, the control unit 4 gives a warning to the driver of the vehicle 1 via the display unit 5. Further, the control unit 4 performs running support such as control of various in-vehicle devices, steering wheel control of the vehicle 1, or brake control based on the analysis result.

FIG. 2 is a schematic block diagram of the imaging unit 2 and the analysis unit 3. As shown in FIG. 2, the imaging unit 2 has, for example, a stereo camera configuration including two imaging units 10A and 10B. The two imaging units 10A and 10B each have the same configuration. Specifically, the image pickup units 10A and 10B include image pickup lenses 11A and 11B, image sensors 12A and 12B in which light receiving elements are two-dimensionally arranged, and controllers 13A and 13B for image pickup driving of the image sensors 12A and 12B. doing.

The analysis unit 3 has an FPGA (Field-Programmable Gate Array) 14, a RAM (Random Access Memory) 15, and a ROM (Read Only Memory) 16. Further, the analysis unit 3 has a CPU (Central Processing Unit) 17, a serial interface (serial IF) 18, and a data IF 19. The FPGA 14 to the data IF 19 are connected to each other via the data bus line 21 of the analysis unit 3. Further, the imaging unit 2 and the analysis unit 3 are connected to each other via the data bus line 21 and the serial bus line 20.

The RAM 15 stores parallax image data and the like generated based on the luminance image data supplied from the imaging unit 2. Various programs such as an operation system and an object detection program are stored in the ROM 16.

Further, the FPGA 14 uses one of the captured images captured by the imaging units 10A and 10B as a reference image and the other as a comparative image. Then, the FPGA 14 calculates the amount of positional deviation between the corresponding image portion on the reference image and the corresponding image portion on the comparison image corresponding to the same point in the imaging region as a parallax value (parallax image data) of the corresponding image portion. ..

FIG. 3 shows the positional relationship between the subject 30 and the imaging lenses 11A and 11B of the imaging units 10A and 10B on the XZ plane. In FIG. 3, the distance b between the imaging lenses 11A and 11B and the focal length f of the imaging lenses 11A and 11B are fixed values, respectively. Further, the amount of deviation of the X coordinate of the image pickup lens 11A with respect to the gazing point P of the subject 30 is set to Δ1. Further, the amount of deviation of the X coordinate of the image pickup lens 11B with respect to the gazing point P of the subject 30 is set to Δ2. In this case, the FPGA 14 calculates the parallax value D, which is the difference between the X coordinates of the imaging lenses 11A and 11B with respect to the gazing point P of the subject 30, by the following equation (1).

Parallax value D = | Δ1-Δ2 | ... (Equation 1 formula)

The FPGA 14 of the analysis unit 3 performs processing that requires real-time performance, such as gamma correction processing and distortion correction processing (parallelization of left and right captured images), on the luminance image data supplied from the imaging unit 2. Further, the FPGA 14 generates parallax image data (parallax value D) by performing the above-mentioned calculation of the equation 1 using the luminance image data that has been subjected to such processing that requires real-time performance, and causes the RAM 15 to generate the parallax image data. Write.

The CPU 17 operates based on the operation system stored in the ROM 16 and controls the overall imaging of the imaging units 10A and 10B. Further, the CPU 17 loads the object detection program from the ROM 16 and executes various processes using the parallax image data written in the RAM 15. Specifically, the CPU 17 obtains CAN (Controller Area Network) information such as vehicle speed, acceleration, steering angle, and yaw rate from each sensor provided in the vehicle 1 based on the object detection program via the data IF19. Refers to, and performs recognition processing of recognition objects such as road surfaces, guard rails, vehicles, and humans, parallax calculation, calculation of the distance to recognition objects, and the like.

Here, the parallax value D can also be calculated by the following equation 2 with the distance from each of the imaging lenses 11A and 11B shown in FIG. 3 to the subject 30 as “Z”.

Parallax value D = (b × f) / Z ... (Equation 2 formula)

As can be seen from the equation (2), the distance Z from each of the imaging lenses 11A and 11B to the subject 30 can be calculated by the following equation (3) using the calculated parallax value D.

Z = (b × f) / D ... (Equation 3 formula)

The CPU 17 calculates the distance Z from the authentication target by using the parallax value D supplied from the image pickup unit 2.

The CPU 17 supplies the processing result to the control unit 4 shown in FIG. 1 via the serial IF18 or the data IF19. The control unit 4 performs, for example, brake control, vehicle speed control, handle control, and the like based on the data resulting from the processing. Further, the control unit 4 displays a warning or the like on the display unit 5 based on the data as the processing result. As a result, the driving support of the driver of the vehicle 1 can be provided.

Next, when detecting an object to be recognized using three-dimensional information (spatial information) in the horizontal direction (width direction), height direction, and depth direction, information in each dimension (each information in width, height, and depth) It is important to use. Conventionally, a recognition target is detected using a two-dimensional map (Real U-Disparity map) in which the horizontal axis (X axis) of the coordinates is the actual distance and the vertical axis (Y axis) of the coordinates is the parallax value. There is. Hereinafter, the "Real U-Disparity map" is simply referred to as a "real U map".

However, in the conventional method, the height information of the recognition object from the ground parallel to the vertical axis (Y axis) of the camera coordinates is lacking in the real U map. Therefore, when a plurality of recognition objects are located adjacent to each other, there are many inconveniences that the plurality of recognition objects are erroneously detected as one recognition object. The occurrence / non-occurrence of such false positives depends on the movement (image) of the recognition object. Therefore, it is difficult to accurately detect the recognition target by the conventional method.

On the other hand, the mobile control system of the embodiment creates a plurality of real U maps with a layered structure (height range) based on the height of the recognition object from the road surface. That is, when voting the parallax points in the parallax image on the real U map, the real U map of each height range is created by voting for the real U map of the preset height range. Hereinafter, the U map of each height is referred to as a "layer U map". Then, the mobile control system of the embodiment performs grouping processing of two-dimensional planes on each layer U map of each height range, and integrates or separates the grouping results.

That is, the parallax points on the parallax image obtained from the stereo image are voted on the layer U map of the height prepared in advance based on the actual height from the road surface, and grouped by the two-dimensional plane in each layer U map. Process and integrate or separate each grouping result. As a result, the recognition target can be detected with high accuracy by efficiently using the height information at high speed. The mobile control system of the embodiment is capable of detecting a human being with high accuracy.

Hereinafter, the recognition operation of the recognition target object in the mobile control system of such an embodiment will be specifically described. FIG. 4 is a functional block diagram of various functions of the mobile control system of the embodiment. In FIG. 4, the parallelized image generator 41 and the parallax image generator 42 are implemented in hardware by the programmable logic blocks of the FPGA 14. In this example, the parallelized image generation unit 41 and the parallax image generation unit 42 are realized by hardware, but may be realized by software by an object detection program or the like described below.

Further, in FIG. 4, the V-map generation unit 45 to the three-dimensional position determination unit 54 is realized by software when the CPU 17 executes an object detection program stored in the ROM 16. That is, by executing the object detection program, the CPU 17 executes the V map generation unit 45, the road surface shape detection unit 46, the list generation unit 47, the U map generation unit 48, the layer U map generation unit 49, and the isolated area detection unit 50. It functions as a classification unit 51, a corresponding area detection unit 52, an area extraction unit 53, and a three-dimensional position determination unit 54.

In this example, the V-map generation unit 45 to the three-dimensional position determination unit 54 are realized by the CPU 17 by software, but a part or all of them are realized by hardware such as an IC (Integrated Circuit). May be good. In addition, the object detection program is a file in an installable format or an executable format that can be read by a computer device such as a CD-ROM, flexible disk (FD), CD-R, DVD, Blu-ray disk (registered trademark), or semiconductor memory. It may be recorded and provided on a possible recording medium. DVD is an abbreviation for "Digital Versatile Disk". Further, the object detection program may be provided in the form of being installed via a network such as the Internet. Further, the object detection program may be provided by incorporating it into a ROM or the like in the device in advance.

In such a mobile control system, the imaging units 10A and 10B of the imaging unit 2 constituting the stereo camera generate luminance image data. Specifically, when each imaging unit 10A and 10B has a color specification, each imaging unit 10A and 10B performs the following calculation of the following equation 4 to obtain brightness (Y) from each RGB (red, green, and blue) signal. Performs color luminance conversion processing to generate a signal. Each of the imaging units 10A and 10B supplies the luminance image data generated by the color luminance conversion process to the parallelized image generation unit 41 of the FPGA 14.

Y = 0.3R + 0.59G + 0.11B ... (Equation 4 formula)

The parallelization image generation unit 41 performs parallelization processing on the luminance image data supplied from the imaging unit 2. This parallelization process is a process of converting the luminance image data output from the imaging units 10A and 10B into an ideal parallelized stereo image obtained when two pinhole cameras are mounted in parallel. Specifically, the amount of distortion of each pixel is output from each imaging unit 10A and 10B using the calculation result calculated using the polynomials Δx = f (x, y) and Δy = g (x, y). Each pixel of the luminance image data to be generated is converted. The polynomial is based on, for example, a sixth degree polynomial with respect to x (horizontal position of the image), y (vertical position of the image). As a result, it is possible to obtain a parallel luminance image in which the distortion of the optical system of the imaging units 10A and 10B is corrected.

Next, the parallax image generation unit 42 generates a parallax image having a parallax value for each pixel, which is an example of a distance image having distance information for each pixel. That is, the parallax image generation unit 42 uses the brightness image data of the imaging unit 10A as the reference image data, the brightness image data of the imaging unit 10B as the comparison image data, and performs the calculation shown in the above equation 1 to perform the reference image. Data and comparison Image data that shows the difference between the image data and the data is generated. Specifically, the parallax image generation unit 42 defines a block composed of a plurality of pixels (for example, 16 pixels × 1 pixel) centered on one pixel of interest for a predetermined “row” of the reference image data. On the other hand, in the same "row" in the comparative image data, a block having the same size as the defined reference image data block is shifted one pixel at a time in the horizontal line direction (X direction). Then, the parallax image generation unit 42 obtains a correlation value indicating the correlation between the feature amount indicating the characteristics of the pixel value of the block defined in the reference image data and the feature amount indicating the characteristics of the pixel value of each block in the comparison image data. Calculate each.

Further, the parallax image generation unit 42 performs a matching process of selecting the block of the comparison image data that has the most correlation with the block of the reference image data among the blocks of the comparison image data based on the calculated correlation value. After that, the amount of positional deviation between the attention pixel of the reference image data block and the corresponding pixel of the comparison image data block selected by the matching process is calculated as the parallax value D. Parallax image data is obtained by performing such a process of calculating the parallax value D for the entire area of the reference image data or a specific area. The parallax image data is a set of three-dimensional data having x-coordinates and y-coordinates which are image coordinates and a parallax value D. Hereinafter, the term “image” will be used, but it does not have to be actually configured as an image, and will be described as representing a set of three-dimensional data.

As the feature amount of the block used for the matching process, for example, the value (luminance value) of each pixel in the block can be used. Further, as the correlation value, for example, the difference between the value (luminance value) of each pixel in the block of the reference image data and the value (luminance value) of each pixel in the block of the comparative image data corresponding to each of these pixels. The sum of the absolute values of can be used. In this case, the block with the smallest sum is detected as the most correlated block.

Examples of the matching process of the parallax image generation unit 42 include SSD (Sum of Squared Difference), ZSD (Zero-mean Sum of Squared Difference), SAD (Sum of Absolute Difference), or ZSAD (Zero-mean Sum). A method such as of Absolute Difference) can be used. If a subpixel level parallax value of less than one pixel is required in the matching process, an estimated value is used. As a method for estimating the estimated value, for example, an isometric straight line method or a quadratic curve method can be used. However, an error occurs in the estimated subpixel level parallax value. Therefore, a method such as EEC (estimation error correction) that reduces the estimation error may be used.

In the present embodiment, since the parallax value and the distance value can be treated equivalently, the parallax image is shown as an example of the distance image, but the present invention is not limited to this. For example, a distance image may be generated by fusing the distance information of a millimeter wave radar or a laser radar with a parallax image generated by a stereo camera.

Next, in the set of parallax images (x-coordinate value, y-coordinate value, parallax value D), a two-dimensional histogram is created in which the X-axis is the parallax value D, the Y-axis is the y-coordinate value, and the Z-axis is the frequency. think of. Hereinafter, this two-dimensional histogram is referred to as a "V-Disparity map (V map)". For example, as shown in FIG. 5, the V-map generation unit 45 has a utility pole 62 on the left side of the vehicle 61 traveling on a flat road surface 60 extending in the center of the screen, and each pixel of the parallax image of the image captured by the imaging unit 2. Among them, the pixel having the parallax value D and the y coordinate value is given a frequency (frequency) counted up by one. Then, the V-map generation unit 45 votes each pixel on a two-dimensional histogram with the X-axis as the differential value D, the Y-axis as the y-coordinate value, and the Z-axis as the frequency, so that the right side is shown in FIG. A V-map in which the vehicle 61 and the electric pole 62 are voted is created on the road surface 60 voted as a descending straight line. By specifying the linear pixel group on the V-map, the pixel group corresponding to the road surface can be specified. In such a V-map, the parallax value of the portion below the road surface is not detected. Therefore, the parallax value corresponding to the area A shown by the diagonal line in FIG. 6 is not counted.

If there are parallax values in the portion below the road surface due to noise or the like, these parallax values may not be used in the subsequent object detection. In order to detect the height of the object, it is necessary to accurately detect the road surface. Therefore, using a virtual straight line K corresponding to the road surface detected when the vehicle 1 is stopped, a V map (restricted V map) that maps only pixels within a predetermined distance from the virtual straight line K is used for road surface detection. You may.

Next, the operation of the road surface shape detecting unit 46 that detects the road surface shape using the V map generated in this way will be described. The road surface shape detection unit 46 is an example of a reference detection unit. The road surface shape detecting unit 46 detects a road surface which is an example of a reference object which is a reference for the height of each object. The road surface shape detecting unit 46 linearly approximates the position estimated to be the road surface on the V map. If the road surface is flat, approximate it with a single straight line. Further, in the case of a road surface whose slope changes on the way, the V map is divided into a plurality of sections and linear approximation is performed. As a result, even in the case of a road surface whose slope changes on the way, linear approximation can be performed with high accuracy.

Specifically, the road surface shape detecting unit 46 first detects a road surface candidate point using a V map. For the detection of road surface candidate points, the horizontal axis is divided into two, and the detection method of the candidate points is changed in each area. Specifically, the road surface shape detection unit 46 detects road surface candidate points by the first candidate point detection method in a short-distance region with a large parallax. Further, the road surface shape detection unit 46 detects the road surface candidate point by the second candidate point detection method in a long-distance region where the parallax is small.

Here, the reason for changing the detection method of the road surface candidate point between the short-distance region with a large parallax and the long-distance region with a small parallax as described above is as follows. For example, as shown in the photographed image shown in FIG. 5, the area of the road surface is large at a short distance and the parallax data on the road surface is frequently voted on the V map, whereas the area of the road surface is small at a long distance and the road surface is small. The frequency of the coordinates representing is relatively small. That is, the frequency value of the points predicted as the road surface on the V map is small at a long distance and large at a short distance. Therefore, if the road surface candidate points are detected based on the same criteria, the road surface candidate points can be detected at a short distance, but the road surface candidate points at a long distance become difficult to detect, and the road surface detection accuracy at a long distance decreases.

In order to eliminate such a defect, the V map is divided into two regions, a region having a large parallax and a region having a small parallax, and the method and criteria for detecting road surface candidate points are changed in each region. Thereby, it is possible to improve the road surface detection accuracy of both the short distance and the long distance.

Next, the operation of the list generation unit 47 will be described. The list generation unit 47 calculates the road surface height and stores (tables) it in a storage unit such as the RAM 15. Since the linear equation representing the road surface is obtained from the V map, if the parallax d is determined, the corresponding y coordinate y0 is determined. The coordinate y0 is the height of the road surface. The list generation unit 47 generates a road surface height list in which the coordinates y0 indicating the height of the road surface and the parallax in the required range are associated with each other. When the parallax is d and the y coordinate is y', y'−y0 indicates the height from the road surface when the parallax is d. The list generation unit 47 calculates the height H of the coordinates (d, y') from the road surface using the calculation formula of "H = (z × (y'−y0)) / f". In this calculation formula, "z" is the distance calculated from the parallax d (z = BF / (d-offset)), and "f" is the focal length of the imaging unit 2 as the unit of (y'-y0). It is a value converted to the same unit. Here, BF is a value obtained by multiplying the baseline length B of the imaging unit 2 by the focal length f, and offset is the parallax when an object at infinity is photographed.

Next, the operation of the U-Disparity map generation unit (U map generation unit) 48 will be described. The U map is a two-dimensional histogram in which the horizontal axis is x of the parallax image, the vertical axis is the parallax d of the parallax image, and the Z axis is the frequency. Points (x, y, d) of the parallax image whose height from the road surface is in a predetermined height range such as 20 cm to 3 m are voted based on the value of (x, d). The U-map generation unit 48 generates a U-map by voting the points (x, y, d) of a certain parallax image for pixels in a predetermined range of the parallax image. That is, the sky is shown in the upper 1/6 area of the parallax image, and in most cases, the object requiring recognition is not shown. Therefore, the U map generation unit 48, for example, votes the pixels of the ground side 5/6 region (the region other than the upper 1/6 region) of the parallax image to generate the frequency U map.

More specifically, in the case of the luminance image shown in FIG. 7, there are guardrails 81 and 82 on the left and right, and the vehicle 77 and the vehicle 79 are facing each other across the center line. One vehicle 77 or 79 is traveling in each traveling lane. In this case, the U-map generation unit 48 generates the U-map with the frequency shown in FIG. That is, the U-map generation unit 48 generates a U-map with a frequency of voting for each guardrail 81, 82 as a straight line extending from the left and right to the upper center. Further, when the horizontal line component and the vehicle 81, 82 are detected between the guardrails 81 and 82 with the side surface visible, the U-map generator 48 connects the diagonal line indicating the side surface of the vehicle to the horizontal line component. By shape, vehicles 77, 79 are voted on the frequency U map.

Further, the U map generation unit 48 generates a height U map together with the U map having such a frequency. The height U map is a map in which the height from the road surface has the highest value among the pairs (x, d) voted for in the U map. In FIG. 9, the heights of the vehicles 77 and 79 are higher than those of the left and right guardrails 81 and 82. As a result, the height information of the object can be used for object detection.

Next, the operation of the layer U map generation unit 49 will be described. The layer U map generation unit 49 is an example of a map generation unit. The layer U map generation unit 49 generates a real U map with the actual distance (actual distance) as the horizontal axis in the above-mentioned U map in which the pixels of the image are on the horizontal axis. Then, the layer U map generation unit 49 votes the points to be voted for the real U map to the real U map (of the layer) of the corresponding height set in advance according to the height from the road surface, and for each layer. Generates a layer U map that is a real U map of. The layer U map is an example of a map.

First, FIG. 10 shows an example of a real U map. As shown in FIG. 10, the horizontal axis of the real U map is the actual distance. Further, the vertical axis of the real U map is the thinned parallax processed by thinning using the thinning rate according to the distance. Specifically, for example, in the case of a long distance of 50 m or more, the layer U map generation unit 49 uses parallax that is not thinned out. Further, in the case of a medium distance such as 20 m to 50 m, the layer U map generation unit 49 uses the thinned parallax processed by 1/2 thinning. Further, in the case of a short distance such as 10 m to 20 m, the layer U map generation unit 49 uses the thinned parallax processed by 1/3 thinning. Further, in the case of a recent distance such as 0 m to 10 m, the layer U map generation unit 49 uses the thinned parallax processed by 1/8 thinning.

At a distance, the object to be recognized is small, the parallax information is small, and the resolution of the distance is also small, so the thinning process is not performed. On the other hand, in the case of a short distance, since the object appears large, the parallax information is large and the resolution of the distance is also large. Therefore, it is possible to perform a large thinning process. FIG. 10 is an example of a real U map of frequency. As can be seen from FIG. 10, in the case of the real U map, the guardrails 81 and 82 are represented by vertical straight lines, and the shapes of the vehicles 77 and 79 are also close to the actual vehicles.

By creating a two-dimensional histogram with the horizontal axis as the actual distance and the vertical axis as the parallax value, the height information can be compressed and the isolated region can be detected in the two-dimensional plane. Therefore, high-speed processing can be enabled.

Next, the layer U map generation unit 49 generates a layer U map which is a real U map of each height range by using the real U map having such a frequency. That is, when voting the points (x, y, d) of the parallax image on the U map of the horizontal axis x, the vertical axis y, and the parallax d, the road surface shape detection process of the road surface shape detection unit 46 creates a real U map. The height (h) from the road surface to be voted (x, d) can be calculated. The layer U map generation unit 49 uses a preset real U map (of the layer) of the corresponding height range based on the height (h) of the points (x, d) to be voted for in the layer U map from the road surface. By voting for, a layer U map for each layer is generated.

Specifically, for example, as shown in FIG. 11, the vehicle 77 and the vehicle 79 pass face-to-face between the left and right guardrails 81 and 82 with the center line in between, and between the vehicle 79 and the right guardrail 82. It is assumed that a captured image in which two humans 83 and 84 exist along the guardrail 82 is obtained. In this case, the layer U map generation unit 49 generates a real U map having a frequency with the horizontal axis as the actual distance and the vertical axis as the thinned parallax. As a result, as shown in FIG. 12, between the guardrails 81 and 82 represented by vertical straight lines, each vehicle 77 and 79, and between the vehicle 79 and the right guardrail 82, two people along the guardrail 82. A real U-map is generated in which humans 83 and 84 are present.

Next, the layer U map generation unit 49 detects the height of each object from the road surface calculated by the road surface shape detection process of the road surface shape detection unit 46. As a result, the height of the guardrails 81 and 82 from the road surface is, for example, less than 150 cm, the height of the vehicle 77 from the road surface is less than 150 cm, the height of the vehicle 79 from the road surface is 150 cm or more, and the road surface of each person 83, 84. The height from the road surface of each object can be detected, such as 150 cm or more.

Next, the layer U map generation unit 49 generates a layer U map for an object having a height of 60 cm to 200 cm from the road surface shown in FIG. Guardrails 81, 82, vehicles 77, 79 and humans 83, 84 are all objects contained in a layer of 60 cm to 200 cm. Therefore, as in the real U map shown in FIG. 12, as shown in FIG. 13, between the guardrails 81 and 82 represented by the vertical straight lines, the respective vehicles 77 and 79, the vehicle 79 and the right guardrail Between 82, a layer U map is generated in which two humans 83, 84 are present along the guardrail 82.

Similarly, the layer U map generation unit 49 generates a layer U map for an object having a height of 150 cm to 200 cm from the road surface shown in FIG. In this case, the guardrails 81, 82 and the vehicle 77, whose height from the road surface is less than 150 cm, generate a layer U map that is not voted. That is, as shown in FIG. 14, a layer U map is generated in which the vehicle 79 and the two humans 83 and 84, which are objects having a height from the road surface of 150 cm to 200 cm, are voted.

At this time, in the above-mentioned two types of layer U maps, since the target heights overlap, the pixels whose heights from the road surface are 150 cm to 200 cm are voted for both layer U maps. In the present embodiment, the target heights of the two types of layer U maps overlap, but the present invention is not limited to this. For example, the first layer U map may be 60 cm to less than 150 cm, and the second layer U map may be 150 cm to 200 cm. However, by providing the overlap, the duplicated processing can be omitted, so that the speed can be increased.

The flowchart of FIG. 15 shows the flow of such a layer U map generation operation. The threshold value of the height to be treated as a layer on the layer U map is set in advance. Although it is only an example, in the case of the example of FIG. 15, a total of three layers, a human area layer having a height from the road surface of 60 cm to 200 cm, an adult area layer of 150 cm to 200 cm, and an area layer of 2 m or more of 200 cm or more are set in advance. Has been done.

In step S101, the layer U map generation unit 49 calculates the corresponding points on the layer U map from the points (x, y, d) on the parallax image (x, d). In step S102, the layer U map generation unit 49 calculates the actual height (h) of the points (x, d) to be voted from the road surface. Then, in step S103, the layer U map generation unit 49 determines and votes for the layer to which the calculated actual height h corresponds among the preset layers (step S104, step S105, step S106).

For example, when the actual height h = 170 cm, the layer U map generation unit 49 votes for the layer U map in the 60 cm to 200 cm region and the layer U map in the 150 cm to 200 cm region. In this case, no voting is performed for the layer U map of 200 cm or more.

Specifically, in the case of this example, as shown in FIG. 16, a layer in which an object having an actual height of 200 cm or more from the road surface is voted and an object having an actual height of 150 cm to 200 cm from the road surface are voted. A layer and a layer on which an object having an actual height of 0 cm to 200 cm from the road surface is voted are set in advance. That is, the layer U map is a U map in which height information is added to the above-mentioned real U map. Therefore, even when humans and objects are adjacent to each other, the objects can be accurately identified and detected.

When multiple people are adjacent to each other on the image, they are generally independent in the part where the height from the road surface is high, but for example, near the road surface or when people are holding hands, etc. Parallax regions are often connected in areas where the height from the road surface is low. Therefore, in the case of the layer U map, as shown in FIG. 17, a plurality of people may be detected as one object. That is, there is a possibility that the three people shown in FIG. 16 are connected and detected on the layer U map as, for example, rectangular objects as shown in FIG.

On the other hand, in the case of a layer U map, data is stored in a layer structure corresponding to a preset height. Therefore, as shown in FIG. 18, even when the object data is combined on the layer U map set at a height of 60 cm to 200 cm, the layer U map set at a height of 150 cm to 200 cm, which is another layer, is used. Above, as shown in FIG. 19, the data of each object appears separately. As a result, each object can be detected accurately. Such a layer U map has various uses depending on the height threshold value to be set. For example, when a threshold value having a height of 200 cm or more is set, it is possible to detect utility poles and poles having a height of 200 cm or more from the road surface as shown in FIG.

Next, the operation of the isolated area detection unit 50 that detects each object using such a layer U map will be described. The isolated region detection unit 50 is an example of a collective detection unit. The isolated area detection unit 50 detects an isolated area (aggregate area), which is an area of a mass of parallax information equal to or larger than a predetermined threshold value, from each layer U map. FIG. 21 is a flowchart showing the operation flow of the isolated area detection unit 50. In step S131, the isolated region detection unit 50 smoothes the frequency layer U map. That is, the parallax has a variance due to a calculation error or the like. Also, the parallax is not calculated for all pixels. Therefore, the layer U map contains noise. The isolated area detection unit 50 performs a layer U map smoothing process in order to remove noise and facilitate separation of objects to be detected. Similar to image smoothing, by applying a smoothing filter (for example, a simple average of 3 pixels × 3 pixels) to the frequency value of the layer U map, the frequency of noise can be reduced. Further, in the object part, a group can be formed with a higher frequency than the surroundings, and the isolated area detection process in the subsequent stage can be facilitated.

Next, the isolated region detection unit 50 sets a threshold value for binarizing the layer U map in step S132. This binarization process is performed depending on whether or not frequency data exists (threshold value 0). In step S133, the isolated region detection unit 50 first binarizes at a threshold value of 0. Objects to be voted on the layer U map include objects that are independent and objects that are connected to other objects due to the height, shape, separation from road parallax, etc. of the objects. Therefore, the isolated region detection unit 50 binarizes the layer U map from a small threshold value, and gradually increases the threshold value to perform the binarization process. As a result, an independent object of an appropriate size (isolated area) can be detected first, and then the connected objects can be separated by increasing the threshold value and detected as an independent size object.

Next, in step S134, the isolated region detection unit 50 detects the isolated region by performing labeling processing of the coordinates having a value. Specifically, the isolated region detection unit 50 labels the coordinates that become black values after binarization (coordinates whose frequency value is higher than the binarization threshold value) based on their connectivity, and attaches the same label. Detect the area as one object. In the labeling method, when the coordinates indicated by the “x” mark shown in FIG. 22 are black and are the coordinates to be labeled, the coordinates indicated by the “x” mark are diagonally above left, above, diagonally above right, and left respectively. If the coordinates of each position of "1", "2", "3", and "4", which are the coordinates of the position, are labeled, the same label is assigned to the coordinates indicated by the "x" mark.

As shown in FIG. 23, when a plurality of labels are assigned around the coordinates indicated by the “x” mark, the isolated area detection unit 50 displays the label with the smallest value as shown in FIG. 24. Assign to the coordinates indicated by the "x" mark. Further, as shown in FIG. 24, the isolated area detection unit 50 replaces the label of the coordinate to which the label is assigned among the coordinates around the coordinates indicated by the “x” mark with the label of the smallest value. In the case of the examples of FIGS. 23 and 24, the isolated area detection unit 50 assigns the label of “8” to the coordinates indicated by the “x” mark. Further, the isolated area detection unit 50 displays the labels of the coordinates indicated by the “x” mark, which are the coordinates to which the labels “9” or “8” are assigned, at the diagonally upper left, diagonally upper right, and left. , Replace with the label "8".

The width of the object (width of the object having the same label) detected by such labeling processing can be regarded as the width W close to the actual object. The isolated area detection unit 50 considers an object having a width W within a predetermined range as an object candidate area.

Next, the isolated region detection unit 50 determines the sizes of the plurality of detected isolated regions in step S135. This size determination process is a process for determining whether or not the width of the isolated region is within the size range of the detection target because the detection target is a pedestrian to a large vehicle. When the width of the isolated region is larger than the size of the detection target (step S135: Yes), the isolated region detection unit 50 returns the process to step S132. Then, the isolated region detection unit 50 increments the binarization threshold value by one, and binarizes the isolated region having a width larger than the size of the detection target again. In addition, the isolated area detection unit 50 labels the isolated area that has been binarized again, detects a smaller isolated area, and determines whether or not the size is larger than the size of the detection target. The isolated area detection unit 50 repeats the labeling process from the threshold setting in this way to detect an isolated area having a predetermined size.

Next, when an isolated region having a predetermined size is detected (step S135: No), the isolated region detection unit 50 performs a peripheral region removal process in step S136. In the case of an object existing in a distant place, the road surface detection accuracy is poor, the parallax of the road surface is voted in the layer U map, and the parallax of the object and the road surface may be detected as a group. In such a case, the isolated area detection unit 50 deletes a peripheral area removal process that deletes a region whose height is close to the road surface on the left, right, and near the region (removal region) where the parallax between the object and the road surface is a mass. I do.

Such a peripheral area removal process will be specifically described. FIG. 25 shows a layer U map in which a distant road surface is detected lower than the actual height and the road surface and the vehicle are detected as one object. In FIG. 25, the area shaded to the right indicates a region having a high frequency, and the region shaded to the left indicates a region having a low frequency. FIG. 26 is a real U map with a height corresponding to the layer U map of FIG. The height real U map referred to here is a map having information on the maximum height for each pixel. In FIG. 26, the area shaded to the right indicates a region having a high height, and the region shaded to the left indicates a region having a low height. As can be seen by comparing FIGS. 25 and 26, the frequently used region in the layer U map and the high height region in the real U map of height do not match. Also, the infrequent areas in the layer U map do not match the low height areas in the real U map of height.

In such a case, the isolated area detection unit 50 uses a real U map of height instead of the layer U map to detect the boundary between the object and the road surface and remove the peripheral area. The layer U map has only the height information of the number of divisions (the number of layers) of each height range, but the real height U map has the height information with fine resolution, so that it is peripheral. This is because it is suitable for removing the region. For example, in the flow shown in the flowchart of FIG. 27, the isolated region detection unit 50 removes the peripheral region in the order of removal of the neighboring region (step S145), removal of the left region (step S146), and removal of the right region (step S147). I do.

In the determination of neighborhood region removal in step S145, the isolated region detection unit 50 sets the height threshold value as follows, and condition 1, condition 2 or condition 3 consists of the lowest end (bottom line) of the isolated region. If you want to continue, change the frequency of the corresponding line and remove it. When setting the height threshold value, the isolated area detection unit 50 sets the height threshold value according to the maximum height of the area (removal area) in which the parallax of the object and the road surface is a mass. For example, when the maximum height is 120 cm or more, the isolated region detection unit 50 sets the height threshold value to 60 cm. When the maximum height is less than 120 cm, the isolated region detection unit 50 sets the height threshold value to 40 cm.

Condition 1 is that "there are no points in the line having heights equal to or less than a predetermined number such as five and having a height equal to or higher than the height threshold value". In addition, condition 2 states that "the number of points in the line having a height equal to or higher than the height threshold is less than the number of points having a height less than the height threshold, and the number of points having a height equal to or higher than the height threshold" The number is less than two. " Further, the condition 3 is "the number of points having a height equal to or higher than the height threshold value is less than 10% of the number of points having a height of the entire line". When the above-mentioned condition 1, condition 2 or condition 3 continues to consist of the lowermost end (bottom line) of the isolated region, the isolated region detection unit 50 changes the frequency of the corresponding line and removes it.

Further, the isolated region detection unit 50 sets the height threshold value as follows for the determination of the left and right region removal in steps S146 and S147, and the condition 4, condition 5 or condition 6 is the left end or right end row of the isolated region. If it continues to consist of, change the frequency of the corresponding column and remove it. When setting the height threshold value, the isolated area detection unit 50 sets according to the maximum height of the area (removal area) in which the parallax between the object and the road surface is a mass. For example, when the maximum height is 120 cm or more, the isolated region detection unit 50 sets the height threshold value to 60 cm. When the maximum height is less than 120 cm, the isolated region detection unit 50 sets the height threshold value to 40 cm.

Condition 4 is that "the number of points having a height in the row is not more than a predetermined number such as 5, and there is no point having a height equal to or more than the height threshold value". Condition 5 states that "the number of points in the column having a height equal to or higher than the height threshold is less than the number of points having a height less than the height threshold, and the number of points having a height equal to or higher than the height threshold is large. There are less than two. " Condition 6 is that "the number of points having a height equal to or higher than the height threshold value is less than 10% of the number of points having a height of the entire column". When the condition 4, condition 5 or condition 6 continues to consist of the leftmost or rightmost rows of the isolated region, the isolated region detection unit 50 changes the frequency of the corresponding rows and removes them.

In FIG. 28, the black-painted region located at the center indicates an isolated region having a high height. In addition, the isolated areas indicated by the right diagonal line, the left diagonal line, and the net line each indicate an isolated area having a low height. Of these, the isolated region indicated by the right diagonal line is removed based on the height threshold value set by the isolated region detecting unit 50 and the above-mentioned condition 1, condition 2 or condition 3. Further, the isolated region indicated by the left diagonal line and the net line is removed based on the height threshold value set by the isolated region detecting unit 50 and the above-mentioned condition 4, condition 5 or condition 6. As a result, only the black-painted isolated region shown in the center of FIG. 28 can be extracted.

Next, the isolated area detection unit 50 determines in step S137 of the flowchart of FIG. 21 whether or not there is another removal area. If there is another removal region (step S137: Yes), the isolated region detection unit 50 returns the processing to step S134, performs the labeling process again, and resets the isolated region.

Next, when the isolated area detection unit 50 determines in step S137 that there is no other removed area (step S137: No), the process proceeds to step S138, and the size of the isolated area where the peripheral area is removed (step S137: No). Width, height, distance) is determined.

For example, when objects are placed side by side and close to each other, such as automobiles and motorcycles, automobiles and pedestrians, automobiles, etc., the smoothing of the layer U map causes the isolated areas of multiple objects to be one. May be detected as an isolated area of one object. Alternatively, when parallax interpolation processing is performed to compensate for the number of pixel points in a parallax image, the parallax between different objects may be connected. In such a case, the isolated area detection unit 50 performs the lateral separation process described below to separate one isolated area in which a plurality of objects overlap into a plurality of isolated areas for each object.

On the other hand, when a plurality of preceding vehicles traveling in the adjacent lane are far away, if the parallax dispersion obtained from the stereo image is large, the parallax of each object extends up and down on the layer U map. May be connected. Then, the isolated area of the concatenated objects may be detected as the isolated area of one object. In such a case, the isolated area detection unit 50 performs the vertical separation process described below, and travels in the isolated area detected as one object with the isolated area of the preceding vehicle running in front and in front of the preceding vehicle. Separate into isolated areas of the vehicle.

In step S138, the isolated area detection unit 50 performs such a horizontal separation process or a vertical separation process by determining the size (width, height, distance) of the isolated area from which the peripheral area has been removed. Whether or not it is determined. When the isolated area detection unit 50 determines that none of the separation processes is performed, the process proceeds to step S139, and the isolated area subjected to the peripheral area removal process is temporarily stored in a storage unit such as RAM 15 as an object candidate. ,sign up.

On the other hand, when another object having a width of 50 cm is close to a car having a width of 2 m, for example, the isolated area detected from the layer U map has a width exceeding 2.5 m in the size determination in step S138. It is judged as an object of. In such a case, the isolated area detection unit 50 proceeds with the process in step S140 to execute the horizontal separation process, and advances the process to step S142 to execute the vertical direction separation process.

FIG. 29 shows a flowchart showing the flow of the lateral separation process executed by the isolated region detection unit 50. In step S151, the isolated region detection unit 50 integrates the frequency information and the height information of each X coordinate as shown in FIG. 30, and integrates the integrated value of each X coordinate in the vertical direction (Y direction). And calculate the evaluation value in the horizontal direction.

FIG. 31 shows an example of the waveform of the evaluation value in the horizontal direction. Of these evaluation values, the bottom BT portion indicates the minimum value (minimum evaluation value) of the evaluation value. In step S152, the isolated region detection unit 50 detects the position of the minimum evaluation value. In step S153, the isolated region detection unit 50 multiplies the average value of each evaluation value by a predetermined coefficient such as 0.5 to calculate the binarization threshold value T of the evaluation value. Then, the isolated region detection unit 50 sets the binarization threshold value T as the evaluation value, as shown by the dotted line in the middle of FIG. Then, in step S154, the evaluation value is binarized based on the binarization threshold value.

The reason for using the evaluation value is as follows. First, it is possible to handle the frequency value of the connected part with a value relatively smaller than the frequency value of the object. Further, in the real U map of height, the connected portion has a height different from that of the object portion, and there is little data having a height. Further, when they are connected due to the influence of parallax interpolation, the variance of parallax in the real U map of height is small.

Next, in step S155, the isolated region detection unit 50 sets a region including the minimum evaluation value and including an evaluation value smaller than the binarization threshold T as the separation region SR as shown in FIG. 31, and sets the separation region SR. Both ends are defined as separation boundary SK. The isolated area detection unit 50 laterally separates the isolated area corresponding to each object by changing the frequency value inside the separation boundary SK.

In step S141 of the flowchart of FIG. 21, the isolated area detection unit 50 registers an object that cannot be separated by such a lateral separation process (step S141: No) in a storage unit such as a RAM 15 as an object candidate. Further, if the isolated region detection unit 50 can perform the lateral separation process (step S141: Yes), the isolated region detection unit 50 may be able to further separate the process. Therefore, the isolated area detection unit 50 returns the process to step S134 and tries the lateral separation process again.

FIG. 32 shows a layer U map after the smoothing process. In the layer U map shown in FIG. 32, the isolated region with the left diagonal line shows the isolated region with high frequency, and the isolated region with the right diagonal line shows the isolated region with low frequency. Further, the frequently isolated region (isolated region on the left diagonal line) on the lower side of FIG. 32 is an isolated region corresponding to the preceding vehicle traveling immediately before. Further, the frequently isolated region (isolated region on the left diagonal line) on the upper side (back side) of FIG. 32 is an isolated region corresponding to the preceding vehicle traveling further ahead of the preceding vehicle traveling immediately before.

In the case of the layer U map of FIG. 32 as described above, the parallax is concentrated on the left and right ends of the preceding vehicle, and the parallax is widely dispersed. Therefore, the parallax extends so as to draw a left curve on both sides of the frequently isolated region. Since the parallax growth is large, as shown by the ellipse in the middle dotted line in FIG. 32, the parallax growth of the preceding vehicle in the foreground and the parallax growth of the preceding vehicle in the back side are connected, and the two leading vehicles are one. Detected as an isolated area. In such a case, the width of the detected isolated area becomes larger than the width of the actual preceding vehicle. Therefore, the isolated area detection unit 50 sets an area for calculating the actual width of the preceding vehicle. As a region for calculating the actual width of the preceding vehicle, the isolated region detection unit 50 sets a predetermined distance range Zr according to the size of the nearest neighbor distance Zmin of the isolated region as shown below. Then, the isolated region detection unit 50 searches for the actual width of the preceding vehicle in the region of the parallax range corresponding to the distance range. The area of the distance range shown in FIG. 32 is the actual width calculation area JR. The value of Zr can be set from the magnitude of the dispersion of parallax obtained from the image captured by the imaging unit 2.

Specifically, the isolated region detection unit 50 sets the distance range Zr of the actual width calculation region as follows. That is, for example, in the case of "Zmin <50 m", the isolated region detection unit 50 is set to Zr = 20 m. Further, for example, in the case of "50 m <= Zmin <100 m", the isolated region detection unit 50 is set to Zr = 25 m. Further, for example, in the case of "Zmin> 100 m", the isolated region detection unit 50 is set to Zr = 30 m. The isolated area detection unit 50 separates a plurality of objects horizontally connected in the isolated area by such a horizontal separation process.

Next, the vertical separation process in step S142 will be described. As shown in FIG. 32, when objects are arranged in the vertical direction on the layer U map and two or more columns are detected as one isolated area, the parallax range and the distance between the two or more vehicles are present. The range in which the distances are combined is the parallax range of the isolated region. For example, when the distance is 20 m or more, the parallax (distance) has a wide range other than the guardrail or the wall, and it becomes difficult to separate in the column direction. Therefore, the isolated region detection unit 50 performs the vertical separation process on the isolated region having a large parallax (distance) range of the isolated region according to the distance. The guardrail or the wall of the building is narrow even if the parallax range is wide. The isolated area detection unit 50 does not execute the vertical separation process on the narrow isolated area.

Specifically, the isolated region detection unit 50 sets the nearest neighbor distance of the isolated region to "Zmin", the farthest distance to "Zmax", and the width to "W", and satisfies either the following condition 7 or condition 8. Occasionally, a vertical separation process is performed. Condition 7 is "Zmax-Zmin> 50m when W> 1500 and Zmin> 100m". The condition is "Zmax-Zmin> 40m when W> 1500 and 100m ≧ Zmin> 20m".

The flow chart of FIG. 33 shows the flow of the vertical separation process. First, FIG. 32 shows a layer U map in which two preceding vehicles traveling in the left adjacent lane are detected as one isolated region as described above. In the case of this example, the frequently shaded area on the lower side (front side) in FIG. 32 represents the preceding vehicle traveling in front of the own vehicle, and the frequently shaded area on the upper side (back side) in FIG. 32. The area shows the preceding vehicle running further ahead of the preceding vehicle. Since the parallax is concentrated on the left and right ends of the preceding vehicle and the dispersion is large, the parallax extends so as to draw a left curve on both sides of the frequently used area. Since the increase in parallax is large, the increase in parallax of the preceding vehicle in the foreground and the increase in parallax of the preceding vehicle in the back are connected, and the two preceding vehicles are detected as one isolated region.

In such a case, since the widths of the two preceding vehicles are detected overlapping, the width of the isolated region is detected as a width larger than the actual width of one preceding vehicle. Therefore, the isolated area detection unit 50 sets the area for calculating the actual width in step S161 of the flowchart of FIG. 33.

Specifically, the isolated region detection unit 50 sets a predetermined distance range Zr according to the magnitude of Zmin to Zmin of the isolated region as described below.

Set Zr = 20m when Zmin <50m Set Zr = 25m when 50m ≤ Zmin <100m Set Zr = 30m when Zmin> 100m

The isolated area detection unit 50 detects the actual width of the preceding vehicle in the area of the parallax range corresponding to the distance range set in this way. The area in the range of parentheses in FIG. 32 is the actual width calculation area JR. The isolated region detection unit 50 sets a value of a predetermined distance range Zr based on the magnitude of the dispersion of parallax obtained from the image pickup unit 2.

Next, the isolated region detection unit 50 calculates the lateral evaluation value in the actual width calculation region JR in step S162. As shown in FIG. 34, the isolated area detection unit 50 uses the frequency value of each line of the actual width calculation area JR as the evaluation value of each line. Further, as shown in FIG. 35, the isolated region detection unit 50 sets the line having the maximum evaluation value as the actual width detection position.

Next, in step S163, as shown in FIG. 35, the isolated region detection unit 50 has the actual width region in which the frequency values at the actual width detection positions are continuously present and have the maximum length (width). Is detected, and the maximum detected length is taken as the actual width. In the case of the example of FIG. 35, the actual width is 5.

Next, in step S164, the isolated region detection unit 50 sequentially calculates and sets the position of the separated boundary in each parallax of the isolated region with the outside of the actual width boundary as the separation boundary and the separation boundary as a reference. That is, as shown in FIG. 36, the lens center of the camera of the imaging unit 2 is O, the direction in which the camera is facing is the camera center axis (vertical axis of the center of the layer U map), and the vertical direction of the camera center axis is the horizontal axis. And. For example, the position of the separation boundary of the actual width region is the distance Z0 and the horizontal position X0. At this time, the position X of the separation boundary at the distance Z is “X = X0 × (Z / Z0)”.

Further, the value obtained by multiplying the baseline length B of the image pickup unit 2 and the focal length F is defined as BF, the parallax corresponding to Z0 is d0, the parallax corresponding to Z is d, and the offset is the parallax at infinity. In this case, "Z = BF / (d-offset)" and "Z0 = BF / (d0-offset)". Therefore, the mathematical formula of "X = X0 × (Z / Z0)" can be transformed into “X = X0 × (d0-offset) / (d-offset)”.

Since the correspondence between the parallax d and the thinned parallax of the layer U map is known, the position of the separation boundary in the isolated region can be determined by the values of all the thinned parallax. The black-painted area in FIG. 37 shows the area serving as the separation boundary. As a result, in step S165, the preceding vehicle (object in front) traveling in front of the own vehicle and the preceding vehicle (object in the back) traveling ahead of the preceding vehicle can be separated.

In the case of the example of FIG. 37, a vertically long region is separated at the lower left of the isolated region. However, since the vertically long region separated is narrow in width, the isolated region detection unit 50 processes it as noise. Further, the example of FIG. 37 is an example in which the preceding vehicle is traveling in the lane adjacent to the left, but when the preceding vehicle is traveling in the lane adjacent to the right, the frequency map of the isolated region is shown in FIG. 32. The example of is a left-right inverted distribution.

When a plurality of preceding vehicles are traveling in the lane next to the driving lane of the own vehicle, the parallax obtained from the captured image is widely dispersed, so that the parallax of each object extends vertically on the layer U map. It may be concatenated and falsely detected as one object. However, by performing such a vertical separation process, the objects that are connected and detected can be separated and detected as a plurality of objects. That is, in the case of this example, the preceding vehicle traveling in front and the preceding vehicle traveling further ahead can be recognized separately.

When such vertical separation processing is performed, the isolated area detection unit 50 proceeds to step S143 of the flowchart of FIG. 21 to determine whether or not the vertical separation of the objects can be performed. Then, when it is determined that the vertical direction separation of the objects can be performed (step S143: Yes), there is a possibility that the objects can be further separated. Therefore, the isolated area detection unit 50 returns the process to step S134 and again in the vertical direction. Attempt separation processing. On the other hand, when it is determined that the vertical separation of the objects could not be performed (step S143: No), the isolated area detection unit 50 proceeds to step S139 and selects the objects that cannot be separated by the vertical separation processing. , Is registered in a storage unit such as RAM 15 as an object candidate.

Next, the flowchart of FIG. 38 shows the flow of the above-mentioned isolated area detection process in the isolated area detection unit 50 and the flow of the object recognition process in the classification unit 51 based on the detected isolated area. Steps S111 to S116 of the flowchart of FIG. 38 show the flow of the isolated area detection process in the isolated area detection unit 50, and steps S117 to S127 show the flow of the object recognition process in the classification unit 51.

In the description of the flowchart of FIG. 38, the layer U map in which the height threshold value is set to 60 cm to 200 cm is referred to as “LRUD_H”. Further, the layer U map in which the height threshold value is set to 150 cm to 200 cm is referred to as "LRUD_A". Further, the layer U map in which the height threshold is set to 200 cm or more is called "LRUD_T".

First, the isolated region detection unit 50 performs a binarization process of the data of LRUD_H in step S111. Next, in step S112, the isolated region detection unit 50 performs a labeling process on the binarized LRUD_H to detect the isolated region.

Similarly, in step S113, the isolated region detection unit 50 binarizes the data of LRUD_A, and in step S114, performs labeling processing on the binarized LRUD_A to detect the isolated region. Further, the isolated region detection unit 50 performs a binarization process of the data of LRUD_T in step S115, and performs a labeling process on the binarized LRUD_T in step S116 to detect the isolated region.

The range of LRUD_A (150 cm to 200 cm) is included in the range of LRUD_H (60 cm to 200 cm). Therefore, the isolated area detection unit 50 performs the binarization process and the labeling process for LRUD_A only within the isolated area detected by LRUD_H. As a result, duplicate processing can be omitted, and binarization processing and labeling processing can be speeded up.

A specific example of the processing up to this point will be described. FIG. 39 is an example of a captured image. In the captured image shown in FIG. 39, the side surface of the vehicle 85 is shown on the back side, the first person group 86 is shown on the front left side, and the second person group 87 is shown on the front right side. In addition, the utility pole 93 is shown on the right side of the second person group 87.

The actual height of the vehicle 85 from the road surface is 150 cm or more and less than 200 cm. The first person group 86 is a group of 88 children whose actual height from the road surface is less than 150 cm and 89 adults whose actual height from the road surface is 150 cm or more and less than 200 cm. The second person group 87 is a group of two adults 91 and 92 whose actual height from the road surface is 150 cm or more and less than 200 cm. The utility pole 93 has an actual height of 200 cm or more from the road surface.

When the parallax image corresponding to the captured image of FIG. 39 is voted on the layer U map whose threshold value is set to 60 cm to 200 cm, the layer U map shown in FIG. 40 is obtained. When the threshold value is 60 cm to 200 cm, the above-mentioned vehicle 85, first and second person groups 86, 87 and utility pole 93 are detected as isolated regions, respectively. That is, in the first and second person groups 86 and 87, although there are a plurality of people (children 88, adults 89, adults 91, adults 92), they are detected as a single isolated region connected to each other.

On the other hand, FIG. 41 is a layer U map in which a parallax image corresponding to the captured image of FIG. 39 is voted on the layer U map whose threshold value is set to 150 cm to 200 cm. When the threshold is 150 cm to 200 cm, the child 88 whose actual height from the road surface is less than 150 cm is not voted on the layer U map. Therefore, only adult 89 is detected as an isolated region. Further, in the second person group 87, when the adult 91 and the adult 92 are in contact with each other at a portion where the height from the road surface is 150 cm or less, the adult 91 and the adult 92 can be detected as two isolated regions, respectively.

Further, FIG. 42 is a layer U map in which a parallax image corresponding to the captured image of FIG. 39 is voted on the layer U map whose threshold value is set to 200 cm or more. When the threshold value is 200 cm or more, only the utility pole 93 is detected as an isolated region. By superimposing the objects shown in FIGS. 40 to 42, the layer U map shown in FIG. 43 is obtained. The isolated region on the left diagonal line is the isolated region detected from the layer U map whose threshold value is set to 60 cm to 200 cm. Further, the black-painted isolated area is an isolated area detected from the layer U map whose threshold value is set to 150 cm to 200 cm, and the isolated area on the left diagonal line is detected from the layer U map whose threshold value is set to 200 cm or more. It is an isolated area to be.

Next, when the detection process of such an isolated region is completed, the classification unit 51 performs the object type classification process. That is, when the isolated region detection process is completed, the process of the flowchart of FIG. 38 proceeds to step S117. In step S117, the classification unit 51 refers to the object information stored in the storage unit such as the RAM 15, and detects the type and size of the object in the isolated area of LRUD_H. This object information is the information shown in Tables 1 and 2. Specifically, as shown in Table 1 as an example, the types of objects having a size of "width less than 1100 mm, height less than 2500 mm, depth less than 1000 mm" are defined as motorcycles and bicycles. Similarly, the type of object having a size of "width less than 1100 mm, height less than 2500 mm, depth 1000 mm or less" is defined as a pedestrian.

Similarly, the type of object having a size of "less than 1700 mm in width, less than 1700 mm in height, less than 10000 mm in depth" is defined as a compact car. In addition, the type of object having a size of "width less than 1700 mm, height less than 2500 mm, depth less than 10000 mm" is defined as an ordinary vehicle. Further, the type of the object having the size of "width less than 3500 mm, height less than 3500 mm, depth less than 15000 mm" is defined as a truck vehicle.

Further, among pedestrian type objects, for example, as shown in Table 2, an object having a width of less than 1200 mm, a height of less than 2000 mm, and a depth of less than 1200 mm is defined as an adult pedestrian. In addition, an object having a width of less than 1200 mm, a height of less than 1500 mm, and a depth of less than 1200 mm is defined as a child pedestrian. In addition, an object having a width of less than 2500 mm, a height of less than 1500 mm, and a depth of less than 2500 mm is defined as a pedestrian with an accessory.

In step S117, the classification unit 51 determines whether or not the isolated region of LRUD_H having an actual height from the road surface in the range of 60 cm to 200 cm is smaller than the pedestrian size with the attachments shown in Table 2. .. Ancillary is, for example, the child 88 shown in FIG. Usually, the isolated areas of adults 89, 91, 92 in the height range of 150 cm to 200 cm have a width of 1200 mm or less as shown in Table 2. On the other hand, as in the first person group 86 shown in FIG. 39, the isolated area corresponding to the adult pedestrian holding hands with the child as an accessory has a width as shown in Table 2. It will be 2500 mm or less.

When the classification unit 51 determines that the isolated region on LRUD_H is larger than the size of a pedestrian with an accessory (step S117: No), the process proceeds to step S120 and step S121. On the other hand, when it is determined that the isolated region of LRUD_H is smaller than the pedestrian size (step S117: Yes), the classification unit 51 proceeds to the process in step S118.

In step S118, the size of the isolated region corresponding to the isolated region of LRUD_H, in which the classification unit 51 appears in LRUD_A whose actual height from the road surface is in the range of 150 cm to 200 cm, is the adult pedestrian size shown in Table 2. It is determined whether or not it is the above. When the child 88 of the first person group 86 shown in FIG. 39 is less than 150 cm tall (step S118: No), it appears in LRUD_A in which the actual height from the road surface is in the range of 150 cm to 200 cm as shown in FIG. Absent. In this case, the classification unit 51 proceeds to step S122. Then, the classification unit 51 classifies the object of the isolated area which is pedestrian size and does not appear in LRUD_A as a pedestrian of a child with a height of less than 150 cm in step S122, and classifies the classification information. Is once registered in a storage unit such as a RAM 15 in association with an object in an isolated area.

Further, when it is determined that the size of the isolated region corresponding to the isolated region of LRUD_H appearing in LRUD_A is equal to or larger than the adult pedestrian size shown in Table 2 (step S118: Yes), the classification unit 51 moves to step S119. Proceed with processing. In step S119, the classification unit 51 includes each isolated region appearing in LRUD_H whose actual height from the road surface shown in FIG. 40 is in the range of 60 cm to 200 cm, and the actual height from the road surface shown in FIG. 42 is 200 cm or more. Compare with each isolated region appearing in LRUD_T in the range of. Then, the classification unit 51 detects the isolated regions having the same positions in each isolated region appearing in LRUD_H and each isolated region appearing in LRUD_T.

For example, in the utility pole 93 shown in FIG. 42, the position of the isolated region on LRUD_H shown in FIG. 40 coincides with the position of the isolated region on LRUD_T shown in FIG. 42 (step S119: Yes). Therefore, in step S123, the classification unit 51 classifies the object type in which the object in the isolated region on LRUD_H is an object of 200 cm or more such as a utility pole 93, and temporarily registers the object in a storage unit such as RAM 15.

On the other hand, the fact that the position of the isolated region appearing in LRUD_H and the position of the isolated region appearing in LRUD_T do not match (step S119: No) means that the isolated region appearing in LRUD_H is, for example, FIG. 41. It means that the object is in the range of 150 cm to 200 cm, such as adults 89, 91, and 92 shown in. Therefore, in step S124, the classification unit 51 classifies the object type on the LRUD_H as a human being having a height of 150 cm or more, and temporarily registers the object in a storage unit such as the RAM 15.

On the other hand, in step S117, when the isolated region of LRUD_H is determined to be equal to or larger than the pedestrian size with the attachments shown in Table 1, and the process proceeds to step S120, the classification unit 51 sees the attachments on LRUD_H. It is determined whether or not the isolated area on LRUD_A corresponding to the isolated area determined to be the size equal to or larger than the pedestrian size provided with is the adult pedestrian size shown in Table 2. In the case of a pedestrian with an accessory, if the child is less than 150 cm tall, only an isolated area of an adult 89 with a height of 150 cm or more appears on LRUD_A. Therefore, when the isolated region of LRUD_H is larger than the pedestrian size with an accessory and the isolated region on the corresponding LRUD_A is the adult pedestrian size (step S120: Yes), the classification unit 51 steps. In S125, the isolated region on LRUD_A is classified as a pedestrian carrying an accessory, and is temporarily registered in a storage unit such as RAM 15.

Further, in step S117, by determining that the isolated region of LRUD_H is equal to or larger than the size of a pedestrian with an accessory (step S120: No), when the process proceeds to step S121, the classification unit 51 of LRUD_A It is determined whether the isolated area is smaller than the pedestrian size of the child shown in Table 2. If it is determined that the size is smaller than the pedestrian size of the child (step S121: Yes), the classification unit 51 proceeds to step S126. Then, in step S126, the classification unit 51 classifies the object type in which the isolated area of LRUD_A is a short and long object such as a guardrail, and temporarily registers the isolated area in the storage unit such as the RAM 15. On the other hand, when it is determined that the size is larger than the pedestrian size of the child (step S121: No), the classification unit 51 proceeds to step S127, and the isolated area of LRUD_A is a bicycle, a motorcycle, a car, a bus, or a bus. The object type is classified into an object having a size larger than that of a person such as a truck, and temporarily registered in a storage unit such as a RAM 15.

In this way, the classification unit 51 classifies the object types corresponding to each isolated region based on the distribution of the isolated regions in each layer U map.

To explain the flow of such an object classification operation again, for example, in the case of the first person group 86 shown in FIG. 39, in the layer U map (LRUD_H) in which the actual height from the road surface is in the range of 60 cm to 200 cm, FIG. 40 As shown in, it appears as an isolated area larger than the size of a pedestrian. However, in the case of the first person group 86, on the layer U map (LRUD_A) in which the actual height from the road surface is in the range of 150 cm to 200 cm, it appears as an adult pedestrian-sized isolated region as shown in FIG. Therefore, the classification unit 51 registers the isolated area having a size larger than the human size in the isolated area on LRUD_H and the isolated area having a human size on LRUD_A as a pedestrian carrying an accessory. (Step S125).

On the other hand, in the case of the second person group 87 shown in FIG. 39, on LRUD_H, one isolated area having a size larger than the person size is formed, and on LRUD_A, there are two isolated areas having a person size. Therefore, each of the two isolated areas is registered as an adult pedestrian with a size of 150 cm or more. Since the vehicle is detected as an isolated area larger than the human size in both LRUD_H and LRUD_A, the vehicle is registered as an object larger than the human size. Further, the utility pole 93 shown in FIG. 39 is human-sized in both LRUD_H and LRUD_A, but is also present as an isolated region on the layer U map (LRUD_T) in a range of 200 cm or more, and therefore has a height of 2 m or more. Register as an object with. Using the isolated area detected in the layer U map as a three-dimensional object candidate in this way, the object types are classified according to their sizes and the correspondence between each layer (between LRUD_H and LRUD_T), and registered in the RAM 15 or the like.

That is, the actual width and height of the object in the parallax image can be calculated by the following mathematical formulas.

Actual width = distance to object x (width of object area in parallax image) / f
Actual height = distance to object x (height of object area in parallax image) / f

Further, the depth of the object can also be calculated by the following formula from the minimum parallax dmin and the maximum parallax dmax in the rectangular area of the layer U map.

Depth = BF × (1 / (dmin-offset) -1 / (dmax-offset))

Based on this information, the classification unit 51 classifies object types such as pedestrians, bicycles, compact cars, and trucks shown in Table 1, for example.

Next, the width of the rectangle surrounding the isolated area detected by the layer U map corresponds to the width of the detected object, and the height corresponds to the depth of the detected object. The height of the object is the maximum height of the layer U map. The corresponding area detection unit 52 shown in FIG. 4 detects the corresponding area of the parallax image corresponding to the isolated area detected by the layer U map in order to detect the accurate position and height of the object.

That is, the range (xmin, xmax) to be detected in the parallax image can be determined from the position, width, and minimum parallax of the isolated region detected in the layer U map. Further, in the parallax image, from the y coordinate corresponding to the maximum height from the road surface at the time of ymin = maximum parallax dmax to y indicating the height of the road surface obtained from ymax = maximum parallax dmax. ) Can be determined. The corresponding area detection unit 52 scans the set image area in order to detect the accurate position of the object, and has a parallax value in the range of the depth (minimum parallax dm, maximum parallax dmax) of the rectangle detected in the isolated area. Is extracted as a candidate pixel. Then, the corresponding area detection unit 52 sets a line existing in a predetermined ratio or more in the lateral direction with respect to the detection width in the extracted candidate pixel group as a candidate line.

The area where the candidate line exists is, for example, as shown in FIG. 44, the rectangles 111 and 112 surrounding the vehicles 77 and 79. Next, the corresponding area detection unit 52 performs vertical scanning, and when there are candidate lines having a density equal to or higher than a predetermined density around the line of interest, the corresponding area detection unit 52 detects the line of interest as an object line.

Next, the area extraction unit 53 shown in FIG. 4 detects the object line in the search area of the parallax image, and determines the lowermost end and the uppermost end of the object line. Then, the area extraction unit 53 determines the circumscribed rectangles 111 and 112 of the object line group as the object area in the parallax image as shown in FIG. 45.

The flowchart of FIG. 46 shows the flow of the corresponding area detection process of the parallax image in the corresponding area detection unit 52 and the area extraction unit 53. First, in step S171, the corresponding region detection unit 52 determines the search range of x in the parallax image from the position, width, and minimum parallax of the isolated region of the layer U map. Next, in step S172, the corresponding region detection unit 52 obtains the maximum search value ymax in the y direction from the relationship between the maximum parallax dmax of the isolated region and the road surface height. Next, in step S173, the corresponding region detection unit 52 obtains the minimum search value ymin from the maximum heights, ymax and dmax of the isolated region of the layer U map.

Next, in step S174, the corresponding area detection unit 52 searches for a parallax image in the set search range, extracts pixels within the range of the maximum parallax dmax from the minimum parallax dm in the isolated region, and uses them as object candidate pixels. .. In step S175, the corresponding area detection unit 52 extracts a line in which the object candidate pixels exist in a certain ratio or more in the horizontal direction as an object candidate line. In step S176, the corresponding area detection unit 52 calculates the density of the object candidate line, and determines a line having a density higher than a predetermined value as an object line. In step S177, the area extraction unit 53 detects the circumscribed rectangle of the object line group as the object area in the parallax image.

Next, the three-dimensional positioning unit 54 shown in FIG. 4 is based on the distance from the parallax information to the object, and the distance on the image from the image center of the parallax image to the area center of the object on the parallax image. Determine the three-dimensional position. Specifically, the three-dimensional positioning unit 54 sets the center coordinates of the area of the object on the parallax image as (region_centerX, region_centerY), the image center coordinates of the parallax image as (image_centerX, image_centerY), and the image pickup unit 2 of the object. The horizontal position and the height position relative to the above are calculated by performing the following mathematical expressions.

Horizontal position = Z × (region_centerX-image_centerX) / f
Height position = Z × (region_centerY-image_centerY) / f

As is clear from the above description, the mobile control system of the embodiment creates a plurality of layer U maps with a layer structure based on height information from the road surface of the object. In other words, in the moving body control system of the embodiment, the layer U having a layer structure in which the disparity points on the disparity image obtained from the imaging unit 2 are set to different heights based on the actual height from the road surface. The parallax points in the parallax image are voted on the map, the grouping process is performed on the two-dimensional plane in each layer U map, and the grouping results are integrated or separated using the respective results.

Due to the erroneous detection of the road surface detection process, the side effect of the parallax interpolation process, or the influence of the dispersion of the parallax, when an object is detected from the layer U map, a plurality of objects are concatenated and erroneously detected as one object.

However, in the case of the moving body control system of the embodiment, since the actual height information from the road surface is used, the connected objects can be divided and detected as individual objects with the correct size, position and distance. As a result, highly accurate and high-speed object detection can be performed. In particular, humans can detect it with high accuracy. Therefore, the vehicle-mounted stereo camera (imaging unit 2) can be used to perform highly accurate vehicle control.

The embodiments described above are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiments and modifications of each embodiment are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

1 Vehicle 2 Imaging unit 3 Analysis unit 4 Control unit 5 Display 10A Imaging unit 10B Imaging unit 11A Imaging lens 11B Imaging lens 12A Image sensor 12B Image sensor 13A controller 13B controller 14 FPGA
15 RAM
16 ROM
17 CPU
18 Serial interface (serial IF)
19 Data IF
20 Serial bus line 21 Data bus line 41 Parallelized image generation unit 42 Disparity image generation unit 45 V map generation unit 46 Road surface shape detection unit 47 List generation unit 48 U map generation unit 49 Layer U map generation unit 50 Isolated area detection unit 51 Classification unit 52 Corresponding area detection unit 53 Area extraction unit 54 Three-dimensional positioning unit 60 Road surface object 61 Vehicle object 62 Electric pole object 71 Disparity division road surface candidate point detection unit 72 Division straight line approximation unit 73 Division approximation straight line continuation unit 74 Straight line correction unit 75 Correction point detection unit 77 Vehicle object 79 Vehicle object 81 Guard rail object 82 Guard rail object 83 Human object 84 Human object 85 Vehicle object 86 First person group 87 Second person group 88 Children Object 89 Adult Object 91 Adult Object 92 Adult Object 93 Electric Pillar Object

JP-A-2014-096005 International Publication No. 2012/017650

Claims (8)

A reference detector that detects a reference object that serves as a reference for the height of an object from a distance image that has distance information for each pixel.
Of a plurality of maps in which the height range from the reference object is set to different height ranges, with respect to the map having a height range corresponding to the height from the reference object in each pixel of the distance image. , A map generator that votes for the distance information of each pixel,
From each of the maps, a set detection unit that detects the set area of the distance information and
A classification unit for classifying the types of objects in the set area based on the distribution of the set area in each map is provided.
The map generation unit generates a two-dimensional histogram generated with the actual distance between the object on the distance image as the horizontal axis and the distance information of the distance image as the vertical axis as the map.
An image processing device characterized by . The classification unit classifies the types of objects in the set area based on the comparison result between the preset reference size of each object and the size of the set area, and the distribution of the set area in each map. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized in that. The map generation unit creates the map in which the target height ranges overlap.
The image processing apparatus according to claim 1 or 2. The distance image is a parallax image and is
The image processing apparatus according to any one of claims 1 to 3, wherein the distance information is a thinned-out parallax, which is a parallax value thinned out according to the actual distance. The image processing apparatus according to any one of claims 1 to 4.
With multiple imaging devices
A parallax image generation unit that generates a parallax image from a plurality of images captured by the plurality of image pickup devices is provided.
The image processing device is an imaging device characterized in that it classifies the types of objects based on the parallax image generated by the parallax image generation unit. The imaging device according to claim 5 is provided.
A mobile device control system that controls devices based on the types of the classified objects. A reference detection step in which the reference detection unit detects a reference object as a reference for the height of an object from a distance image having distance information for each pixel.
Of a plurality of maps in which the map generation unit sets the height range from the reference object to different height ranges, the height range corresponding to the height from the reference object in each pixel of the distance image. A map generation step for voting the distance information of each pixel with respect to the map,
A set detection step in which the set detection unit detects a set area of the distance information from each of the maps,
The classification unit includes a classification step of classifying the types of objects in the set area based on the distribution of the set area in each map .
In the map generation step, a two-dimensional histogram generated with the actual distance to the object on the distance image as the horizontal axis and the distance information of the distance image as the vertical axis is generated as the map.
An image processing method characterized by . Computer,
A reference detector that detects a reference object that serves as a reference for the height of an object from a distance image that has distance information for each pixel.
Of a plurality of maps in which the height range from the reference object is set to different height ranges, with respect to the map having a height range corresponding to the height from the reference object in each pixel of the distance image. , A map generator that votes for the distance information of each pixel,
From each of the maps, a set detection unit that detects the set area of the distance information and
Based on the distribution of the set area in each map, it functions as a classification unit for classifying the types of objects in the set area .
The map generation unit is characterized in that a two-dimensional histogram generated with the actual distance between the object on the distance image as the horizontal axis and the distance information of the distance image as the vertical axis is generated as the map. Image processing program.

Images