JP6907513B2 - Information processing equipment, imaging equipment, equipment control system, information processing method and program - Google Patents

Description

The present invention relates to an information processing device, an imaging device, a device control system, an information processing method and a program.

Conventionally, in terms of automobile safety, the body structure of an automobile has been developed from the viewpoint of how to protect pedestrians and occupants when a pedestrian collides with the automobile. However, in recent years, with the development of information processing technology and image processing technology, technology for detecting people and automobiles at high speed has been developed. By applying these technologies, automobiles have already been developed that automatically apply the brakes before the vehicle collides with an object to prevent the collision. For automatic vehicle control, it is necessary to accurately measure the distance to an object such as a person or another vehicle. For that purpose, distance measurement using millimeter-wave radar and laser radar, distance measurement using a stereo camera, etc. are practical. Has been converted. For example, when measuring a distance with a stereo camera, a parallax image can be generated based on the amount of deviation (parallax) of a local region taken by the left and right cameras, and the distance between the object in front and the own vehicle can be measured. Then, a clustering process is performed to detect a group of parallax pixels existing at the same distance (having the same parallax value) as one object.

Here, if all the parallax pixels (parallax points) are clustered, the parallax points of the white lines on the road surface can be found separately from the object to be detected, and a part of the road surface that should be flat can be misrecognized as an object. There is a problem that it is detected as. In this case, the system determines that there is an object in front of it, causing a problem of sudden braking. In order to solve this problem, each disparity point (x-coordinate value of the disparity image, y-coordinate value of the disparity image, disparity value d) is set on the horizontal axis as the disparity value d and the vertical axis as the x-coordinate value of the disparity image and the depth. Information (V-DisparityMap) obtained by voting on a two-dimensional histogram with the axis of direction as the frequency value is generated, and the road surface shape is obtained from the point cloud voted for this information using a statistical method such as the minimum square method. There is known a technique for avoiding detecting a road surface as a false perceptible object by estimating and clustering using only the disparity points existing at a position higher than a predetermined height of the estimated road surface (for example, Patent Documents). 1).

However, in the prior art, for example, in a scene where an object such as a vehicle exists in a situation where the parallax corresponding to the road surface is small, the parallax corresponding to the object is erroneously selected as the parallax corresponding to the road surface, and the estimated road surface becomes There is a problem that it will be pulled up more than it actually is. That is, in the prior art, there is a problem that it is difficult to sufficiently secure the detection accuracy of the object because the object is detected by using the estimated road surface different from the actual road surface.

The present invention has been made in view of the above, and an object of the present invention is to provide an information processing device, an imaging device, a device control system, an information processing method, and a program capable of sufficiently ensuring the detection accuracy of an object.

In order to solve the above-mentioned problems and achieve the object, the present invention is based on an acquisition unit that acquires a distance image having distance information for each pixel and a plurality of pixels included in the distance image in the vertical direction. An estimation unit that generates correspondence information in which a position and a position in the depth direction are associated with each other, and an estimation that estimates the shape of a reference object that serves as a reference for the height of the object for each of a plurality of segments in which the correspondence information is divided. parts and, for each of the segments, to the reference extension shape is extended to an adjacent segment of based rather shape the estimated shape indicating a shape of the reference object estimated by the estimating unit, than the predetermined shape It is provided with a rejection unit that rejects the estimated shape according to the distribution of the distance information existing below or the ratio of the total number of coordinates in which the pixels having the distance information are voted to the extension shape. It is an information processing device.

According to the present invention, it is possible to sufficiently secure the detection accuracy of the object.

FIG. 1 is a schematic diagram showing a schematic configuration of a mobile control system according to an embodiment. FIG. 2 is a schematic block diagram of an imaging unit and an analysis unit. FIG. 3 is a diagram showing the positional relationship between the subject and the image pickup lens of each camera unit. FIG. 4 is a diagram for schematically explaining the function of the analysis unit. FIG. 5 is a diagram showing an example of the function of the object detection processing unit. FIG. 6 is a diagram showing an example of the function of the road surface detection processing unit. FIG. 7 is a diagram showing an example of a parallax image. FIG. 8 is a diagram showing an example of a detailed configuration of the clustering processing unit. FIG. 9 is a diagram showing an example of a captured image. FIG. 10 is a diagram showing an example of the Adult Large Umap. FIG. 11 is a diagram showing an example of Large Umap. FIG. 12 is a diagram showing an example of a captured image. FIG. 13 is a diagram showing an example of an isolated region. FIG. 14 is a diagram showing a region on the parallax image corresponding to the isolated region shown in FIG. FIG. 15 is a diagram showing a size range defined for each object type. FIG. 16 is a diagram for explaining a rejection process. FIG. 17 is a diagram showing an example of the function of the road surface estimation unit. FIG. 18 is a diagram showing an example of a V map generated by the first generation unit. FIG. 19 is a diagram showing an example of a plurality of segments obtained by division by the division portion. FIG. 20 is a diagram for explaining a rejection determination of the first embodiment. FIG. 21 is a diagram for explaining a method of measuring the frequency value of the parallax value existing below the extended road surface. FIG. 22 is a diagram for explaining another example of a method of measuring the frequency value of the parallax value existing below the extended road surface. FIG. 23 is a diagram for explaining a default road surface. FIG. 24 is a flowchart showing an example of processing by the road surface detection processing unit. FIG. 25 is a diagram for explaining a rejection determination of the second embodiment.

Hereinafter, embodiments of the information processing device, the imaging device, the device control system, the information processing method, and the program according to the present invention will be described in detail with reference to the accompanying drawings.

(First Embodiment)
FIG. 1 is a schematic diagram showing a schematic configuration of the mobile control system 100 of the embodiment. As shown in FIG. 1, the mobile body control system 100 is provided in a vehicle 101 such as an automobile, which is an example of a mobile body. The mobile control system 100 includes an imaging unit 102, an analysis unit 103, a control unit 104, and a display unit 105.

The image pickup unit 102 is provided near the rearview mirror of the windshield 106 of the vehicle 101, and captures an image of the vehicle 101, for example, the traveling direction. Various data including the image data obtained by the imaging operation of the imaging unit 102 are supplied to the analysis unit 103. The analysis unit 103 analyzes a recognition target such as a road surface on which the vehicle 101 is traveling, a vehicle in front of the vehicle 1, a pedestrian, an obstacle, etc., based on various data supplied from the imaging unit 102. The control unit 104 warns the driver of the vehicle 101 or the like via the display unit 105 based on the analysis result of the analysis unit 103. Further, the control unit 104 performs running support such as control of various in-vehicle devices, steering wheel control of the vehicle 101, and brake control based on the analysis result.

FIG. 2 is a schematic block diagram of the imaging unit 102 and the analysis unit 103. In this example, the analysis unit 103 functions as an "information processing device", and the pair of the imaging unit 102 and the analysis unit 103 functions as an "imaging device". The control unit 104 described above functions as a "control unit" and controls the device (vehicle 101 in this example) based on the output result of the imaging device. The image pickup unit 102 is configured by assembling two camera units, a first camera unit 1A for the left eye and a second camera unit 1B for the right eye, in parallel. That is, the image pickup unit 102 is configured as a stereo camera that captures a stereo image. The stereo image is an image including a plurality of captured images (a plurality of captured images corresponding to one-to-one with a plurality of viewpoints) obtained by imaging for each of a plurality of viewpoints, and the imaging unit 102 captures the stereo images. (Functions as an "imaging unit"). Each camera unit 1A and 1B includes a lens 5, an image sensor 6, and a sensor controller 7, respectively. The image sensor 6 is, for example, a CCD image sensor or a CMOS image sensor. CCD is an abbreviation for "Charge Coupled Device". CMOS is an abbreviation for "Complementary Metal-Oxide Semiconductor". The sensor controller 7 performs exposure control of the image sensor 6, image readout control, communication with an external circuit, transmission control of image data, and the like.

The analysis unit 103 has a data bus line 10, a serial bus line 11, a CPU 15, an FPGA 16, a ROM 17, a RAM 18, a serial IF 19, and a data IF 20. CPU is an abbreviation for "Central Processing Unit". FPGA is an abbreviation for "Field-Programmable Gate Array". ROM is an abbreviation for "Read Only Memory". RAM is an abbreviation for "Random Access Memory". IF is an abbreviation for "interface".

The image pickup unit 102 described above is connected to the analysis unit 103 via a data bus line 10 and a serial bus line 11. The CPU 15 executes and controls the operation, image processing, and image recognition processing of the entire analysis unit 103. The luminance image data of the captured image captured by the image sensor 6 of the first camera unit 1A and the second camera unit 1B is written to the RAM 18 of the analysis unit 103 via the data bus line 10. The sensor exposure value change control data, the image reading parameter change control data, various setting data, and the like from the CPU 15 or the FPGA 16 are transmitted and received via the serial bus line 11.

The FPGA 16 is a process that requires real-time performance for the image data stored in the RAM 18. The FPGA 16 uses one of the luminance image data (captured images) captured by the first camera unit 1A and the second camera unit 1B as a reference image and the other as a comparative image. Then, the FPGA 16 calculates the amount of positional deviation between the corresponding image portion on the reference image and the corresponding image portion on the comparison image corresponding to the same point in the imaging region as a parallax value (parallax image data) of the corresponding image portion. ..

FIG. 3 shows the positional relationship between the subject 30 on the XZ plane, the image pickup lens 5A of the first camera unit 1A, and the image pickup lens 5B of the second camera unit 1B. In FIG. 3, the distance b between the imaging lenses 5A and 5B and the focal length f of the imaging lenses 5A and 5B are fixed values, respectively. Further, the amount of deviation of the X coordinate of the image pickup lens 5A with respect to the gazing point P of the subject 30 is set to Δ1. Further, the amount of deviation of the X coordinate of the image pickup lens 5B with respect to the gazing point P of the subject 30 is set to Δ2. In this case, the FPGA 16 calculates the parallax value d, which is the difference between the X coordinates of the imaging lenses 5A and 5B with respect to the gazing point P of the subject 30, by the following equation 1.

The FPGA 16 of the analysis unit 103 performs processing that requires real-time performance, such as gamma correction processing and distortion correction processing (parallelization of left and right captured images), on the luminance image data supplied from the imaging unit 102. Further, the FPGA 16 generates parallax image data (parallax value D) by performing the calculation of the above equation 1 using the luminance image data that has been subjected to such processing that requires real-time performance, and writes it to the RAM 15. ..

The explanation will be continued by returning to FIG. The CPU 15 controls each sensor controller 7 of the image pickup unit 102 and controls the analysis unit 103 as a whole. Further, the ROM 17 stores a three-dimensional object recognition program for executing situational awareness, prediction, three-dimensional object recognition, and the like, which will be described later. The three-dimensional object recognition program is an example of an image processing program. The CPU 15 acquires, for example, CAN information (vehicle speed, acceleration, steering angle, yaw rate, etc.) of the own vehicle as parameters via the data IF20. Then, the CPU 15 executes and controls various processes such as situation recognition using the luminance image and the parallax image stored in the RAM 18 according to the three-dimensional object recognition program stored in the ROM 17, for example, the preceding vehicle or the like. Recognize the recognition target. CAN is an abbreviation for "Controller Area Network".

The recognition data to be recognized is supplied to the control unit 104 via the serial IF 19. The control unit 104 uses the recognition data of the recognition target to perform running support such as brake control of the own vehicle and speed control of the own vehicle.

FIG. 4 is a diagram for schematically explaining the function of the analysis unit 103. The stereo image captured by the image pickup unit 102 constituting the stereo camera is supplied to the analysis unit 103. For example, when the first camera unit 1A and the second camera unit 1B have color specifications, each of the first camera unit 1A and the second camera unit 1B performs RGB ( A color luminance conversion process is performed to generate a luminance (Y) signal from each of the red, green, and blue) signals. Each of the first camera unit 1A and the second camera unit 1B supplies the luminance image data (captured image) generated by the color luminance conversion process to the preprocessing unit 111 included in the analysis unit 103. It can be considered that the set of the luminance image data (captured image) captured by the first camera unit 1A and the luminance image data (captured image) captured by the second camera unit 1B is a stereo image. In this example, the preprocessing unit 111 is realized by the FPGA 16.

The preprocessing unit 111 preprocesses the luminance image data received from the first camera unit 1A and the second camera unit 1B. In this example, gamma correction processing is performed as preprocessing. Then, the preprocessing unit 111 supplies the luminance image data after the preprocessing to the parallelized image generation unit 112.

The parallelization image generation unit 112 performs parallelization processing (distortion correction processing) on the luminance image data supplied from the preprocessing unit 111. This parallelization process is an ideal parallelization stereo image obtained when two pinhole cameras are mounted in parallel with the brightness image data output from the first camera unit 1A and the second camera unit 1B. It is a process to convert to. Specifically, the first camera unit 1A, the first camera unit 1A, using the calculation result calculated by using the polynomials Δx = f (x, y) and Δy = g (x, y) for the amount of distortion of each pixel. Each pixel of the brightness image data output from the camera unit 1B of 2 is converted. The polynomial is based on, for example, a fifth-order polynomial with respect to x (horizontal position of the image) and y (vertical position of the image). As a result, it is possible to obtain a parallel luminance image in which the distortion of the optical system of the first camera unit 1A and the second camera unit 1B is corrected. In this example, the parallelized image generation unit 112 is realized by the FPGA 16.

The parallax image generation unit 113 is an example of a “distance image generation unit”, and obtains a parallax value for each pixel, which is an example of a distance image having distance information for each pixel from a stereo image captured by the imaging unit 102. Generate a parallax image provided. Here, the parallax image generation unit 113 uses the brightness image data of the first camera unit 1A as the reference image data and the brightness image data of the second camera unit 1B as the comparison image data, and performs the calculation shown in the above equation 1. By doing so, disparity image data indicating the disparity between the reference image data and the comparison image data is generated. Specifically, the parallax image generation unit 113 defines a block composed of a plurality of pixels (for example, 16 pixels × 1 pixel) centered on one pixel of interest for a predetermined “row” of the reference image data. On the other hand, in the same "row" in the comparative image data, a block having the same size as the defined reference image data block is shifted one pixel at a time in the horizontal line direction (X direction). Then, the parallax image generation unit 113 sets a correlation value indicating the correlation between the feature amount indicating the characteristic of the pixel value of the block defined in the reference image data and the feature amount indicating the characteristic of the pixel value of each block in the comparison image data. Calculate each.

Further, the parallax image generation unit 113 performs a matching process of selecting the block of the comparison image data that has the most correlation with the block of the reference image data among the blocks of the comparison image data based on the calculated correlation value. After that, the amount of positional deviation between the pixel of interest of the block of the reference image data and the corresponding pixel of the block of the comparative image data selected by the matching process is calculated as the parallax value D. Parallax image data is obtained by performing such a process of calculating the parallax value D for the entire area of the reference image data or a specific area. As a method for generating a parallax image, various known techniques can be used. In short, it can be considered that the parallax image generation unit 113 calculates (generates) a distance image (parallax image in this example) having distance information for each pixel from the stereo image captured by the stereo camera.

As the feature amount of the block used for the matching process, for example, the value (luminance value) of each pixel in the block can be used. Further, as the correlation value, for example, the difference between the value (brightness value) of each pixel in the block of the reference image data and the value (brightness value) of each pixel in the block of the comparative image data corresponding to each of these pixels. The sum of the absolute values of can be used. In this case, the block with the smallest sum is detected as the most correlated block.

Examples of the matching process of the parallax image generation unit 113 include SSD (Sum of Squared Difference), ZSD (Zero-mean Sum of Squared Difference), SAD (Sum of Absolute Difference), or ZSAD (Zero-mean Sum). A method such as of Absolute Difference) can be used. If a subpixel level parallax value of less than one pixel is required in the matching process, an estimated value is used. As a method for estimating the estimated value, for example, an isometric straight line method, a quadratic curve method, or the like can be used. However, an error occurs in the estimated sub-pixel level parallax value. Therefore, a method such as EEC (estimation error correction) that reduces the estimation error may be used.

In this example, the parallax image generation unit 113 is realized by the FPGA 16. The parallax image generated by the parallax image generation unit 113 is supplied to the object detection processing unit 114. In this example, the function of the object detection processing unit 114 is realized by the CPU 15 executing the three-dimensional object recognition program.

FIG. 5 is a diagram showing an example of the function of the object detection processing unit 114. As shown in FIG. 5, the object detection processing unit 114 includes a road surface detection processing unit 122, a clustering processing unit 123, and a tracking processing unit 124.

The road surface detection processing unit 122 detects a road surface which is an example of a reference object that serves as a reference for the height of the object by using the parallax image input from the parallax image generation unit 113. As shown in FIG. 6, the road surface detection processing unit 122 includes an acquisition unit 125, a first generation unit 126, and a road surface estimation unit 127. The acquisition unit 125 acquires a parallax image which is an example of a distance image having distance information for each pixel. The parallax image acquired by the acquisition unit 125 is input to the first generation unit 126 and the clustering processing unit 123 described later.

The first generation unit 126 is an example of the “generation unit”, and is in the depth direction indicating the vertical position of the parallax image and the direction of the optical axis of the stereo camera based on a plurality of pixels included in the parallax image. Generates correspondence information associated with the position. In this example, the first generation unit 126 votes each pixel of the parallax image on a two-dimensional histogram having the coordinates (y) in the vertical direction of the image as the vertical axis and the parallax value d as the horizontal axis. Generate the correspondence information of. In the following description, this correspondence information will be referred to as a "V map (V-Disparity map)". In the V map, among the sets of (x coordinate value, y coordinate value, difference value d) of the difference image, the horizontal axis (x axis) is the difference value d, the vertical axis (y axis) is the y coordinate value, and the depth direction. It is a two-dimensional histogram with the axis (z-axis) as a frequency. In short, the V-map can be considered as information that records the frequency value of the parallax value d for each combination of the position in the vertical direction and the parallax value d (corresponding to the position in the depth direction). In the following description, among the coordinate points in the V map, the coordinates in which the pixel having the parallax value d included in the parallax image (parallax pixel) is voted may be referred to as the parallax point. In the generation of the V-map, the y-coordinate of the differential image and the y-coordinate of the V-map have a corresponding relationship, and the differential value d on the horizontal line of the specific y-coordinate of the differential image is the corresponding y-coordinate of the V-map. Of the horizontal lines of, the points (coordinate points on the V map) corresponding to the disparity value d are voted. Therefore, since some parallax values d included in the same horizontal line of the parallax image have the same value, a frequency value indicating the number of parallax values d of the same value is stored at an arbitrary coordinate point of the V map. become. In a specific horizontal line of the parallax image, if the road surface is the same, the parallax values d are similar to each other, so that the parallax pixels corresponding to the road surface in the V map are densely voted.

The first generation unit 126 may vote for all the parallax pixels in the parallax image, but like the parallax image Ip shown in FIG. 7, a predetermined region (for example, the voting region 701 shown in FIG. 7). ~ 703) may be set and only the parallax pixels included in the area may be voted. For example, by utilizing the property that the road surface becomes narrower toward the vanishing point as the road surface becomes farther, as shown in FIG. 7, a method of setting a predetermined number of voting areas suitable for the width of the road surface can be considered. By limiting the voting area in this way, it is possible to prevent noise other than the road surface from being mixed into the V-map. Further, the parallax pixels in one horizontal line in the parallax image may be appropriately thinned out for voting. Further, the thinning may be performed not only in the horizontal direction but also in the vertical direction.

The V map generated by the first generation unit 126 is input to the road surface estimation unit 127 shown in FIG. The road surface estimation unit 127 selects a sample point from the voted disparity points in the V map by a predetermined method, and estimates the shape of the road surface by linearly approximating (or curving) the selected point cloud. Here, the reference object, which is the reference for the height of the object, corresponds to the road surface. The specific contents of the road surface estimation unit 127 will be described later. The estimation result (road surface estimation information) by the road surface estimation unit 127 is input to the clustering processing unit 123.

The clustering processing unit 123 detects the position of the object on the parallax image acquired by the acquisition unit 125 by using the road surface estimation information. FIG. 8 is a diagram showing an example of a detailed configuration of the clustering processing unit 123. As shown in FIG. 8, the clustering processing unit 123 includes a second generation unit 130, an isolated region detection processing unit 140, a parallax image processing unit 150, and a rejection processing unit 150. The second generation unit 130 uses a plurality of pixels existing in a range higher than the road surface (an example of a reference object) in the parallax image, and has a lateral position indicating a direction orthogonal to the optical axis of the stereo camera. , Generates a second correspondence information associated with a position in the depth direction indicating the direction of the optical axis of the stereo camera. In this example, the second correspondence information is that the horizontal axis (X-axis) is the actual distance in the horizontal direction (actual distance), the vertical axis (Y-axis) is the parallax value d of the parallax image, and the axis in the depth direction (Z-axis). ) Is a two-dimensional histogram with a frequency. The second correspondence information can be considered to be information that records the frequency value of parallax for each combination of the actual distance and the parallax value d.

Here, since the linear equation representing the road surface is obtained by the road surface estimation of the road surface estimation unit 127 described above, if the parallax d is determined, the corresponding y coordinate y0 is determined, and this coordinate y0 becomes the height of the road surface. For example, when the parallax value is d and the y coordinate is y', the height from the road surface when y'−y0 is the parallax value d is indicated. The height H of the above-mentioned coordinates (d, y') from the road surface can be obtained by the arithmetic expression H = (z × (y'−y0)) / f. In this calculation formula, "z" is the distance calculated from the parallax value d (z = Bf / (d-offset)), and "f" is the unit of the focal length of the imaging unit 102 (y'-y0). It is a value converted to the same unit as. Here, BF is a value obtained by multiplying the baseline length B of the imaging unit 102 by the focal length f, and offset is the parallax when an object at infinity is photographed.

The second generation unit 130 generates at least one of "Adult Large Umap", "Large Umap", and "Small Umap" as the second correspondence information. Hereinafter, these maps will be described. First, "Adult Large Umap" will be described. Assuming that the horizontal position of the parallax image is x, the vertical position is y, and the parallax value set for each pixel is d, the second generation unit 130 is the first parallax image higher than the road surface. Parallax the horizontal axis by voting based on the value of (x, d) the points (x, y, d) existing in the second range indicating the range of height equal to or higher than the predetermined value in the range. A two-dimensional histogram is generated in which x of the image, the vertical axis is the parallax value d, and the axis in the depth direction is the frequency. Then, the horizontal axis of this two-dimensional histogram is converted into an actual distance to generate an Adult Range Umap.

For example, in the captured image shown in FIG. 9, a person group 1 including an adult and a child, a person group 2 between adults, a pole, and a vehicle are reflected. In this example, a range in which the actual height from the road surface is 150 cm to 200 cm is set as the second range, and the Parallax Umap in which the parallax value d in the second range is voted is as shown in FIG. The parallax value d of a child with a height of less than 150 cm will not appear on the map because it is not voted. The vertical axis is the thinned parallax obtained by thinning out the parallax value d using the thinning rate according to the distance. The Adult Large Umap generated by the second generation unit 130 is input to the isolated region detection processing unit 140.

Next, "Large Umap" will be described. Assuming that the horizontal position of the parallax image is x, the vertical position is y, and the parallax value set for each pixel is d, the second generation unit 130 exists within the first range of the parallax image. By voting the points (x, y, d) based on the values of (x, d), the horizontal axis is the x of the parallax image, the vertical axis is the parallax value d, and the axis in the depth direction is the frequency. Generate a parallax. Then, the horizontal axis of this two-dimensional histogram is converted into an actual distance to generate a Large Umap. In the example of FIG. 9, a range of 0 cm to 200 cm (including the second range described above) is set as the first range, and the Parallax Umap in which the parallax value d of the first range is voted is shown in FIG. become that way. Further, the second generation unit 130, together with the Large Umap, has the highest height (h) from the road surface among the parallax points (the pair of the actual distance and the parallax value d) voted for by the Large Umap. It is also possible to record the height of the difference point, set the horizontal axis as the actual distance (distance in the left-right direction of the camera) and the vertical axis as the parallax value d, and generate height information in which the height is recorded for each corresponding point. can. The height information may be considered as information that records the height for each combination of the actual distance and the parallax value d. In the following description, this height information will be referred to as a "Large Umap height map". The position of each pixel included in the "Large Umap height map" corresponds to the position of each pixel included in the Large Umap. The height maps of Large Umap and Large Umap generated by the second generation unit 130 are input to the isolated area detection processing unit 140.

Next, "Small Umap" will be described. Assuming that the horizontal position of the parallax image is x, the vertical position is y, and the parallax value set for each pixel is d, the second generation unit 130 exists within the first range of the parallax image. By voting the points (x, y, d) based on the value of (x, d) (voting a smaller number than when creating a Large Umap), the horizontal axis is x of the parallax image and the vertical axis is the vertical axis. A two-dimensional histogram is generated with the parallax value d and the axis in the depth direction as the frequency. Then, the horizontal axis of this two-dimensional histogram is converted into an actual distance to generate Small Umap. Small Umap has a lower resolution of one pixel than Large Umap. Further, the second generation unit 130, together with Small Umap, has the highest height (h) from the road surface among the parallax points (the pair of the actual distance and the parallax value d) voted for Small Umap. It is also possible to record the height of the difference point, set the horizontal axis as the actual distance (distance in the left-right direction of the camera) and the vertical axis as the parallax value d, and generate height information in which the height is recorded for each corresponding point. can. The height information may be considered as information that records the height for each combination of the actual distance and the parallax value d. In the following description, this height information will be referred to as "Small U map height map". The position of each pixel included in "Small Umap height map" corresponds to the position of each pixel included in Small Umap. The height maps of the Small Umap and Small U maps generated by the second generation unit 130 are input to the isolated area detection processing unit 140.

In this example, the case where the second generation unit 130 generates the Large Umap and the generated Large Umap is input to the isolated area detection processing unit 140 will be described as an example, but the present invention is not limited to this, for example. When object detection is performed using "Input Large Umap", "Large Umap", and "Small Umap", the second generation unit 130 generates "Adult Large Umap", "Large Umap", and "Small Umap". Then, these maps may be input to the isolated area detection processing unit 140.

The explanation will be continued by returning to FIG. The isolated region detection processing unit 140 detects an isolated region (aggregate region) which is a region of a mass having a parallax value d from the above-mentioned second correspondence information (Large Umap in this example). For example, in the case of the captured image shown in FIG. 12, there are guardrails 81 and 82 on the left and right, and the vehicle 77 and the vehicle 79 are facing each other with the center line in between. One vehicle 77 or 79 is traveling in each traveling lane. There are two poles 80A and 80B between the vehicle 79 and the guardrail 82. FIG. 13 is a Large Umap obtained based on the captured image shown in FIG. 12, and the region surrounded by the frame corresponds to the isolated region.

The parallax image processing unit 150 shown in FIG. 8 performs parallax image processing for detecting a region on the parallax image corresponding to the isolated region detected by the isolated region detection processing unit 140 and object information in the real space. FIG. 14 is a diagram showing a region on the parallax image (result of processing by the parallax image processing unit 150) corresponding to the isolated region shown in FIG. 13, and region 91 in FIG. 14 is a region corresponding to the guardrail 81. The area 92 is an area corresponding to the vehicle 77, the area 93 is an area corresponding to the vehicle 79, the area 94 is an area corresponding to the pole 80A, and the area 95 is an area corresponding to the pole 80B, and the area 96. Is an area corresponding to the guardrail 82.

The rejection processing unit 160 shown in FIG. 8 performs a rejection process for selecting objects to be output based on the object information in the parallax image region and the real space detected by the parallax image processing unit 150. The rejection processing unit 160 executes size rejection focusing on the size of the object and overlap rejection focusing on the positional relationship between the objects. For example, in size rejection, the detection result of a size that does not fall within the size range defined for each object type shown in FIG. 15 is rejected. For example, in the example of FIG. 16, regions 91 and 96 are rejected. Further, in the overlap rejection, the result of overlapping is selected for the regions corresponding to the isolated regions on the parallax image detected by the parallax image processing.

The output information (detection result) from the clustering processing unit 123 is input to the tracking processing unit 124 shown in FIG. The tracking processing unit 124 determines that the detection result (detected object) by the clustering processing unit 123 is a tracking target when it appears continuously over a plurality of frames, and if it is a tracking target, the detection result. Is output to the control unit 104 as an object detection result. The control unit 104 actually controls the vehicle 101 based on the object detection result.

Hereinafter, the specific contents of the road surface estimation unit 127 (FIG. 6) for estimating the shape of the road surface, which is an example of the reference object, will be described. FIG. 17 is a diagram showing an example of the function of the road surface estimation unit 127. As shown in FIG. 17, the road surface estimation unit 127 includes a division unit 171, an estimation unit 172, a rejection unit 173, and an interpolation unit 174.

The division unit 171 divides the V map (correspondence information) input from the first generation unit 126 into a plurality of segments. In this example, the division unit 171 divides the V-map into a plurality of segments continuous in the depth direction (the direction of the parallax value d, the direction of the horizontal axis of the V-map). However, the present invention is not limited to this, and for example, the parallax image may be divided in the y direction (vertical direction of the V map). Moreover, the position of the segment can be set to an arbitrary position. Normally, it is desirable to make the segments continuous, but it may be discontinuous (for example, when the estimation in a predetermined distance range (d value) is not performed intentionally). In this embodiment, two or more segments are set. The segments may be set with a predetermined width instead of being set at equal intervals. For example, since it is known that the distant region has a low resolution (the road surface resolution is low), it is conceivable to divide the segment into smaller pieces as the distance goes far. Therefore, the number of segments may be determined according to the above.

For example, it is assumed that the first generation unit 126 generates the V map shown in FIG. 18 (B) from the parallax image Ip2 shown in FIG. 18 (A), and this V map is input to the division unit 171. do. The parallax image Ip2 shown in FIG. 18A shows the road surface 600 and the light truck 601. The light track 600 in the parallax image Ip2 corresponds to the voting point group (parallax point group) shown in 603 in the V map shown in FIG. 18 (B). The dividing unit 171 divides the V map shown in FIG. 18B into a plurality of segments divided within a predetermined d-coordinate range. FIG. 19 is a diagram showing an example of a plurality of segments obtained by division by the division unit 171. In order from the right, the first segment seg1, the second segment seg2, the third segment seg3, the fourth segment seg4, and the first segment are shown. It is referred to as 5 segment seg5, 6th segment seg6, and 7th segment seg7.

The description of FIG. 17 will be continued. The estimation unit 172 estimates the shape of the road surface (an example of a reference object) for each of a plurality of segments in which the corresponding information is divided. More specifically, the estimation unit 172 performs the following processing for each segment. First, the estimation unit 172 describes each coordinate (hereinafter, may be referred to as “d coordinate”) in the direction (depth direction) of the parallax value d in the segment to be processed (hereinafter, may be referred to as “target segment”). From the position, a predetermined number (for example, one point) of representative points (hereinafter, referred to as “sample points”) are selected. As a method of selecting a sample point, for example, for each d-coordinate, the most frequent disparity point (most frequent point) may be simply selected from the disparity points existing in the vertical (vertical) direction. Or, the d-coordinate of interest and a plurality of pixels on the left and right of the d-coordinate are raised from the bottom to the top of the V-map to limit the area where the discrepancy point of the road surface can be included, and then the most frequent of them. A method of capturing the difference point on the road surface more accurately, such as selecting a point, may be used. Alternatively, a position (coordinates) without a discrepancy point may be selected as a sample point. For example, if there is no discriminant point at the coordinate (d, y) of interest, but frequent discriminant points are concentrated around it, the discriminant point at the coordinate (d, y) accidentally becomes. Since it may be missing, it is possible to select this missing position as the sample point.

Further, the estimation unit 172 may remove an inappropriate sample point from the sample points selected as described above. As a result, it is possible to prevent the road surface estimation result from becoming inappropriate due to the influence of an inappropriate sample point (off point) in the case of linear approximation with respect to the sample point group described later. As a method for removing the outliers, for example, all the sample points in the target segment may be linearly approximated by the least squares method, and the sample points separated from the approximate straight line by a predetermined distance may be removed. In this case, the final estimation result is the result estimated by the least squares method again with the outliers removed.

The estimation unit 172 estimates the shape of the road surface using the remaining sample points. As a method of estimating the shape of the road surface, for example, there are a method of performing a straight line approximation to a sample point cloud by a least squares method or the like, a method of estimating a curve shape using a polynomial approximation or the like, and the like. At the same time, a numerical scale such as a correlation coefficient based on these methods may be calculated for use in the success / failure judgment of the latter stage (success / failure judgment based on the result of estimating the shape of the road surface). In the following description, unless otherwise specified, the shape estimation of the road surface will be described by linear approximation. Further, the estimation result of the shape of the road surface may be referred to as an estimated road surface.

Here, for example, it is assumed that the parallax pixels in the trapezoidal region 711 in the parallax image Ip2 of FIG. 18A are the voting targets at the time of V-map generation. Of the two regions 712 and 713 included in the region 711, the region 712 represents a region in which parallax pixels (pixels having a parallax value) on the road surface exist, and the region 713 is a region in which the parallax pixels on the road surface do not exist. It shall be represented. Therefore, in the correspondence shown in FIG. 18, assuming that the parallax pixels existing at the y-coordinate of the parallax image Ip2 are voted for the corresponding coordinates (d, y) on the V-map, the parallax corresponding to the road surface in the region 712. The pixels are voted as the road surface, but are not voted because there are no parallax pixels corresponding to the road surface in the area 713. Further, since the distance from the camera of the light truck 601 is different between the loading platform portion 612 and the cabin portion 611, the parallax pixels corresponding to each are voted for different segments on the V map. Further, since the light truck 601 has a loading platform cover, the distance gradually changes from the bottom to the top of the image, which is similar to the parallax distribution of the road surface on the V map. Therefore, the estimated road surface is affected by the parallax (object parallax) corresponding to the light truck, and is estimated at a higher position than the correct road surface (actual road surface).

Therefore, in the present embodiment, the estimated road surface in the segment is extended to set the extended road surface for each segment, and when the frequency value of the parallax value d existing below the extended road surface exists more than a certain level, the said It is judged that the estimated road surface corresponding to the extended road surface is pulled up due to the influence of the object parallax, and the corresponding estimated road surface is rejected (the estimation is judged to be a failure). The specific contents will be described below.

The rejection unit 173 shown in FIG. 17 is a parallax of the pixels voted based on a predetermined shape set based on the estimated road surface (estimated shape indicating the shape of the road surface) estimated by the estimation unit 172 for each segment. If the distribution of the value d (more specifically, the distribution of voting points on the V map (distribution of the frequency value of the parallax value d)) meets a predetermined criterion, the estimated road surface is rejected. The predetermined shape includes a shape obtained by extending a shape based on an estimated road surface to an adjacent segment. In this example, the shape obtained by extending the estimated road surface to the adjacent segment as it is (hereinafter, referred to as “extended road surface”) is defined as the above-mentioned predetermined shape, but the above-mentioned predetermined shape is not limited to this, and the above-mentioned predetermined shape is a margin described later. It may be a line. The rejection unit 173 determines whether or not to reject the estimated road surface according to the distribution of the frequency values of the parallax value d existing below the extended road surface. More specifically, the rejection unit 173 rejects the estimated road surface when the frequency value of the parallax value d existing below the extended road surface is equal to or greater than the threshold value.

Normally, when the correct road surface can be estimated like the estimated road surface B in the 7-segment segment 7 of FIG. 20, the parallax value existing below the extended road surface B extending the estimated road surface B to the adjacent 6-segment seg6. The frequency value of d is small (or does not appear). On the other hand, when the road surface is pulled up due to object parallax like the estimated road surface A in the third segment seg3, an object is below the extended road surface A in which the estimated road surface A is extended to the adjacent second segment seg2. There is parallax (parallax corresponding to the loading platform 612). Therefore, the rejection unit 173 measures the frequency value of the parallax value d existing below the extended road surface A, and rejects the estimated road surface A when the frequency value of the parallax value d is equal to or greater than the threshold value. Such processing is executed for each segment. In addition to the above processing, different success / failure judgments such as success / failure judgment based on the angle and success / failure judgment based on the dispersion of the sample point group may be applied together as the success / failure judgment of the estimated road surface. As an example of success / failure judgment based on the angle, there is a mode in which the estimated road surface is rejected when the actual angle of the estimated road surface exceeds a predetermined value. Further, as an example of success / failure judgment based on the dispersion of the sample point cloud, there is an embodiment in which the estimated road surface is rejected because the shape of the road surface estimated from the scattered point cloud is unreliable.

In reality, the road surface parallax may be dispersed (for example, the parallax accuracy deteriorates as the distance increases). Therefore, a predetermined margin line is provided as shown in FIGS. 20A and 20 to provide a margin line. The measurement range may be below. In short, the predetermined shape may include a shape obtained by extending a shape based on an estimated road surface to an adjacent segment. The "shape based on the estimated road surface" may be the estimated road surface itself or a margin line. Further, with respect to the extended road surface, a segment closer to the segment of interest may be used, or a segment farther away may be used. Of course, you may use the road surface extended to both. Further, the length to be extended can also be set to a predetermined length. For example, it may be extended by one segment or longer. Further, the extension is not limited to the segment unit, and may be extended by a predetermined distance. Further, the range to which this process is applied may be limited. For example, if the road surface estimation fails in the near vicinity, considering the risk that the object in front will be unrecognized when the estimated road surface is pulled up, only for the segment closer than the predetermined segment. This process may be executed. Of course, it can be executed only for distant segments, so the scope of application is arbitrary.

Next, a method of measuring the frequency value of the parallax value d existing below the extended road surface will be described. In the present embodiment, the rejection unit 173 has a frequency value of the parallax value d existing below the extension road surface for each position in the depth direction (for each position in the direction of the parallax value d of the V map (horizontal axis direction)). Generates a frequency histogram associated with the frequency values that count. Then, with reference to the frequency histogram, the number of bins indicating the number of positions in the depth direction in which the corresponding frequency value is equal to or greater than a predetermined value is measured, and when the ratio of the number of bins to the segment length is equal to or greater than the threshold value, the extended road surface Reject the estimated road surface corresponding to. Since the frequency value of the parallax value d is stored in each coordinate of the V map, the total value of the frequency values may be used when creating the frequency histogram, or the frequency value becomes a predetermined value or more. You may count the number of coordinates. Then, the number of coordinates of the frequency histogram associated with the frequency value equal to or higher than the bin judgment threshold value (the number of coordinates in the direction of the parallax value d) is counted, and the number of bins equal to or higher than the bin judgment threshold value in the segment length Calculate the ratio (bin ratio). Then, when the bin ratio is equal to or higher than the threshold value (referred to as “ratio threshold value”), it is determined that the object parallax exists below the extended road surface (or the margin line).

For example, in the example of FIG. 21, the frequency histogram A corresponding to the second segment seg2 has 2 bins equal to or greater than the bin determination threshold value. For example, if the ratio threshold value is 50%, the segment length (corresponding to 3 bins). ), Since the ratio of the number of bins equal to or greater than the bin determination threshold value is equal to or greater than the ratio threshold value, it is determined that there is object parallax below the extension road surface A, and the estimated road surface A corresponding to the extension road surface A is rejected. On the other hand, in the frequency histogram B corresponding to the sixth segment seg6, since there is no bin exceeding the bin determination threshold value, the bin ratio is less than the ratio threshold value, and it is determined that there is no object parallax below the extended road surface B. However, the estimated road surface B corresponding to the extended road surface B is not rejected. Here, the reason for using the bin determination threshold is to make it robust against noise. If the number of bins having a frequency of 1 or more is counted without setting the bin determination threshold value, the bin ratio tends to exceed the ratio threshold value in a scene such as rainy weather with a lot of noise. Of course, this bin determination threshold may not be provided (the bin determination threshold may be 0).

Alternatively, as another method, the rejection unit 173 is a predetermined number (which may be 1 or a number larger than 1 as a noise countermeasure) occupying a predetermined area below the extension road surface (or margin line) in the V map. If the ratio of the total number of coordinates (coordinates having a frequency value of a predetermined number or more) voted for the parallax pixels or more is equal to or more than the threshold value, the estimated road surface corresponding to the extended road surface can be rejected. .. The shape of the predetermined region is arbitrary, but as shown in FIG. 22, for example, it may be trapezoidal in order to preferably capture the region below the margin line (or may be an extension road surface), but the present invention is not limited to this. , For example, it may be a rectangle. In the example of FIG. 22, since more than half of the coordinates in which a predetermined number or more of parallax pixels are voted exist in the measurement area A, for example, when the threshold value is 50%, the rejection unit 173 is larger than the margin line A. It is determined that the object parallax exists below, and the estimated road surface A corresponding to the margin line A is rejected. On the other hand, since there are no coordinates in the measurement area B in which a predetermined number or more of parallax pixels have been voted, the rejection unit 173 determines that there is no object parallax below the margin line B, and sets the margin line B. The corresponding estimated road surface B is not rejected.

Returning to FIG. 17, the description will be continued. The interpolation unit 174 is an example of the “setting unit”, and when the estimated road surface corresponding to the segment is rejected by the rejection unit 173, a predetermined road surface (predetermined shape) is set as the shape of the road surface corresponding to the segment. (Interpolate). As an example of a predetermined road surface, there is a default road surface (default shape) assuming a flat shape, or a historical road surface (history shape) showing a shape estimated in a past frame. FIG. 23 (A) shows a parallax image Ip3 when the vehicle is traveling on a flat road surface, and FIG. 23 (B) shows a V-map generated from the parallax image Ip3. As shown in FIG. 23 (B), when traveling on a flat road surface, the estimated road surface ER1 substantially coincides with the default road surface DR, which is a road surface assumed to be a flat road surface. The default road surface DR can be calculated in advance from the mounting height of the camera and the pitching angle. Further, the historical road surface indicates a past estimated road surface estimated in a frame one frame or more before, and may be a road surface obtained by averaging the road surfaces estimated in a predetermined number of frames in the past.

FIG. 24 is a flowchart showing an example of processing by the road surface detection processing unit 122 of the present embodiment. Since the specific contents of each step are as described above, detailed description thereof will be omitted as appropriate. First, the acquisition unit 125 acquires a parallax image (step S1). The acquisition unit 125 may directly acquire the parallax image generated by the parallax image generation unit 113, or store the parallax image in advance in a recording medium such as a CD, DVD, or HDD or network storage, and when necessary. You may read and use these. Further, only one parallax image may be acquired, or moving image data may be acquired sequentially for each frame. It is also possible to construct a V-map in advance and input it to the road surface detection processing unit 122. In this case, step S1 and the next step S2 are skipped, and the process starts from step S3.

Next, the first generation unit 126 generates a V map using the parallax image acquired in step S1 (step S2). The specific contents are as described above.

Next, the road surface estimation unit 127 (division unit 171) divides the V map generated in step S2 into a plurality of segments (step S3). The specific contents are as described above.

The following processes of steps S4 to S7 are repeatedly executed for the number of segments. Here, after the processing of steps S4 to S7 is completed for one segment, the processing of steps S4 to S7 is executed for the next segment, but the configuration is not limited to this. For example, it may be in the form of moving to the next step after executing the processing of each step for all the segments. For example, after the processing of step S4 is executed for all the segments, the process proceeds to step S5.

In step S4, the road surface estimation unit 127 (estimation unit 172) performs a sample point search for each d coordinate (step S4). At this time, the sample points are not limited to one point, and a plurality of sample points may be determined. Further, since there are d-coordinates in which the parallax does not exist in the vertical direction, there may be d-coordinates in which the sample point is not determined. The specific contents are as described above.

In step S5, the road surface estimation unit 127 (estimation unit 172) estimates the shape of the road surface (step S5). The specific contents are as described above.

In step S6, the road surface estimation unit 127 (rejection unit 173) sets an extension road surface that extends the estimated road surface, and determines whether or not the frequency value of the parallax value d existing below the extension road surface is equal to or greater than the threshold value. .. That is, the road surface estimation unit 127 (rejection unit 173) determines whether or not to reject the estimated road surface (step S6). The specific contents are as described above.

If the result of step S6 is negative (step S6: No), the estimated road surface is adopted as it is. On the other hand, when the result of step S6 is affirmative (step S6: Yes), the estimated road surface is rejected, and the road surface estimation unit 127 (interpolation unit 174) has the above-mentioned predetermined road surface (for example, default) as the road surface corresponding to the segment. The road surface, historical road surface, etc.) are set (interpolated) as a new estimated road surface (step S7).

Not limited to the above, for example, a process for removing the above-mentioned deviation point may be inserted between steps S4 and S5. Further, after the processes of steps S4 to S7 are completed for all the segments, a smoothing process for correcting the estimated road surface between the segments so as to be smoothly continuous may be performed. As an example of smoothing processing, for example, of the estimated road surfaces of two segments, the d coordinate corresponding to the start point of one estimated road surface and the d coordinate corresponding to the end point of the other estimated road surface (the end point when there is no break between the segments). And the start point point to the same d-coordinate) so that it passes through the predetermined y-coordinate position (correction agrees with the change of the slope and intercept in the V-map of the estimated road surface). May be good. This smoothing process ensures the continuity of the estimated road surface across all segments. As the predetermined y-coordinate position, for example, a method of using the y-coordinate of the midpoint between the y-coordinate corresponding to the start point and the y-coordinate corresponding to the end point can be considered. The smoothing process has the effect of improving the accuracy of road surface estimation because it may be corrected when the estimated road surface in a certain segment is not suitable. The smoothed estimated road surface is the final result. Further, in this smoothing process, each time the processes of steps S4 to S7 for one segment are completed, the smoothing process of the estimated road surface corresponding to the one segment and the estimated road surface corresponding to the previous segment is performed. It may be in the form of performing. In addition, the form may be in a form in which the process of removing the outliers or the smoothing process is not performed.

As described above, in the present embodiment, the V map is divided into a plurality of segments, and the road surface is estimated for each segment. Then, an extension road surface that extends the estimated road surface is set for each segment, and when the frequency value of the parallax value d existing below the extension road surface exists above a certain level, the estimated road surface corresponding to the extension road surface is the object parallax. It is judged that the road surface has been pulled up due to the influence of the above, and the estimated road surface is rejected (the estimation is judged to be a failure). As a result, it is possible to prevent the object detection from being performed using an estimated road surface different from the actual road surface, and as a result, it is possible to sufficiently secure the detection accuracy of the object.

(Second embodiment)
Next, the second embodiment will be described. The description of the parts common to the first embodiment described above will be omitted as appropriate. The basic configuration is the same as that of the first embodiment described above, but in the present embodiment, the rejection unit 173 is a pixel (parallax) having a parallax value d in the predetermined shape (extended road surface or margin line). If the percentage of the total number of coordinates voted for (pixels in the image) is less than the threshold, the estimated road surface is rejected.

Here, for example, it is assumed that the parallax pixels in the trapezoidal region 811 in the parallax image Ip4 of FIG. 25 (A) are the voting targets at the time of V-map generation. Of the three regions 812, 813, and 814 included in the region 811, the region 812 and the region 814 represent a region in which the parallax pixels exist, and the region 813 represents a region in which the parallax pixels do not exist. When the road surface parallax is small or does not exist, and when an upright object such as a large truck 820 is present, the estimated road surface may be pulled up by the object parallax. For example, when a large truck 820 is present as shown in FIG. 25, the parallax corresponding to the truck 820 is distributed in the vertical direction in the corresponding segment on the V map. Normally, if the road surface parallax is sufficient and the road surface can be estimated correctly, the estimated road surface exists on the voting point group corresponding to the road surface parallax. Therefore, the extended road surface also exists on the voting point group corresponding to the parallax of the road surface. However, when the estimated road surface is pulled up with an inappropriate inclination due to object parallax, the extended road surface deviates from the distribution of coordinates (point cloud) in which the pixels with parallax (pixels in the parallax image) are voted. It will be estimated at the position. As shown in FIG. 25 (B), the extended road surface B obtained by extending the estimated road surface B is estimated on the distribution of the coordinates in which the pixels having parallax are voted, but it is not possible due to the object parallax like the estimated road surface A. When properly pulled up, the extended road surface A will be in a region where the pixels with parallax are not on the distribution of the voted coordinates.

Therefore, in the present embodiment, the estimated road surface is extended and the extended road surface is set for each segment, and the coordinates at which the pixels having parallax are voted on the extended road surface are the points with respect to the length of the extended road surface. It counts whether it exists and determines whether the ratio (number of counts / length of extended road surface (segment length may be used)) is equal to or greater than the threshold value. If it is equal to or higher than the threshold value, it is considered that the estimated road surface in the segment of interest has correctly picked up the road surface parallax and the road surface can be estimated, and the estimation is successful (the estimated road surface is not rejected). On the other hand, if it is less than the threshold value, it is judged that the estimation has failed, the estimated road surface in the segment of interest is rejected, and the default road surface or the historical road surface is set.

Here, it was explained that the number of coordinate points on the extended road surface (the number of coordinates in which pixels with parallax were voted) is counted, but if only the points on the line are focused on, it will be affected by a slight difference in inclination. There is a possibility that it will not be possible to measure correctly. Therefore, a margin line may be provided to count the number of coordinate points in the region sandwiched between the extension road surface and the margin line. Here, it can be considered that the region sandwiched between the extension road surface and the margin line corresponds to the above-mentioned predetermined shape (a predetermined shape set based on the estimated road surface). The margin line is not limited to the downward direction of the extension road surface, and may be provided in the upward direction. When there is one margin line, the number of coordinate points in the area sandwiched between the margin line and the extension road surface is counted, and when there are two margin lines, the number of coordinate points in the area sandwiched between the two outermost lines is counted. Count the number of coordinate points. To supplement the count, the number of coordinate points whose frequency value is 1 or more may be counted, or if it is less than a predetermined value, it may be regarded as noise parallax and the coordinates having a frequency value equal to or higher than the predetermined value may be counted. good. Further, the road surface to be extended may be executed only for a closer segment or a predetermined distance near, or may be extended to a distant segment or a predetermined distance far away. That is, the length of the extended road surface can be set arbitrarily. Further, the range to which this process is applied may be limited to a predetermined segment or a predetermined distance (for example, it can be used to execute only to the second segment and the third segment). In addition to the above processing, different success / failure judgments such as success / failure judgment based on the angle and success / failure judgment based on the dispersion of the sample point group may be applied together as the success / failure judgment of the estimated road surface. An example of success / failure judgment by angle and an example of success / failure judgment by dispersion are as described above.

The processing flow by the road surface detection processing unit 122 of this embodiment is the same as the flowchart shown in FIG. 24, and the determination processing of step S6 is different from that of the first embodiment described above. More specifically, the rejection unit 173 sets an extension road surface that is an extension of the estimated road surface in the segment of interest, and counts how many coordinates the pixels with parallax have voted for on the extension road surface. The ratio (count number / length of extended road surface (segment length may be used)) is calculated. Then, when the calculated ratio is less than the threshold value, it is determined that the estimated road surface corresponding to the extended road surface (estimated road surface corresponding to the segment of interest) is rejected. On the other hand, if the calculated ratio is equal to or greater than the threshold value, it is determined that the estimated road surface corresponding to the extended road surface is not rejected.

As described above, in the present embodiment, the V map is divided into a plurality of segments, and the road surface is estimated for each segment. Then, an extension road surface obtained by extending the estimated road surface is set for each segment, and when the ratio of the total number of coordinates voted by the pixels having the parallax value d to the extension road surface on the V map is less than the threshold value, It is judged that the estimated road surface corresponding to the extended road surface is pulled up due to the influence of the object parallax, and the estimated road surface is rejected (the estimation is judged to be a failure). As a result, it is possible to prevent the object detection from being performed using an estimated road surface different from the actual road surface, and as a result, it is possible to sufficiently secure the detection accuracy of the object.

It is also possible to use the above-mentioned first embodiment and the second embodiment in combination. For example, the rejection unit 173 may be in a form capable of switching between the rejection determination of the first embodiment and the rejection determination of the second embodiment for each segment. For example, when the position of the estimated road surface in the interest segment is higher than the default road surface or the historical road surface by a predetermined value or more, the rejection unit 173 determines the rejection as described in the second embodiment as the rejection determination of the estimated road surface in the interest segment. If the height is less than a predetermined value, the rejection determination described in the first embodiment may be performed. Further, for example, when the absolute value of the slope of the extended road surface set by extending the estimated road surface corresponding to the segment of interest is equal to or greater than a predetermined value (in the case of steepness), the rejection unit 173 determines the estimated road surface corresponding to the segment of interest. As the rejection determination, the rejection determination described in the second embodiment may be performed, and if the absolute value of the slope is less than a predetermined value, the rejection determination described in the first embodiment may be performed.

Although the embodiments according to the present invention have been described above, the present invention is not limited to the above-described embodiments as they are, and at the implementation stage, the components can be modified and embodied within a range that does not deviate from the gist thereof. .. In addition, various inventions can be formed by an appropriate combination of the plurality of components disclosed in the above-described embodiment. For example, some components may be removed from all the components shown in the embodiments.

Further, the program executed by the mobile control system 100 of the above-described embodiment is a file in an installable format or an executable format, and is a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versaille Disk). ), USB (Universal Serial Bus), etc., may be configured to be recorded and provided on a computer-readable recording medium, or may be provided or distributed via a network such as the Internet. Further, various programs may be configured to be provided by incorporating them into a ROM or the like in advance.

1A 1st camera unit 1B 2nd camera unit 5 Lens 6 Image sensor 7 Sensor controller 10 Data bus line 11 Serial bus line 15 CPU
16 FPGA
17 ROM
18 RAM
19 Serial IF
20 data IF
100 Mobile control system 101 Vehicle 102 Imaging unit 103 Analysis unit 104 Control unit 105 Display unit 106 Windshield 111 Preprocessing unit 112 Parallelized image generation unit 113 Parallax image generation unit 114 Object detection processing unit 122 Road surface detection processing unit 123 Clustering processing Unit 124 Tracking processing unit 125 Acquisition unit 126 First generation unit 127 Road surface estimation unit 130 Second generation unit 140 Isolated area detection processing unit 150 Parallax image processing unit 160 Rejection processing unit 171 Dividing unit 172 Estimating unit 173 Rejection unit 174 Interpolation Department

Japanese Unexamined Patent Publication No. 2011-128844

Claims (13)

An acquisition unit that acquires a distance image that has distance information for each pixel,
A generation unit that generates correspondence information in which a position in the vertical direction and a position in the depth direction are associated with each other based on a plurality of pixels included in the distance image.
An estimation unit that estimates the shape of the reference object, which is the reference for the height of the object, for each of the plurality of segments in which the correspondence information is divided.
For each segment, the distance existing below the extension shape is based on an extension shape obtained by extending a shape based on the estimated shape indicating the shape of the reference object estimated by the estimation unit to the adjacent segment. It includes a rejection unit that rejects the estimated shape according to the distribution of information or the ratio of the total number of coordinates voted by the pixels having the distance information to the extension shape.
Information processing device. The shape based on the estimated shape is the estimated shape itself.
The information processing device according to claim 1. The shape based on the estimated shape is a margin line of the estimated shape.
The information processing device according to claim 1. When the frequency value of the distance information existing below the extension shape is equal to or greater than the threshold value, the rejection unit rejects the estimated shape.
The information processing device according to claim 1. The rejection unit generates a frequency histogram associated with a frequency value that counts the distance information existing below the predetermined shape for each position in the depth direction, and the corresponding frequency value is equal to or higher than the predetermined value. The number of bins indicating the number of positions in the depth direction is measured, and when the ratio of the number of bins to the length of the segment is equal to or greater than the threshold value, the estimated shape is rejected.
The information processing device according to claim 4. When the ratio of the total number of coordinates voted by the pixels having the distance information to the predetermined area below the extension shape in the corresponding information is equal to or more than the threshold value, the rejection unit obtains the estimated shape. Reject,
The information processing device according to claim 4. The rejection unit rejects the estimated shape when the ratio of the total number of coordinates voted by the pixels having the distance information to the extension shape is less than the threshold value.
The information processing device according to claim 1. When the estimated shape corresponding to the segment is rejected by the rejection unit, a setting unit for setting a predetermined shape as the shape of the reference object corresponding to the segment is further provided.
The information processing device according to any one of claims 1 to 7. The predetermined shape includes a default shape that is assumed to be a flat shape, or a historical shape that indicates a shape estimated in a past frame.
The information processing device according to claim 8. An imaging unit that captures stereo images,
A distance image generation unit that generates a distance image having distance information for each pixel from the stereo image captured by the imaging unit.
A generation unit that generates correspondence information in which a position in the vertical direction and a position in the depth direction are associated with each other based on a plurality of pixels included in the distance image.
An estimation unit that estimates the shape of the reference object, which is the reference for the height of the object, for each of the plurality of segments in which the correspondence information is divided.
For each segment, the distance existing below the extension shape is based on an extension shape obtained by extending a shape based on the estimated shape indicating the shape of the reference object estimated by the estimation unit to the adjacent segment. It includes a rejection unit that rejects the estimated shape according to the distribution of information or the ratio of the total number of coordinates voted by the pixels having the distance information to the extension shape.
Imaging device. A device control system including an image pickup device and a control unit that controls the device based on the output result of the image pickup device.
The image pickup device
An imaging unit that captures stereo images,
A distance image generation unit that generates a distance image having distance information for each pixel from the stereo image captured by the imaging unit.
A generation unit that generates correspondence information in which a position in the vertical direction and a position in the depth direction are associated with each other based on a plurality of pixels included in the distance image.
An estimation unit that estimates the shape of the reference object, which is the reference for the height of the object, for each of the plurality of segments in which the correspondence information is divided.
The distance existing below a predetermined shape for each of the segments, based on an extension shape obtained by extending a shape based on the estimated shape indicating the shape of the reference object estimated by the estimation unit to the adjacent segment. It includes a rejection unit that rejects the estimated shape according to the distribution of information or the ratio of the total number of coordinates voted by the pixels having the distance information to the extension shape.
Equipment control system. An acquisition step of acquiring a distance image having distance information for each pixel,
A generation step of generating correspondence information in which a vertical position and a depth position are associated with each other based on a plurality of pixels included in the distance image.
An estimation step for estimating the shape of a reference object, which is a reference for the height of the object, for each of a plurality of segments obtained by dividing the correspondence information.
The distance existing below a predetermined shape for each of the segments, based on an extension shape obtained by extending a shape based on the estimated shape indicating the shape of the reference object estimated by the estimation step to the adjacent segment. A rejection step of rejecting the estimated shape, depending on the distribution of information or the ratio of the total number of coordinates voted by the pixels having the distance information to the extension shape, is included.
Information processing method. On the computer
An acquisition step of acquiring a distance image having distance information for each pixel,
A generation step of voting a plurality of pixels included in the distance image to generate correspondence information in which a vertical position and a depth position are associated with each other.
An estimation step for estimating the shape of a reference object, which is a reference for the height of an object, for each of a plurality of segments obtained by dividing the correspondence information.
For each segment, the distance existing below the extension shape is based on an extension shape obtained by extending the shape based on the estimated shape indicating the shape of the reference object estimated by the estimation step to the adjacent segment. A program for executing a rejection step of rejecting the estimated shape according to the distribution of information or the ratio of the total number of coordinates voted by the pixels having the distance information to the extended shape.

Images