JP2018085059A - Information processing device, imaging apparatus, device control system, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device, an imaging apparatus, a device control system, an information processing method and a program that can sufficiently secure detection accuracy of objects.SOLUTION: An information processing device of the present invention comprises an acquisition part, a generation part, a division part, an estimation part, and a rejection part. The acquisition part acquires a distance image including distance information of each picture element. The generation part generates, based on a plurality of the picture elements included in the distance image, mapping information in which a position in the longitudinal direction and a position in the depth direction are mapped each other. The estimation part estimates, for each plurality of segments in which the mapping information is divided, the shape of a referential object representing a referential height of the object. The rejection part rejects, for each segment, using a predetermined referential shape set on the basis of the estimated shape indicating the shape of the referential object that is estimated by the estimation part, rejects the estimated shape in the event when the distribution of the distance information of the voted picture elements coincide with the predetermined referential shape.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an information processing device, an imaging device, a device control system, an information processing method, and a program.

  Conventionally, in terms of safety of automobiles, body structures of automobiles have been developed from the viewpoint of how to protect pedestrians and protect passengers when pedestrians and automobiles collide. However, in recent years, with the development of information processing technology and image processing technology, technology for detecting people and cars at high speed has been developed. Automobiles that apply these technologies to automatically apply a brake before an automobile collides with an object to prevent the collision have already been developed. For automatic vehicle control, it is necessary to accurately measure the distance to an object such as a person or other vehicle. For this purpose, distance measurement using millimeter wave radar and laser radar, distance measurement using a stereo camera, etc. are practical. It has become. For example, when ranging with a stereo camera, it is possible to generate a parallax image based on the amount of deviation (parallax) between local regions captured by the left and right cameras, and measure the distance between the front object and the vehicle. Then, clustering processing is performed to detect a group of parallax pixels existing at the same distance (having the same parallax value) as one object.

  Here, if all the parallax pixels (parallax points) are clustered, the white line parallax points on the road surface will be picked up separately from the object to be detected, and a part of the road surface that should be flat will be misrecognized. As a result, a problem occurs. In this case, the system determines that an object is present ahead, and causes a problem of sudden braking. In order to solve this problem, each parallax point (the x-coordinate value of the parallax image, the y-coordinate value of the parallax image, the parallax value d) is represented by the parallax value d on the horizontal axis, the x-coordinate value of the parallax image, and the depth. Information (V-DisparityMap) obtained by voting on a two-dimensional histogram with the direction axis as a frequency value is generated, and the road surface shape is calculated from a point group voted for this information using a statistical method such as the least square method. A technique is known that avoids detecting a road surface as a misrecognized object by performing estimation and clustering using only parallax points existing at a predetermined height or higher than the estimated road surface (for example, Patent Documents). 1).

  However, in the prior art, for example, in a scene where an object such as a vehicle exists in a situation where there is little parallax corresponding to the road surface, the parallax corresponding to the object is erroneously selected as the parallax corresponding to the road surface, and the estimated road surface is There is a problem that it is raised more than the actual. That is, in the prior art, since object detection is performed using an estimated road surface different from the actual road surface, there is a problem that it is difficult to ensure sufficient object detection accuracy.

  The present invention has been made in view of the above, and an object thereof is to provide an information processing apparatus, an imaging apparatus, a device control system, an information processing method, and a program capable of sufficiently ensuring the detection accuracy of an object.

  In order to solve the above-described problems and achieve the object, the present invention provides an acquisition unit that acquires a distance image having distance information for each pixel, and a vertical direction based on a plurality of pixels included in the distance image. A generation unit that generates correspondence information in which a position and a position in the depth direction are associated with each other, and estimation that estimates the shape of a reference object that is a reference for the height of the object for each of a plurality of segments obtained by dividing the correspondence information And the distribution of the distance information of the voted pixels is predetermined for each segment based on a predetermined shape set based on the estimated shape indicating the shape of the reference object estimated by the estimation unit. And a rejection unit that rejects the estimated shape when the criterion is met.

  According to the present invention, sufficient object detection accuracy can be ensured.

Drawing 1 is a mimetic diagram showing a schematic structure of a mobile control system of an embodiment. FIG. 2 is a schematic block diagram of the imaging unit and the analysis unit. FIG. 3 is a diagram illustrating a positional relationship between the subject and the imaging lens of each camera unit. FIG. 4 is a diagram for schematically explaining the functions of the analysis unit. FIG. 5 is a diagram illustrating an example of functions of the object detection processing unit. FIG. 6 is a diagram illustrating an example of functions of the road surface detection processing unit. FIG. 7 is a diagram illustrating an example of a parallax image. FIG. 8 is a diagram illustrating an example of a detailed configuration of the clustering processing unit. FIG. 9 is a diagram illustrating an example of a captured image. FIG. 10 is a diagram illustrating an example of the “Adult Large Umap”. FIG. 11 is a diagram illustrating an example of Large Umap. FIG. 12 is a diagram illustrating an example of a captured image. FIG. 13 is a diagram illustrating an example of an isolated region. FIG. 14 is a diagram illustrating a region on the parallax image corresponding to the isolated region illustrated in FIG. 13. FIG. 15 is a diagram showing a size range defined for each object type. FIG. 16 is a diagram for explaining the rejection process. FIG. 17 is a diagram illustrating an example of functions of the road surface estimation unit. FIG. 18 is a diagram illustrating an example of the V map generated by the first generation unit. FIG. 19 is a diagram illustrating an example of a plurality of segments obtained by the division by the division unit. FIG. 20 is a diagram for explaining a rejection determination according to the first embodiment. FIG. 21 is a diagram for explaining a method of measuring the frequency value of the parallax value existing below the extended road surface. FIG. 22 is a diagram for explaining another example of a method of measuring the frequency value of the parallax value existing below the extended road surface. FIG. 23 is a diagram for explaining a default road surface. FIG. 24 is a flowchart illustrating an example of processing by the road surface detection processing unit. FIG. 25 is a diagram for explaining a rejection determination according to the second embodiment.

  Hereinafter, embodiments of an information processing device, an imaging device, a device control system, an information processing method, and a program according to the present invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
Drawing 1 is a mimetic diagram showing a schematic structure of mobile control system 100 of an embodiment. As shown in FIG. 1, the mobile body control system 100 is provided in a vehicle 101 such as an automobile that is an example of a mobile body. The moving body control system 100 includes an imaging unit 102, an analysis unit 103, a control unit 104, and a display unit 105.

  The imaging unit 102 is provided in the vicinity of the rear mirror of the windshield 106 of the vehicle 101 and captures an image of the vehicle 101 such as a traveling direction. Various data including image data obtained by the imaging operation of the imaging unit 102 is supplied to the analysis unit 103. The analysis unit 103 analyzes recognition objects such as a road surface on which the vehicle 101 is traveling, a vehicle ahead of the vehicle 1, a pedestrian, and an obstacle based on various data supplied from the imaging unit 102. The control unit 104 issues a warning or the like to the driver of the vehicle 101 via the display unit 105 based on the analysis result of the analysis unit 103. Further, the control unit 104 performs traveling support such as control of various in-vehicle devices, steering wheel control or brake control of the vehicle 101 based on the analysis result.

  FIG. 2 is a schematic block diagram of the imaging unit 102 and the analysis unit 103. In this example, the analysis unit 103 functions as an “information processing apparatus”, and the set of the imaging unit 102 and the analysis unit 103 functions as an “imaging apparatus”. The control unit 104 described above functions as a “control unit” and controls the device (in this example, the vehicle 101) based on the output result of the imaging device. The imaging unit 102 is configured by assembling two camera units in parallel, a first camera unit 1A for the left eye and a second camera unit 1B for the right eye. That is, the imaging unit 102 is configured as a stereo camera that captures a stereo image. The stereo image is an image including a plurality of captured images (a plurality of captured images corresponding to a plurality of viewpoints on a one-to-one basis) obtained by imaging at a plurality of viewpoints, and the imaging unit 102 captures the stereo images. Device (functioning as an “imaging unit”). Each camera unit 1A and 1B includes a lens 5, an image sensor 6, and a sensor controller 7, respectively. The image sensor 6 is, for example, a CCD image sensor or a CMOS image sensor. CCD is an abbreviation for “Charge Coupled Device”. CMOS is an abbreviation for “Complementary Metal-Oxide Semiconductor”. The sensor controller 7 performs exposure control of the image sensor 6, image readout control, communication with an external circuit, transmission control of image data, and the like.

  The analysis unit 103 includes a data bus line 10, a serial bus line 11, a CPU 15, an FPGA 16, a ROM 17, a RAM 18, a serial IF 19, and a data IF 20. CPU is an abbreviation for “Central Processing Unit”. FPGA is an abbreviation for “Field-Programmable Gate Array”. ROM is an abbreviation for “Read Only Memory”. RAM is an abbreviation for “Random Access Memory”. IF is an abbreviation for “interface”.

  The above-described imaging unit 102 is connected to the analysis unit 103 via the data bus line 10 and the serial bus line 11. The CPU 15 controls the operation of the entire analysis unit 103, image processing, and image recognition processing. Luminance image data of captured images captured by the image sensors 6 of the first camera unit 1A and the second camera unit 1B is written to the RAM 18 of the analysis unit 103 via the data bus line 10. Sensor exposure value change control data, image read parameter change control data, various setting data, and the like from the CPU 15 or the FPGA 16 are transmitted and received via the serial bus line 11.

  The FPGA 16 is a process that requires real-time performance for image data stored in the RAM 18. The FPGA 16 uses one of the luminance image data (captured images) captured by the first camera unit 1A and the second camera unit 1B as a reference image and the other as a comparison image. Then, the FPGA 16 calculates a positional shift amount between the corresponding image portion on the reference image corresponding to the same point in the imaging region and the corresponding image portion on the comparison image as a parallax value (parallax image data) of the corresponding image portion. .

  FIG. 3 shows the positional relationship among the subject 30, the imaging lens 5A of the first camera unit 1A, and the imaging lens 5B of the second camera unit 1B on the XZ plane. In FIG. 3, the distance b between the imaging lenses 5A and 5B and the focal length f of the imaging lenses 5A and 5B are fixed values. Further, the amount of deviation of the X coordinate of the imaging lens 5A with respect to the gazing point P of the subject 30 is assumed to be Δ1. Further, the amount of deviation of the X coordinate of the imaging lens 5B with respect to the gazing point P of the subject 30 is assumed to be Δ2. In this case, the FPGA 16 calculates a parallax value d, which is a difference between the X coordinates of the imaging lenses 5 </ b> A and 5 </ b> B with respect to the gazing point P of the subject 30 using the following Equation 1.

  The FPGA 16 of the analysis unit 103 performs processing requiring real-time properties such as gamma correction processing and distortion correction processing (parallelization of the left and right captured images) on the luminance image data supplied from the imaging unit 102. Further, the FPGA 16 generates the parallax image data (parallax value D) by performing the calculation of the above-described Expression 1 using the luminance image data subjected to such processing that requires real-time property, and writes the parallax image data in the RAM 15. .

  Returning to FIG. 2, the description will be continued. The CPU 15 performs control of each sensor controller 7 of the imaging unit 102 and overall control of the analysis unit 103. The ROM 17 stores a three-dimensional object recognition program for executing situation recognition, prediction, three-dimensional object recognition, and the like, which will be described later. The three-dimensional object recognition program is an example of an image processing program. The CPU 15 acquires, for example, CAN information (vehicle speed, acceleration, steering angle, yaw rate, etc.) of the host vehicle as a parameter via the data IF 20. Then, the CPU 15 executes and controls various processes such as situation recognition using the luminance image and the parallax image stored in the RAM 18 in accordance with the three-dimensional object recognition program stored in the ROM 17. Recognize the recognition target. CAN is an abbreviation for “Controller Area Network”.

  The recognition data to be recognized is supplied to the control unit 104 via the serial IF 19. The control unit 104 performs traveling support such as brake control of the host vehicle and speed control of the host vehicle using the recognition data of the recognition target.

  FIG. 4 is a diagram for schematically explaining the functions of the analysis unit 103. A stereo image picked up by the image pickup unit 102 constituting the stereo camera is supplied to the analysis unit 103. For example, when the first camera unit 1A and the second camera unit 1B have color specifications, each of the first camera unit 1A and the second camera unit 1B performs RGB ( Color luminance conversion processing for generating a luminance (Y) signal from each signal (red, green, and blue) is performed. Each of the first camera unit 1A and the second camera unit 1B supplies luminance image data (captured image) generated by the color luminance conversion processing to the preprocessing unit 111 included in the analysis unit 103. It can be considered that a set of luminance image data (captured image) captured by the first camera unit 1A and luminance image data (captured image) captured by the second camera unit 1B is a stereo image. In this example, the preprocessing unit 111 is realized by the FPGA 16.

  The preprocessing unit 111 performs preprocessing of the luminance image data received from the first camera unit 1A and the second camera unit 1B. In this example, gamma correction processing is performed as preprocessing. Then, the preprocessing unit 111 supplies the luminance image data after the preprocessing to the parallelized image generation unit 112.

  The parallelized image generation unit 112 performs parallelization processing (distortion correction processing) on the luminance image data supplied from the preprocessing unit 111. This parallelization processing is an ideal parallel stereo image obtained when two pinhole cameras are attached in parallel to the luminance image data output from the first camera unit 1A and the second camera unit 1B. It is processing to convert to. Specifically, the first camera unit 1A, the first camera unit 1A, the second camera unit 1A, the second camera unit 1A, the second camera unit 1A, the second camera unit 1A, the second camera unit 1A, the second camera unit 1A, the second camera unit 1A, Each pixel of the luminance image data output from the second camera unit 1B is converted. The polynomial is based on, for example, a quintic polynomial relating to x (the horizontal position of the image) and y (the vertical position of the image). Thereby, the parallel brightness image which correct | amended distortion of the optical system of the 1st camera part 1A and the 2nd camera part 1B can be obtained. In this example, the parallelized image generation unit 112 is realized by the FPGA 16.

  The parallax image generation unit 113 is an example of a “distance image generation unit”, and a parallax value for each pixel, which is an example of a distance image having distance information for each pixel, from a stereo image captured by the imaging unit 102. The provided parallax image is generated. Here, the parallax image generation unit 113 uses the luminance image data of the first camera unit 1A as reference image data, the luminance image data of the second camera unit 1B as comparison image data, and performs the calculation shown in Equation 1 above. As a result, parallax image data indicating the parallax between the reference image data and the comparison image data is generated. Specifically, the parallax image generation unit 113 defines a block including a plurality of pixels (for example, 16 pixels × 1 pixel) centered on one target pixel for a predetermined “row” of the reference image data. On the other hand, in the same “row” in the comparison image data, a block having the same size as the defined reference image data block is shifted by one pixel in the horizontal line direction (X direction). Then, the parallax image generation unit 113 calculates a correlation value indicating a correlation between a feature amount indicating the feature of the pixel value of the block defined in the reference image data and a feature amount indicating the feature of the pixel value of each block in the comparison image data, Calculate each.

  Further, the parallax image generation unit 113 performs a matching process of selecting the block of the comparison image data that is most correlated with the block of the reference image data among the blocks of the comparison image data based on the calculated correlation value. Thereafter, a positional deviation amount between the target pixel of the block of the reference image data and the corresponding pixel of the block of the comparison image data selected by the matching process is calculated as the parallax value D. By performing such a process of calculating the parallax value D for the entire area of the reference image data or a specific area, parallax image data is obtained. As a method for generating a parallax image, various known techniques can be used. In short, it can be considered that the parallax image generation unit 113 calculates (generates) a distance image (in this example, a parallax image) having distance information for each pixel from a stereo image captured by a stereo camera.

  As the feature amount of the block used for the matching process, for example, the value (luminance value) of each pixel in the block can be used. Further, as the correlation value, for example, the difference between the value (luminance value) of each pixel in the block of reference image data and the value (luminance value) of each pixel in the block of comparison image data corresponding to each of these pixels The sum of absolute values of can be used. In this case, the block with the smallest sum is detected as the most correlated block.

  As such matching processing of the parallax image generation unit 113, for example, SSD (Sum of Squared Difference), ZSSD (Zero-mean Sum of Squared Difference), SAD (Sum of Absolute Difference), or ZSAD (Zero-mean Sum) of Absolute Difference) can be used. In the matching process, when a sub-pixel level disparity value of less than one pixel is required, an estimated value is used. As an estimation method of the estimated value, for example, an equiangular straight line method or a quadratic curve method can be used. However, an error occurs in the estimated sub-pixel level parallax value. For this reason, a technique such as EEC (estimation error correction) for reducing the estimation error may be used.

  In this example, the parallax image generation unit 113 is realized by the FPGA 16. The parallax image generated by the parallax image generation unit 113 is supplied to the object detection processing unit 114. In this example, the function of the object detection processing unit 114 is realized by the CPU 15 executing a three-dimensional object recognition program.

  FIG. 5 is a diagram illustrating an example of functions of the object detection processing unit 114. As illustrated in FIG. 5, the object detection processing unit 114 includes a road surface detection processing unit 122, a clustering processing unit 123, and a tracking processing unit 124.

  The road surface detection processing unit 122 uses the parallax image input from the parallax image generation unit 113 to detect a road surface that is an example of a reference object that serves as a reference for the height of the object. As illustrated in FIG. 6, the road surface detection processing unit 122 includes an acquisition unit 125, a first generation unit 126, and a road surface estimation unit 127. The acquisition unit 125 acquires a parallax image that is an example of a distance image having distance information for each pixel. The parallax image acquired by the acquisition unit 125 is input to the first generation unit 126 and a clustering processing unit 123 described later.

  The first generation unit 126 is an example of a “generation unit”, and based on a plurality of pixels included in the parallax image, the vertical direction of the parallax image and the depth direction indicating the direction of the optical axis of the stereo camera. Correspondence information in which the position is associated is generated. In this example, the first generation unit 126 votes each pixel of the parallax image on a two-dimensional histogram with the vertical coordinate (y) of the image as the vertical axis and the parallax value d as the horizontal axis. The correspondence information of is generated. In the following description, this correspondence information is referred to as a “V map (V-Disparity map)”. The V map has a parallax value d on the horizontal axis (x axis), a y coordinate value on the vertical axis (y axis), and a depth direction in a set of (x coordinate value, y coordinate value, parallax value d) of parallax images. It is a two-dimensional histogram with the axis (z-axis) as the frequency. In short, the V map can be considered as information in which the frequency value of the parallax value d is recorded for each combination of the position in the vertical direction and the parallax value d (corresponding to the position in the depth direction). In the following description, among the coordinate points in the V map, the coordinates at which a pixel (parallax pixel) having a parallax value d included in the parallax image is voted may be referred to as a parallax point. In the generation of the V map, the y coordinate of the parallax image and the y coordinate of the V map are in a correspondence relationship, and the parallax value d on the horizontal line of the specific y coordinate of the parallax image is the corresponding y coordinate of the V map. Among the horizontal lines, the points corresponding to the parallax value d (coordinate points on the V map) are voted. Accordingly, there are some disparity values d included in the same horizontal line of the disparity image, and therefore a frequency value indicating the number of disparity values d of the same value is stored at any coordinate point of the V map. become. In a specific horizontal line of the parallax image, the parallax values d are similar to each other on the same road surface, so that the parallax pixels corresponding to the road surface in the V map are densely voted.

  The first generation unit 126 may vote for all the parallax pixels in the parallax image, but a predetermined area (for example, the voting area 701 shown in FIG. 7) as in the parallax image Ip shown in FIG. 7. ˜703) may be set, and only the parallax pixels included in the area may be voted for. For example, a method of setting a predetermined number of voting areas corresponding to the width of the road surface as shown in FIG. By restricting the voting area in this way, it is possible to suppress noise other than the road surface from being mixed into the V map. Further, the voting may be performed by appropriately thinning out parallax pixels in one horizontal line in the parallax image. Further, the thinning may be performed not only in the horizontal direction but also in the vertical direction.

  The V map generated by the first generation unit 126 is input to the road surface estimation unit 127 shown in FIG. The road surface estimation unit 127 selects a sample point from the voted parallax points in the V map by a predetermined method, and estimates the shape of the road surface by linearly approximating the selected point group (or curve approximation). Here, the reference object serving as a reference for the height of the object corresponds to the road surface. Specific contents of the road surface estimation unit 127 will be described later. The estimation result (road surface estimation information) by the road surface estimation unit 127 is input to the clustering processing unit 123.

  The clustering processing unit 123 detects the object position on the parallax image acquired by the acquisition unit 125 using the road surface estimation information. FIG. 8 is a diagram illustrating an example of a detailed configuration of the clustering processing unit 123. As illustrated in FIG. 8, the clustering processing unit 123 includes a second generation unit 130, an isolated region detection processing unit 140, a parallax image processing unit 150, and a rejection processing unit 150. The second generation unit 130 uses a plurality of pixels present in a range higher than the road surface (an example of the reference object) in the parallax image, and a horizontal position indicating a direction orthogonal to the optical axis of the stereo camera. Second correspondence information in which the position in the depth direction indicating the direction of the optical axis of the stereo camera is associated is generated. In this example, the second correspondence information is such that the horizontal axis (X axis) is the actual distance (actual distance) in the horizontal direction, the vertical axis (Y axis) is the parallax value d of the parallax image, and the axis in the depth direction (Z axis) ) As a frequency. The second correspondence information can be considered to be information in which the frequency value of the parallax is recorded for each combination of the actual distance and the parallax value d.

  Here, since the linear equation representing the road surface is obtained by the road surface estimation of the road surface estimation unit 127 described above, if the parallax d is determined, the corresponding y coordinate y0 is determined, and this coordinate y0 becomes the height of the road surface. For example, when the parallax value is d and the y coordinate is y ′, the height from the road surface when y′−y0 is the parallax value d is indicated. The height H of the coordinates (d, y ′) from the road surface can be obtained by an arithmetic expression H = (z × (y′−y0)) / f. In this arithmetic expression, “z” is a distance (z = Bf / (d−offset)) calculated from the parallax value d, and “f” is a focal length of the imaging unit 102 (y′−y0). Is the value converted to the same unit. Here, BF is a value obtained by multiplying the base line length B of the imaging unit 102 and the focal length f, and offset is a parallax when an object at infinity is photographed.

  The second generation unit 130 generates at least one of “Adult Large Umap”, “Large Umap”, and “Small Umap” as the second correspondence information. Hereinafter, these maps will be described. First, “Adult Large Umap” will be described. If the position in the horizontal direction of the parallax image is x, the position in the vertical direction is y, and the parallax value set for each pixel is d, the second generation unit 130 includes the first higher parallax image than the road surface. By voting a point (x, y, d) that exists in a second range indicating a range of heights greater than or equal to a predetermined value within the range based on the value of (x, d), the horizontal axis is parallax. A two-dimensional histogram is generated with the x of the image, the vertical axis representing the parallax value d, and the depth axis representing the frequency. Then, the horizontal axis of this two-dimensional histogram is converted into an actual distance to generate an “Adult Large Umap”.

  For example, in the captured image shown in FIG. 9, a person group 1 including adults and children, a person group 2 between adults, a pole, and a vehicle are reflected. In this example, the range from 150 cm to 200 cm in actual height from the road surface is set as the second range, and the Large Large Umap in which the parallax value d of the second range is voted is as shown in FIG. The parallax value d of a child whose height is less than 150 cm is not voted and therefore does not appear on the map. The vertical axis represents the thinning parallax obtained by thinning the parallax value d using the thinning rate according to the distance. The Result Large Umap generated by the second generation unit 130 is input to the isolated region detection processing unit 140.

  Next, “Large Umap” will be described. When the horizontal position of the parallax image is x, the vertical position is y, and the parallax value set for each pixel is d, the second generation unit 130 exists in the first range of the parallax images. By voting the point (x, y, d) based on the value of (x, d), the horizontal axis is x of the parallax image, the vertical axis is the parallax value d, and the axis in the depth direction is two-dimensional. Generate a histogram. Then, the horizontal axis of the two-dimensional histogram is converted into an actual distance to generate a Large Umap. In the example of FIG. 9, a range from 0 cm to 200 cm (including the above-described second range) is set as the first range, and the Large Umap in which the parallax value d of the first range is voted is shown in FIG. become that way. In addition to the Large Umap, the second generation unit 130 has the highest height (h) from the road surface among the parallax points (a set of the real distance and the parallax value d) voted for the Large Umap. The height of the difference point is recorded, the horizontal axis is the actual distance (the distance in the left-right direction of the camera), the vertical axis is the parallax value d, and the height information in which the height is recorded for each corresponding point can be generated. it can. You may think that height information is the information which recorded height for every combination of real distance and the parallax value d. In the following description, this height information is referred to as a “large Umap height map”. The position of each pixel included in the “Large Umap height map” corresponds to the position of each pixel included in the Large Umap. The Large Umap and the Large Umap height map generated by the second generation unit 130 are input to the isolated region detection processing unit 140.

  Next, “Small Umap” will be described. When the horizontal position of the parallax image is x, the vertical position is y, and the parallax value set for each pixel is d, the second generation unit 130 exists in the first range of the parallax images. By voting a point (x, y, d) based on the value of (x, d) (voting a smaller number than when creating a Large Umap), the horizontal axis is x of the parallax image, and the vertical axis is A two-dimensional histogram is generated with the parallax value d and the axis in the depth direction as the frequency. Then, the horizontal axis of the two-dimensional histogram is converted into an actual distance to generate a Small Umap. The Small Umap has a lower distance resolution of one pixel than the Large Umap. In addition to the Small Umap, the second generation unit 130 has the highest height (h) from the road surface among the parallax points voted by the Small Umap (a set of the actual distance and the parallax value d). The height of the difference point is recorded, the horizontal axis is the actual distance (the distance in the left-right direction of the camera), the vertical axis is the parallax value d, and the height information in which the height is recorded for each corresponding point can be generated. it can. You may think that height information is the information which recorded height for every combination of real distance and the parallax value d. In the following description, this height information is referred to as “Small U map height map”. The position of each pixel included in the “Small Umap height map” corresponds to the position of each pixel included in the Small Umap. The height map of the Small Umap and Small U map generated by the second generation unit 130 is input to the isolated region detection processing unit 140.

  In this example, the second generation unit 130 generates a Large Umap and the case where the generated Large Umap is input to the isolated region detection processing unit 140 will be described as an example. In the case of performing object detection using “Adult Large Umap”, “Large Umap”, and “Small Umap”, the second generating unit 130 generates “Adult Large Umap”, “Large Umap”, and “Small Umap”. These maps may be input to the isolated region detection processing unit 140.

  Returning to FIG. The isolated area detection processing unit 140 detects an isolated area (aggregate area) that is a lump area of the parallax value d from the above-described second correspondence information (in this example, Large Umap). For example, in the case of the captured image shown in FIG. 12, there are guard rails 81 and 82 on the left and right, and the vehicles 77 and 79 are facing each other across the center line. One vehicle 77 or vehicle 79 is traveling in each traveling lane. Two poles 80 </ b> A and 80 </ b> B exist between the vehicle 79 and the guardrail 82. FIG. 13 is a Large Umap obtained based on the captured image shown in FIG. 12, and an area surrounded by a frame corresponds to an isolated area.

  The parallax image processing unit 150 illustrated in FIG. 8 performs parallax image processing for detecting an area on a parallax image corresponding to the isolated area detected by the isolated area detection processing unit 140 or object information in real space. FIG. 14 is a diagram showing a region on the parallax image corresponding to the isolated region shown in FIG. 13 (result of processing by the parallax image processing unit 150), and a region 91 in FIG. 14 is a region corresponding to the guardrail 81, The region 92 is a region corresponding to the vehicle 77, the region 93 is a region corresponding to the vehicle 79, the region 94 is a region corresponding to the pole 80A, and the region 95 is a region corresponding to the pole 80B. Is an area corresponding to the guardrail 82.

  The rejection processing unit 160 illustrated in FIG. 8 performs a rejection process for selecting an object to be output based on a region on the parallax image detected by the parallax image processing unit 150 or object information in real space. The rejection processing unit 160 executes size rejection focusing on the object size and overlap rejection focusing on the positional relationship between the objects. For example, in the size rejection, a detection result of a size that does not fall within the size range defined for each object type shown in FIG. 15 is rejected. For example, in the example of FIG. 16, the area 91 and the area 96 are rejected. In the overlap rejection, a result having an overlap is selected for areas corresponding to isolated areas on the parallax image detected by the parallax image processing.

  The output information (detection result) from the clustering processing unit 123 is input to the tracking processing unit 124 shown in FIG. The tracking processing unit 124 determines that the detection result (detected object) by the clustering processing unit 123 appears as a tracking target when it continuously appears across a plurality of frames. Is output to the control unit 104 as an object detection result. The control unit 104 actually controls the vehicle 101 based on the object detection result.

  Below, the specific content of the road surface estimation part 127 (FIG. 6) which estimates the shape of the road surface which is an example of a reference | standard object is demonstrated. FIG. 17 is a diagram illustrating an example of functions of the road surface estimation unit 127. As illustrated in FIG. 17, the road surface estimation unit 127 includes a division unit 171, an estimation unit 172, a rejection unit 173, and an interpolation unit 174.

  The dividing unit 171 divides the V map (corresponding information) input from the first generating unit 126 into a plurality of segments. In this example, the dividing unit 171 divides the V map into a plurality of segments that are continuous in the depth direction (the direction of the parallax value d, the direction of the horizontal axis of the V map). However, the present invention is not limited to this. For example, the parallax image may be divided in the y direction (vertical direction of the V map). The segment position can be set to an arbitrary position. Usually, it is desirable that the segments are continuous, but they may be discontinuous (for example, when estimation in a predetermined distance range (d value) is not intentionally executed). In this embodiment, two or more segments are set. The segments can be set with a predetermined width without being set at equal intervals. For example, since it is known that the distant area has a low resolution (road surface resolution is low), a method of finely dividing the segment as it goes far can be considered. Therefore, the number of segments may be determined according to the above.

  For example, it is assumed that the first generation unit 126 generates the V map shown in FIG. 18B from the parallax image Ip2 shown in FIG. 18A and this V map is input to the dividing unit 171. To do. In the parallax image Ip2 shown in FIG. 18A, the road surface 600 and the light truck 601 are reflected. The light truck 600 in the parallax image Ip2 corresponds to the voting point group (parallax point group) indicated by 603 in the V map shown in FIG. The dividing unit 171 divides the V map shown in FIG. 18B into a plurality of segments divided by a predetermined d coordinate range. FIG. 19 is a diagram illustrating an example of a plurality of segments obtained by the division by the dividing unit 171. In order from the right, the first segment seg1, the second segment seg2, the third segment seg3, the fourth segment seg4, They are referred to as 5 segment seg5, 6th segment seg6, and 7th segment seg7.

  The description of FIG. 17 is continued. The estimation unit 172 estimates the shape of the road surface (an example of a reference object) for each of a plurality of segments obtained by dividing the correspondence information. More specifically, the estimation unit 172 performs the following process for each segment. First, the estimation unit 172 calculates the coordinates (hereinafter may be referred to as “d-coordinates”) in the direction (depth direction) of the parallax value d in the segment to be processed (hereinafter may be referred to as “target segment”). A predetermined number (for example, one point) of representative points (hereinafter referred to as “sample points”) is selected from the positions. As a sample point selection method, for example, a parallax point (most frequent point) having the highest frequency may be simply selected from among the parallax points existing in the vertical (vertical) direction for each d coordinate. Alternatively, the d coordinate of interest and a plurality of pixels on the left and right sides thereof are raised together from the lower direction to the upper direction of the V map, and the region where the parallax points on the road surface can be included is limited. A method of capturing a parallax point on the road surface more accurately, such as selecting a point, may be used. Alternatively, a position (coordinate) where there is no parallax point may be selected as a sample point. For example, when there are no parallax points at the coordinate (d, y) of interest, but parallax points with high frequency are concentrated around, the parallax point at the coordinates (d, y) is accidentally changed. Since there is a possibility of missing, it is also possible to select this missing position as a sample point.

  Further, the estimation unit 172 may remove inappropriate sample points from the sample points selected as described above. Accordingly, it is possible to suppress the estimation result of the road surface from being inappropriate due to the influence of an inappropriate sample point (outlier point) when performing linear approximation on a sample point group described later. As a method for removing outliers, for example, linear approximation may be performed by a least-squares method using all sample points in the target segment, and sample points that are separated from the approximate line by a predetermined distance may be removed. In this case, the final estimation result is the result of estimation by the least square method again with the outliers removed.

  The estimation unit 172 estimates the shape of the road surface using the remaining sample points. As a method of estimating the shape of the road surface, for example, there are a method of performing linear approximation on a sample point group by the least square method or the like, a method of estimating a curve shape by using polynomial approximation or the like. At the same time, a numerical scale such as a correlation coefficient based on these methods may be calculated for use in subsequent success / failure determination (success / failure determination for the result of estimating the road surface shape). In the following description, the road surface shape estimation is assumed to be based on linear approximation unless otherwise specified. In addition, the estimation result of the shape of the road surface may be referred to as an estimated road surface.

  Here, for example, a case is assumed where the parallax pixels in the trapezoidal region 711 in the parallax image Ip2 in FIG. 18A are voting targets at the time of V map generation. Of the two regions 712 and 713 included in the region 711, the region 712 represents a region where a road surface parallax pixel (a pixel having a parallax value) exists, and the region 713 represents a region where a road surface parallax pixel does not exist. It shall represent. Therefore, assuming that the parallax pixel existing at the y coordinate of the parallax image Ip2 is voted to the corresponding coordinate (d, y) on the V map in the correspondence relationship shown in FIG. 18, the parallax corresponding to the road surface in the region 712 The pixel is voted as a road surface, but the region 713 is not voted because there is no parallax pixel corresponding to the road surface. Further, since the light truck 601 has a different distance from the camera in the loading platform portion 612 and the cabin portion 611, the corresponding parallax pixels are voted for different segments on the V map. Further, since the light truck 601 has a loading platform cover, the distance gradually changes from the bottom to the top of the image, so that it is similar to the parallax distribution on the road surface on the V map. For this reason, the estimated road surface is affected by the parallax (object parallax) corresponding to the light truck, and is estimated at a position higher than the correct answer road surface (actual road surface).

  Therefore, in this embodiment, for each segment, the estimated road surface in the segment is extended to set an extended road surface, and when the frequency value of the parallax value d existing below the extended road surface is greater than a certain value, It is determined that the estimated road surface corresponding to the extended road surface is pulled up under the influence of the object parallax, and the corresponding estimated road surface is rejected (estimation is determined to be unsuccessful). Specific contents will be described below.

  The rejection unit 173 shown in FIG. 17 has, for each segment, the parallax of the voted pixels based on a predetermined shape set based on the estimated road surface (estimated shape indicating the shape of the road surface) estimated by the estimation unit 172. When the distribution of the value d (more specifically, the distribution of the voting points on the V map (the distribution of the frequency values of the parallax value d)) matches a predetermined criterion, the estimated road surface is rejected. The predetermined shape includes a shape obtained by extending a shape based on the estimated road surface to an adjacent segment. In this example, a shape obtained by extending the estimated road surface as it is to an adjacent segment (hereinafter referred to as “extended road surface”) is the predetermined shape. However, the present invention is not limited to this, and the predetermined shape is a margin described later. It may be a line. The rejection unit 173 determines whether to reject the estimated road surface according to the distribution of the frequency values of the parallax values d existing below the extended road surface. More specifically, the rejection unit 173 rejects the estimated road surface when the frequency value of the parallax value d existing below the extended road surface is equal to or greater than a threshold value.

  Usually, when the correct road surface can be estimated like the estimated road surface B in the seventh segment seg7 in FIG. 20, the disparity value existing below the extended road surface B obtained by extending the estimated road surface B to the adjacent sixth segment seg6 The frequency value of d is small (or does not appear). On the other hand, when the road surface has been pulled up due to the object parallax as in the estimated road surface A in the third segment seg3, the object is placed below the extended road surface A that extends to the adjacent second segment seg2. There is parallax (parallax corresponding to the loading platform 612). Therefore, the rejection unit 173 measures the frequency value of the parallax value d existing below the extended road surface A, and rejects the estimated road surface A when the frequency value of the parallax value d is greater than or equal to the threshold value. Such processing is executed for each segment. In addition to the above-described processing, different success / failure determinations such as success / failure determination by angle and success / failure determination by dispersion of sample point groups may be applied in addition to the above processing. In addition, as an example of the success / failure determination based on the angle, there is an aspect in which the estimated road surface is rejected when the actual angle of the estimated road surface exceeds a predetermined value. Further, as an example of success / failure determination based on the dispersion of sample point groups, there is a mode in which the estimated road surface is rejected on the assumption that the road surface estimated from the scattered point group has an unreliable shape.

  Actually, since the road surface parallax may be dispersed (for example, the farther away, the parallax accuracy becomes worse), a predetermined margin line is provided like the margin lines A and B shown in FIG. The lower range may be the measurement range. In short, the predetermined shape may include a shape obtained by extending a shape based on the estimated road surface to an adjacent segment. The “shape based on the estimated road surface” may be the estimated road surface itself or a margin line. As for the extended road surface, a nearer segment than a target segment may be used, or a far segment may be used. Of course, you may use the road surface extended to both. Further, the length to be extended can be set to a predetermined length. For example, it may be extended by one segment or may be extended further. Moreover, it is not limited to the segment unit, and it may be extended by a predetermined distance. Moreover, you may limit the range which applies this process. For example, considering the risk that if the road surface estimation fails in the vicinity and the estimated road surface is pulled up, the object in front will be unrecognized, only the segments closer to the predetermined segment This process may be executed. Of course, since it may be performed only for a distant segment, the range to be applied is arbitrary.

  Next, a method for measuring the frequency value of the parallax value d existing below the extended road surface will be described. In the present embodiment, the rejection unit 173 performs the frequency value of the parallax value d existing below the extended road surface for each position in the depth direction (for each position in the direction of the parallax value d (horizontal axis direction) of the V map). A frequency histogram is generated in which the frequency values counted are associated with each other. Then, referring to the frequency histogram, the number of bins indicating the number of positions in the depth direction where the corresponding frequency value is equal to or greater than a predetermined value is measured, and when the ratio of the number of bins to the segment length is equal to or greater than a threshold value, the extended road surface The estimated road surface corresponding to is rejected. Since the frequency value of the parallax value d is stored in each coordinate of the V map, when creating the frequency histogram, the total value of the frequency values may be used, or the frequency value becomes a predetermined value or more. The number of coordinates may be counted. Then, the number of frequency histogram coordinates (number of coordinates in the direction of the parallax value d) associated with the frequency value equal to or higher than the bin determination threshold is counted, and the number of bins equal to or higher than the bin determination threshold in the length of the segment is counted. The ratio (bin ratio) is calculated. When the bin ratio is equal to or greater than a threshold (referred to as “ratio threshold”), it is determined that the object parallax exists below the extended road surface (or margin line).

  For example, in the example of FIG. 21, the frequency histogram A corresponding to the second segment seg2 has 2 bins equal to or greater than the bin determination threshold. For example, if the ratio threshold is 50%, the segment length (corresponds to three bins). ), The ratio of the number of bins equal to or greater than the bin determination threshold is equal to or greater than the ratio threshold. Therefore, it is determined that object parallax exists below the extended road surface A, and the estimated road surface A corresponding to the extended road surface A is rejected. On the other hand, the frequency histogram B corresponding to the sixth segment seg6 has no bin exceeding the bin determination threshold, and therefore the bin ratio is less than the ratio threshold, and it is determined that there is no object parallax below the extended road surface B. However, the estimated road surface B corresponding to the extended road surface B is not rejected. Here, the reason for using the bin determination threshold is to make it robust against noise. If the number of bins having a frequency of 1 or more is counted without providing a bin determination threshold, the bin ratio is likely to exceed the ratio threshold in a scene such as rainy weather with a lot of noise. Of course, this bin determination threshold value may not be provided (bin determination threshold value = 0 may be satisfied).

  Further, as another method, the rejection unit 173 occupies a predetermined number (1 may be used) or a number larger than 1 as a noise countermeasure in a predetermined area below the extended road surface (or margin line) in the V map. The estimated road surface corresponding to the extended road surface may be rejected when the ratio of the total number of coordinates (coordinates having a frequency value greater than or equal to a predetermined number) on which the above parallax pixels are voted is equal to or greater than a threshold value. . Although the shape of the predetermined area is arbitrary, for example, as shown in FIG. 22, a trapezoidal shape may be used in order to appropriately capture an area below the margin line (or an extended road surface), but is not limited thereto. For example, it may be a rectangle. In the example of FIG. 22, since more than half of the coordinates where a predetermined number or more of parallax pixels have been voted exist in the measurement region A, for example, if the threshold is 50%, the rejection unit 173 is more than the margin line A. It is determined that there is an object parallax below, and the estimated road surface A corresponding to the margin line A is rejected. On the other hand, since there are no coordinates in which the predetermined number or more of the parallax pixels are voted in the measurement area B, the rejection unit 173 determines that there is no object parallax below the margin line B, and the margin line B The corresponding estimated road surface B is not rejected.

  Returning to FIG. 17, the description will be continued. The interpolation unit 174 is an example of a “setting unit”. When the estimated road surface corresponding to the segment is rejected by the rejection unit 173, a predetermined road surface (predetermined shape) is set as the shape of the road surface corresponding to the segment. (Interpolate). As an example of the predetermined road surface, there is a default road surface (default shape) assumed to be a flat shape, a history road surface (history shape) indicating a shape estimated in a past frame, or the like. FIG. 23A shows the parallax image Ip3 when the vehicle is traveling on a flat road surface, and FIG. 23B shows a V map generated from the parallax image Ip3. As shown in FIG. 23B, when the vehicle is traveling on a flat road surface, the estimated road surface ER1 that is estimated substantially matches the default road surface DR that is a road surface assumed to be a flat road surface. The default road surface DR can be calculated in advance from the camera mounting height and the pitching angle. The history road surface indicates a past estimated road surface estimated in a frame one frame or more before, and may be a road surface obtained by averaging road surfaces estimated in a past predetermined number of frames.

  FIG. 24 is a flowchart illustrating an example of processing by the road surface detection processing unit 122 of the present embodiment. Since the specific contents of each step are as described above, detailed description will be omitted as appropriate. First, the acquisition unit 125 acquires a parallax image (step S1). The acquisition unit 125 may directly acquire the parallax image generated by the parallax image generation unit 113, or store the parallax image in advance in a recording medium such as a CD, DVD, HDD, or a network storage, and when necessary These may be read and used. Further, only one parallax image may be acquired, or moving image data may be sequentially acquired for each frame. A method of building a V map in advance and inputting it to the road surface detection processing unit 122 is also possible. In this case, step S1 and the next step S2 are skipped, and the process starts from step S3.

  Next, the 1st production | generation part 126 produces | generates V map using the parallax image acquired by step S1 (step S2). The specific contents are as described above.

  Next, the road surface estimation unit 127 (dividing unit 171) divides the V map generated in step S2 into a plurality of segments (step S3). The specific contents are as described above.

  The following steps S4 to S7 are repeated for the number of segments. Here, after the processing of step S4 to step S7 is completed for one segment, the processing of step S4 to step S7 is executed for the next segment, but the present invention is not limited to such a configuration. For example, after executing the processing of each step for all segments, the mode may be shifted to the next step. For example, after the process of step S4 is executed for all segments, the process proceeds to step S5.

  In step S4, the road surface estimation unit 127 (estimation unit 172) performs a sample point search for each d coordinate (step S4). At this time, the number of sample points is not limited to one, and a plurality of points may be determined. In addition, since there is a d coordinate where the parallax does not exist in the vertical direction, there may exist a d coordinate where the sample point is not determined. The specific contents are as described above.

  In step S5, the road surface estimation unit 127 (estimation unit 172) estimates the shape of the road surface (step S5). The specific contents are as described above.

  In step S6, the road surface estimation unit 127 (rejection unit 173) sets an extended road surface obtained by extending the estimated road surface, and determines whether or not the frequency value of the parallax value d existing below the extended road surface is equal to or greater than a threshold value. . That is, the road surface estimation unit 127 (rejection unit 173) determines whether or not to reject the estimated road surface (step S6). The specific contents are as described above.

  When the result of step S6 is negative (step S6: No), the estimated road surface is adopted as it is. On the other hand, when the result of step S6 is affirmative (step S6: Yes), the estimated road surface is rejected, and the road surface estimation unit 127 (interpolation unit 174) uses the predetermined road surface (for example, default) as the road surface corresponding to the segment. A road surface, a history road surface, etc.) are set (interpolated) as a new estimated road surface (step S7).

  Note that the present invention is not limited to the above, and for example, a process for removing the above-described outliers may be inserted between step S4 and step S5. Moreover, after the process of step S4-step S7 is completed about all the segments, the form which performs the smoothing process which corrects so that the estimated road surface between segments may continue smoothly may be sufficient. As an example of the smoothing process, for example, of the estimated road surfaces of two segments, the d coordinate corresponding to the start point of one estimated road surface and the d coordinate corresponding to the end point of the other estimated road surface (if there is no break between segments, the end point (The start point indicates the same d coordinate) and pass through a predetermined y coordinate position (correction is equivalent to changing the slope and intercept in the V map of the estimated road surface) Also good. This smoothing process ensures the continuity of the estimated road surface among all segments. As the predetermined y-coordinate position, for example, a method of using the y-coordinate of the middle point between the y-coordinate corresponding to the start point and the y-coordinate corresponding to the end point can be considered. By performing the smoothing process, there is a possibility that the estimated road surface in a certain segment is not suitable, so that there is an effect of improving the accuracy of the road surface estimation. The smoothed estimated road surface is the final result. In addition, this smoothing process performs a smoothing process between the estimated road surface corresponding to the one segment and the estimated road surface corresponding to the previous segment every time the processes of steps S4 to S7 are completed for the one segment. The form to perform may be sufficient. In addition, the form which does not perform the process which removes a detached point, or a smoothing process may be sufficient.

  As described above, in this embodiment, the V map is divided into a plurality of segments, and the road surface is estimated for each segment. Then, for each segment, an extended road surface obtained by extending the estimated road surface is set, and when the frequency value of the parallax value d existing below the extended road surface is greater than a certain value, the estimated road surface corresponding to the extended road surface is the object parallax. The estimated road surface is rejected (estimated is judged to have failed). As a result, it is possible to prevent object detection from being performed using an estimated road surface that is different from the actual road surface, and as a result, sufficient object detection accuracy can be ensured.

(Second Embodiment)
Next, a second embodiment will be described. Description of parts common to the above-described first embodiment will be omitted as appropriate. Although the basic configuration is the same as that of the first embodiment described above, in this embodiment, the rejection unit 173 has pixels (parallax) having a parallax value d occupying the predetermined shape (extended road surface or margin line). If the ratio of the total number of coordinates from which the image pixels are voted is less than the threshold, the estimated road surface is rejected.

  Here, for example, a case is assumed where the parallax pixels in the trapezoidal region 811 in the parallax image Ip4 in FIG. 25A are to be voted at the time of V map generation. Of the three areas 812, 813, and 814 included in the area 811, the area 812 and the area 814 represent areas where parallax pixels exist, and the area 813 represents an area where no parallax pixels exist. When there is a small amount of road surface parallax or when there is an upright object such as a large truck 820, the estimated road surface may be pulled up by the object parallax. For example, when a large track 820 exists as shown in FIG. 25, the parallax corresponding to the track 820 is distributed in the vertical direction in the corresponding segment on the V map. Normally, if there is sufficient road surface parallax and the road surface can be estimated correctly, the estimated road surface is present on a voting point group corresponding to the road surface parallax. Therefore, the extended road surface also exists on the voting point group corresponding to the parallax of the road surface. However, when the estimated road surface is pulled up with an inappropriate inclination due to object parallax, the extended road surface deviates from the coordinate distribution (voting point group) where pixels having parallax (pixels of the parallax image) are voted. Will be estimated at the position. As shown in FIG. 25 (B), the extended road surface B obtained by extending the estimated road surface B is estimated on the distribution of the coordinates where the pixels having the parallax are voted. When it is properly lifted, the extended road surface A is present in a region that is not on the coordinate distribution in which the pixels having parallax have been voted.

  Therefore, in this embodiment, for each segment, an estimated road surface is extended to set an extended road surface, and the number of coordinates at which pixels having parallax are voted on the extended road surface with respect to the length of the extended road surface. It is counted whether it exists, and it is determined whether the ratio (count number / length of extended road surface (the length of the segment may be used)) is equal to or greater than a threshold value. If it is equal to or greater than the threshold, it is assumed that the estimated road surface in the target segment has been correctly estimated by picking up the road surface parallax, and the estimation is successful (the estimated road surface is not rejected). On the other hand, if it is less than the threshold, it is determined that the estimation has failed, the estimated road surface in the segment of interest is rejected, and a default road surface or a history road surface is set.

  Here, it has been described that the number of coordinate points on the extended road surface (the number of coordinates where a pixel having a parallax has been voted) is counted. However, if attention is paid only to points on the line, there is a slight difference in inclination. May not be measured correctly. Therefore, a margin line may be provided to count the number of coordinate points in an area between the extended road surface and the margin line. Here, it can be considered that the region between the extended road surface and the margin line corresponds to the predetermined shape (a predetermined shape set based on the estimated road surface). The margin line is not limited to the downward direction of the extended road surface, and may be provided in the upward direction. If there is only one margin line, the number of coordinate points in the area sandwiched between the margin line and the extended road surface is counted. If there are two margin lines, the number of coordinates in the area sandwiched between the two outermost lines is counted. Count the number of coordinate points. In addition, when supplementing the count, the number of coordinate points having a frequency value of 1 or more may be counted, and if it is less than a predetermined value, it may be regarded as noise parallax, and coordinates having a frequency value greater than or equal to the predetermined value may be counted. Good. Further, the road surface to be extended may be executed only for a closer segment or a predetermined distance in the vicinity, or may be extended to a remote segment or a predetermined distance in the distance. That is, the length of the extended road surface can be arbitrarily set. In addition, the range to which this process is applied may be limited to a predetermined segment or a predetermined distance (for example, it can be used only for the second segment and the third segment). In addition to the above-described processing, different success / failure determinations such as success / failure determination by angle and success / failure determination by dispersion of sample point groups may be applied in addition to the above processing. An example of success / failure determination based on an angle and an example of success / failure determination based on dispersion are as described above.

  The flow of processing by the road surface detection processing unit 122 of this embodiment is the same as the flowchart shown in FIG. 24, and the determination processing in step S6 is different from that of the first embodiment described above. More specifically, the rejection unit 173 sets an extended road surface obtained by extending the estimated road surface in the focused segment, and counts the number of coordinates on which the pixels having parallax have been voted on the extended road surface, The ratio (count number / length of extended road surface (segment length may be used)) is calculated. When the calculated ratio is less than the threshold value, it is determined that the estimated road surface corresponding to the extended road surface (the estimated road surface corresponding to the target segment) is rejected. On the other hand, when the calculated ratio is equal to or greater than the threshold value, it is determined that the estimated road surface corresponding to the extended road surface is not rejected.

  As described above, in this embodiment, the V map is divided into a plurality of segments, and the road surface is estimated for each segment. Then, for each segment, an extended road surface obtained by extending the estimated road surface is set, and on the V map, when the ratio of the total number of coordinates in which the pixels having the parallax value d occupying the extended road surface is less than the threshold value, It is determined that the estimated road surface corresponding to the extended road surface is pulled up under the influence of object parallax, and the estimated road surface is rejected (estimation is determined to be unsuccessful). As a result, it is possible to prevent object detection from being performed using an estimated road surface that is different from the actual road surface, and as a result, sufficient object detection accuracy can be ensured.

  Note that the first embodiment and the second embodiment described above can be used in combination. For example, the rejection unit 173 may be configured to switch between the rejection determination of the first embodiment and the rejection determination of the second embodiment for each segment. For example, if the position of the estimated road surface in the target segment is higher than the default road surface or the history road surface by a predetermined value or more, the rejection unit 173 performs the rejection determination described in the second embodiment as a rejection determination of the estimated road surface in the target segment. If the height is less than a predetermined value, the rejection determination described in the first embodiment may be performed. For example, when the absolute value of the slope of the extended road surface set by extending the estimated road surface corresponding to the target segment is greater than or equal to a predetermined value (steep), the rejection unit 173 extends the estimated road surface corresponding to the target segment. The rejection determination described in the second embodiment may be performed as the rejection determination, and when the absolute value of the slope is less than a predetermined value, the rejection determination described in the first embodiment may be performed.

  As mentioned above, although embodiment which concerns on this invention was described, this invention is not limited to each above-mentioned embodiment as it is, A component can be deform | transformed and embodied in the range which does not deviate from the summary in an implementation stage. . Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiment.

  The program executed in the mobile control system 100 according to the above-described embodiment is a file in an installable or executable format, and is a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disk). ), A computer-readable recording medium such as a USB (Universal Serial Bus), or the like, or may be provided or distributed via a network such as the Internet. Various programs may be provided by being incorporated in advance in a ROM or the like.

DESCRIPTION OF SYMBOLS 1A 1st camera part 1B 2nd camera part 5 Lens 6 Image sensor 7 Sensor controller 10 Data bus line 11 Serial bus line 15 CPU
16 FPGA
17 ROM
18 RAM
19 Serial IF
20 Data IF
DESCRIPTION OF SYMBOLS 100 Mobile body control system 101 Vehicle 102 Imaging unit 103 Analysis unit 104 Control unit 105 Display unit 106 Windshield 111 Preprocessing unit 112 Parallelized image generation unit 113 Parallax image generation unit 114 Object detection processing unit 122 Road surface detection processing unit 123 Clustering process Unit 124 tracking processing unit 125 acquisition unit 126 first generation unit 127 road surface estimation unit 130 second generation unit 140 isolated region detection processing unit 150 parallax image processing unit 160 rejection processing unit 171 division unit 172 estimation unit 173 rejection unit 174 interpolation Part

JP 2011-128844 A

Claims (15)

An acquisition unit for acquiring a distance image having distance information for each pixel;
Based on a plurality of pixels included in the distance image, a generation unit that generates correspondence information in which a position in the vertical direction is associated with a position in the depth direction;
For each of the plurality of segments obtained by dividing the correspondence information, an estimation unit that estimates the shape of a reference object that serves as a reference for the height of the object;
For each segment, the distribution of the distance information of the voted pixels matches a predetermined criterion based on a predetermined shape set based on the estimated shape indicating the shape of the reference object estimated by the estimation unit. And a rejection unit for rejecting the estimated shape,
Information processing device. The predetermined shape includes a shape obtained by extending a shape based on the estimated shape to the adjacent segment.
The information processing apparatus according to claim 1. The shape based on the estimated shape is the estimated shape itself.
The information processing apparatus according to claim 2. The shape based on the estimated shape is a margin line of the estimated shape.
The information processing apparatus according to claim 3. The rejection unit determines whether the estimated shape can be rejected according to the distribution of the distance information existing below the predetermined shape.
The information processing apparatus according to any one of claims 2 to 4. The rejecting unit rejects the estimated shape when the frequency value of the distance information existing below the predetermined shape is a threshold value or more.
The information processing apparatus according to claim 5. The rejection unit generates, for each position in the depth direction, a frequency histogram that associates a frequency value obtained by counting the distance information existing below the predetermined shape, and the corresponding frequency value is equal to or greater than a predetermined value. Measure the number of bins indicating the number of positions in the depth direction, and if the ratio of the number of bins to the length of the segment is equal to or greater than the threshold, reject the estimated shape,
The information processing apparatus according to claim 6. The rejecting unit, when the ratio of the total number of coordinates in which the pixels having the distance information are voted in the predetermined area below the predetermined shape in the correspondence information is equal to or greater than the threshold, the estimated shape Reject
The information processing apparatus according to claim 6. The rejection unit rejects the estimated shape when the ratio of the total number of coordinates in which the pixels having the distance information occupy the predetermined shape is less than the threshold value,
The information processing apparatus according to claim 2. When the estimated shape corresponding to the segment is rejected by the rejecting unit, the setting unit further includes a setting unit that sets a predetermined shape as the shape of the reference object corresponding to the segment.
The information processing apparatus according to any one of claims 1 to 9. The predetermined shape includes a default shape assumed to be a flat shape or a history shape indicating a shape estimated in a past frame.
The information processing apparatus according to claim 10. An imaging unit for imaging a stereo image;
A distance image generating unit that generates a distance image having distance information for each pixel from the stereo image captured by the imaging unit;
Based on a plurality of pixels included in the distance image, a generation unit that generates correspondence information in which a position in the vertical direction is associated with a position in the depth direction;
For each of the plurality of segments obtained by dividing the correspondence information, an estimation unit that estimates the shape of a reference object that serves as a reference for the height of the object;
For each segment, the distribution of the distance information of the voted pixels matches a predetermined criterion based on a predetermined shape set based on the estimated shape indicating the shape of the reference object estimated by the estimation unit. And a rejection unit for rejecting the estimated shape,
Imaging device. An apparatus control system including an imaging apparatus and a control unit that controls the apparatus based on an output result of the imaging apparatus,
The imaging device
An imaging unit for imaging a stereo image;
A distance image generating unit that generates a distance image having distance information for each pixel from the stereo image captured by the imaging unit;
Based on a plurality of pixels included in the distance image, a generation unit that generates correspondence information in which a position in the vertical direction is associated with a position in the depth direction;
For each of the plurality of segments obtained by dividing the correspondence information, an estimation unit that estimates the shape of a reference object that serves as a reference for the height of the object;
For each segment, the distribution of the distance information of the voted pixels matches a predetermined criterion based on a predetermined shape set based on the estimated shape indicating the shape of the reference object estimated by the estimation unit. And a rejection unit for rejecting the estimated shape,
Equipment control system. An acquisition step of acquiring a distance image having distance information for each pixel;
Based on a plurality of pixels included in the distance image, a generation step for generating correspondence information in which a position in the vertical direction and a position in the depth direction are associated with each other;
For each of a plurality of segments obtained by dividing the correspondence information, an estimation step for estimating the shape of a reference object that is a reference for the height of the object;
For each of the segments, the distribution of the distance information of the voted pixels matches a predetermined criterion based on a predetermined shape set based on the estimated shape indicating the shape of the reference object estimated by the estimating step. And a rejection step of rejecting the estimated shape,
Information processing method. On the computer,
An acquisition step of acquiring a distance image having distance information for each pixel;
A step of voting a plurality of pixels included in the distance image and generating correspondence information in which a position in the vertical direction and a position in the depth direction are associated with each other,
For each of a plurality of segments obtained by dividing the correspondence information, an estimation step for estimating the shape of a reference object that is a reference for the height of the object;
For each of the segments, the distribution of the distance information of the voted pixels matches a predetermined criterion based on a predetermined shape set based on the estimated shape indicating the shape of the reference object estimated by the estimating step. And a rejection step for rejecting the estimated shape.