JP2016206801A - Object detection device, mobile equipment control system and object detection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly detect objects by suppressing such a situation that image regions showing a plurality of approaching objects are erroneously selected as one candidate region.SOLUTION: On the basis of parallax image information for an imaging region, parallax histogram information (real frequency U map information) showing a frequency distribution of parallax values d in each row region obtained by dividing the photographed image into plural pieces in the longitudinal direction is generated. On the basis of the parallax histogram information, image regions approaching in the longitudinal direction of the photographed image, and having proximate parallax values with a frequency equal to or greater than a prescribed value are selected as candidate regions (isolated regions) of a detection object image region showing a detection object. With this, the detection object existing in the imaging region is detected. The image region having a real space height satisfying a prescribed height condition is selected as the candidate region using height distribution information showing a distribution of real space heights on the parallax image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an object detection device, a mobile device control system, and an object detection program.

  Conventionally, parallax image information is acquired from two captured images obtained by capturing a front area (imaging area) of a moving body such as an automobile with a stereo camera, and the other area exists in the front area of the moving body based on the parallax image information. An object detection device that detects a detection target such as a vehicle is known.

  For example, Patent Document 1 discloses this type of object detection device. In this object detection device, parallax histogram information (U map information) indicating the frequency distribution of parallax values in each row region obtained by dividing a captured image into a plurality of left and right directions is generated from parallax image information. Then, based on the generated parallax histogram information, detection is performed in which a detection target is projected in an image area that is close in the left-right direction of the captured image and that has a value of a parallax value having a frequency equal to or higher than a predetermined value. The candidate image area is selected as a candidate area. Then, the width and height of the object projected in the selected candidate area are calculated in real space, and the calculated width and height are compared with the table data. It is determined whether or not it is a detection object image area to be projected.

  However, in the object detection device described in Patent Document 1, when selecting as candidate areas, a plurality of image areas that project a plurality of adjacent objects are erroneously selected as one candidate area (an image area that displays one object). May end up. For example, when the host vehicle tries to pass a road on which another vehicle is parked near a wall that is located on the side of the road, the image areas in which these walls and the other vehicle are respectively shown in the parallax histogram information The values of parallax values that are close in the left-right direction and have a frequency equal to or higher than a predetermined value are close to each other. For this reason, the image areas in which these walls and other vehicles are projected are selected as one candidate area. As described above, when a plurality of image areas displaying a plurality of objects are selected as one candidate area, these objects cannot be appropriately detected.

  In order to solve the above-described problem, the present invention provides a parallax image information generation unit that generates parallax image information about an imaging region, and the captured image based on the parallax image information generated by the parallax image information generation unit. Based on the parallax histogram information generating means for generating parallax histogram information indicating the frequency distribution of the parallax values in each row region obtained by dividing into multiple in the left-right direction, and the parallax histogram information generated by the parallax histogram information generating means, An image area that is close in the left-right direction of the captured image and that is close to a parallax value having a frequency equal to or higher than a predetermined value is selected as a candidate area for the detection target image area that displays the detection target. Detection object image area specifying means for specifying a detection object image area from the selected candidate areas according to a predetermined specific condition, and the imaging area A height distribution information generating unit configured to generate height distribution information indicating a real space height distribution on the parallax image based on the parallax image information in an object detection device that detects a detection target existing in the object; And the detection object image area specifying means uses the height distribution information to select an image area whose real space height satisfies a predetermined height condition as the candidate area.

  According to the present invention, it is possible to suppress a situation in which a plurality of image regions that project a plurality of adjacent objects are erroneously selected as one candidate region, and it is possible to appropriately detect the object. can get.

It is a mimetic diagram showing a schematic structure of an in-vehicle device control system in an embodiment. It is a schematic diagram which shows schematic structure of the imaging unit which comprises the same vehicle equipment control system. FIG. 3 is a processing block diagram for explaining object detection processing realized by the image processing board and the image analysis unit in FIG. 2. It is explanatory drawing which shows an example of an object data list. (A) is explanatory drawing which shows an example of the parallax value distribution of a parallax image. (B) is explanatory drawing which shows the row parallax distribution map (V map) which shows the parallax value frequency distribution for every row | line | column of the parallax image of the same (a). It is an example of an image showing typically an example of a standard picture imaged with one image pick-up part. It is explanatory drawing which shows the V map corresponding to the example of an image of FIG. It is explanatory drawing which shows the V map for demonstrating the extraction conditions in embodiment. It is a processing block diagram in a road surface shape detection part. It is a flowchart which shows the flow of the road surface candidate point detection process performed in a road surface candidate point detection part. It is a flowchart which shows the flow of the piecewise straight line approximation process performed in a piecewise straight line approximation part. It is an example of an image showing typically an example of a standard picture imaged with one image pick-up part. It is explanatory drawing which shows the frequency U map corresponding to the example of an image of FIG. It is explanatory drawing which shows the height U map corresponding to the example of an image of FIG. It is a real frequency U map corresponding to the frequency U map shown in FIG. It is a flowchart which shows the flow of the process performed in an isolated area | region detection part. It is explanatory drawing which shows the real frequency U map which set the rectangular area | region which the isolated area detected by the isolated area detection part inscribed. It is explanatory drawing which showed typically the parallax image corresponding to the real frequency U map shown in FIG. It is explanatory drawing which shows the parallax image which set the circumscribed rectangle of an object line group. It is a table | surface which shows an example of the table data for classifying an object type. It is the example of an image showing typically an example of a standard picture of the situation where other vehicles are parked near the wall which exists on the roadside. It is explanatory drawing which shows the real frequency U map corresponding to the example of an image of FIG. It is explanatory drawing which shows the real height U map corresponding to the example of an image of FIG. It is a flowchart which shows the flow of the height separation process in embodiment. It is explanatory drawing for demonstrating the setting method of the search range in the same height separation process. It is a flowchart which shows the flow of the search process with respect to the search range of the wall in the same height separation process. It is a flowchart which shows the flow of the search process with respect to the search range of the vehicle in the same height separation process.

Hereinafter, an embodiment in which an object detection device according to the present invention is used in an in-vehicle device control system that is a mobile device control system will be described.
FIG. 1 is a schematic diagram illustrating a schematic configuration of an in-vehicle device control system according to the present embodiment.
The in-vehicle device control system detects a detection target from captured image data of a forward area (imaging area) in the traveling direction of the host vehicle captured by the imaging unit 101 mounted on the host vehicle 100 such as an automobile that is a moving body, Various in-vehicle devices are controlled using the detection results.

  The in-vehicle device control system of the present embodiment is provided with an image pickup unit 101 that picks up an image of an area in the traveling direction of the traveling vehicle 100 as an image pickup area. For example, the imaging unit 101 is installed in the vicinity of a room mirror of the windshield 105 of the host vehicle 100. Various data such as captured image data obtained by imaging by the imaging unit 101 is input to the image analysis unit 102. The image analysis unit 102 analyzes the data transmitted from the imaging unit 101 and detects a detection target such as another vehicle existing in the imaging region.

  The detection result of the image analysis unit 102 is sent to the vehicle travel control unit 106. Based on the detection result detected by the image analysis unit 102, the vehicle travel control unit 106 notifies the driver of the host vehicle 100 of a warning using the display monitor 103, or controls the handle or brake of the host vehicle. Carry out driving support control.

FIG. 2 is a schematic diagram illustrating a schematic configuration of the imaging unit 101.
The imaging unit 101 includes a stereo camera including two imaging units 110A and 110B as imaging means, and the two imaging units 110A and 110B are the same. Each of the imaging units 110A and 110B includes imaging lenses 111A and 111B, image sensors 113A and 113B in which light receiving elements are two-dimensionally arranged, and image sensor controllers 115A and 115B, respectively. The image sensor controllers 115A and 115B play a role such as exposure control of the image sensors 113A and 113B, image readout control, communication with an external circuit, transmission of image data, and the like. From the image sensor controllers 115A and 115B, captured image data (luminance image data) obtained by converting analog electrical signals (the amount of light received by the respective light receiving elements on the image sensor) output from the image sensors 113A and 113B into digital electrical signals. Is output.

  The imaging unit 101 includes an image processing board 120 connected to the two imaging units 110A and 110B by a data bus and a serial bus. The image processing board 120 is provided with a CPU (Central Processing Unit), RAM, ROM, FPGA (Field-Programmable Gate Array), serial IF (interface), data IF, and the like. The data bus of the image processing board 120 transfers luminance image data output from the two imaging units 110A and 110B to the RAM on the image processing board 120. The serial bus of the image processing board 120 transmits / receives a sensor exposure control value change command, an image read parameter change command, various setting data, and the like from the CPU and FPGA. The FPGA of the image processing board 120 performs a parallax image by performing a process requiring real-time processing on the image data stored in the RAM, for example, gamma correction, distortion correction (parallelization of left and right images), and parallax calculation by block matching. Data is generated and written back to the RAM. The CPU loads a program for executing road surface shape detection processing, object detection processing, and the like from the ROM, inputs luminance image data and parallax image data stored in the RAM, and executes various processes. Various types of detected data are output from the data IF or serial IF to the outside. When executing various processes, vehicle information (vehicle speed, acceleration, rudder angle, yaw rate, etc.) is appropriately input using the data IF and used as parameters for various processes.

Next, the object detection process in this embodiment will be described.
FIG. 3 is a processing block diagram for explaining object detection processing realized mainly by the CPU on the image processing board 120 executing a program or by an FPGA.
Luminance image data is output from the two imaging units 110A and 110B constituting the stereo camera. At this time, when the imaging units 110A and 110B are in color, color luminance conversion for obtaining a luminance signal (Y) from the RGB signals is performed using, for example, the following equation (1).
Y = 0.3R + 0.59G + 0.11B (1)

  When the luminance image data is input, first, the parallelized image generation unit 131 executes a parallelized image generation process. This parallelized image generation processing is performed using luminance image data (reference image) output from each of the imaging units 110A and 110B based on the distortion of the optical system in the imaging units 110A and 110B and the relative positional relationship between the left and right imaging units 110A and 110B. And the comparison image) are converted into an ideal parallel stereo image obtained when two pinhole cameras are mounted in parallel. This is because the amount of distortion in each pixel is calculated using polynomials Δx = f (x, y) and Δy = g (x, y), and the calculation results are used to calculate the distortion amount from each of the imaging units 110A and 110B. Each pixel of the output luminance image data (reference image and comparison image) is converted. The polynomial is based on, for example, a quintic polynomial relating to x (the horizontal position of the image) and y (the vertical position of the image).

  After performing the parallelized image processing in this way, next, the parallax image generation unit 132 configured by FPGA or the like performs parallax image generation processing for generating parallax image data (parallax image information). In the parallax image generation processing, first, the luminance image data of one imaging unit 110A of the two imaging units 110A and 110B is set as reference image data, and the luminance image data of the other imaging unit 110B is set as comparison image data, and these are used. The parallax between them is calculated to generate and output parallax image data. The parallax image data indicates a parallax image in which pixel values corresponding to the parallax value d calculated for each image portion on the reference image data are represented as pixel values of the respective image portions.

  Specifically, the parallax image generation unit 132 defines a block composed of a plurality of pixels (for example, 16 pixels × 1 pixel) centered on one target pixel for a certain row of reference image data. On the other hand, in the same row in the comparison image data, a block having the same size as the block of the defined reference image data is shifted by one pixel in the horizontal line direction (X direction) to show the feature of the pixel value of the block defined in the reference image data. Correlation values indicating the correlation between the feature amount and the feature amount indicating the feature of the pixel value of each block in the comparison image data are calculated. Then, based on the calculated correlation value, a matching process is performed for selecting a block of comparison image data that is most correlated with the block of reference image data among the blocks in the comparison image data. Thereafter, a positional deviation amount between the target pixel of the block of the reference image data and the corresponding pixel of the block of the comparison image data selected by the matching process is calculated as the parallax value d. The parallax image data can be obtained by performing such processing for calculating the parallax value d for the entire area of the reference image data or a specific area.

  As the feature amount of the block used for the matching process, for example, the value (luminance value) of each pixel in the block can be used, and as the correlation value, for example, the value (luminance) of each pixel in the block of the reference image data Value) and the sum of absolute values of the differences between the values (luminance values) of the pixels in the block of comparison image data corresponding to these pixels, respectively. In this case, it can be said that the block having the smallest sum is most correlated.

  When the matching processing in the parallax image generation unit 132 is realized by hardware processing, for example, SSD (Sum of Squared Difference), ZSSD (Zero-mean Sum of Squared Difference), SAD (Sum of Absolute Difference), ZSAD ( A method such as Zero-mean Sum of Absolute Difference can be used. In the matching process, only a parallax value in units of pixels can be calculated. Therefore, when a sub-pixel level parallax value less than one pixel is required, an estimated value needs to be used. As the estimation method, for example, an equiangular straight line method, a quadratic curve method, or the like can be used. However, since an error occurs in the estimated parallax value at the sub-pixel level, EEC (estimated error correction) or the like for reducing the estimated error may be used.

Next, the object tracking unit 133 executes processing for tracking the object (detection target) detected in the image detection processing of the past imaging frame.
More specifically, in the present embodiment, an object data list 135 indicating information on objects detected by past image detection processing is stored. As shown in FIG. 4, the object data list 135 includes object prediction data (the object in the next imaging frame) in addition to the latest information (latest position, size, distance, relative speed, parallax information) of the detected object data. Information indicating the position of the object), the object feature amount used in the object tracking unit 133 and the object matching unit 147 described later, and how many frames the object is detected or not detected continuously. This includes the number of detected / undetected frames, tracking accuracy (stability flag) indicating whether the object is a target to be tracked, and the like.

  The object selection unit 134 determines whether the object stored in the object data list 135 has a high probability of existence of the object (S = 1), whether the object is in a position suitable for tracking, and the like. To select whether to track the object. Specifically, after setting the prediction range in the parallax image data where the object is predicted to be located based on the object prediction data of the object data list 135, and specifying the height of the object within the prediction range, With reference to the object feature amount of the object data list 135, the width of the object is specified from the specified height, and the horizontal position of the object on the parallax image data is estimated from the specified width. The lateral position of the object estimated in this way is a predetermined tracking target condition (when the object is in a position where the probability that the object exists in the image is high, or when the object is a position suitable for tracking) If this condition is satisfied, the object is selected as a tracking target, and the result is output to the object tracking unit 133. Note that the information in the object data list 135 is updated based on the result of the object selection unit 134.

  The object tracking unit 133 performs tracking processing on the object selected as the tracking target by the object selection unit 134. For this tracking process, the object tracking unit 133 receives the parallax image data generated by the parallax image generation unit 132 and the object prediction data and the object feature amount of the object data selected by the object selection unit 134.

  The parallax interpolation unit 136 performs a parallax interpolation process for interpolating the parallax value of the image portion for which the parallax value could not be calculated by the parallax image generation process in the parallax image generation unit 132 described above. For example, in the luminance image data, block matching in the parallax image generation processing cannot be calculated correctly at the edge portion in the horizontal direction or a portion where the luminance change is small, and the parallax value may not be calculated. In such a case, in the parallax interpolation process, for an image portion that satisfies a predetermined interpolation condition, a process for filling between two distant pixels with the interpolation parallax is performed. As an interpolation condition, for example, a condition that the distance between two points in the same lateral coordinate is smaller than a predetermined size (width), and no other parallax value exists between the two points.

  After performing the parallax interpolation processing as described above, the V map generation unit 137 next executes V map generation processing for generating a V map. Each piece of parallax pixel data included in the parallax image data is indicated by a set (x, y, d) of an x-direction position, a y-direction position, and a parallax value d, which is d on the X axis and y on the Y axis. , Converted to three-dimensional coordinate information (d, y, f) with the frequency f set on the Z-axis, or tertiary limited to information exceeding a predetermined frequency threshold from the three-dimensional coordinate information (d, y, f) Original coordinate information (d, y, f) is generated. The three-dimensional coordinate information (d, y, f) distributed in the XY two-dimensional coordinate system is called a V map.

  More specifically, the V map generation unit 137 calculates a disparity value frequency distribution for each row region of disparity image data obtained by dividing an image into a plurality of parts in the vertical direction. To explain with a specific example, when parallax image data having a parallax value distribution as shown in FIG. 5A is input, the V map generation unit 137 determines the number of pieces of data of each parallax value for each row. A disparity value frequency distribution as a distribution is calculated and output. The parallax value frequency distribution information of each row obtained in this way is represented on a two-dimensional orthogonal coordinate system in which the y-axis position on the parallax image (the vertical position of the captured image) is taken on the Y-axis and the parallax value is taken on the X-axis. By expressing in V, a V map as shown in FIG. 5B can be obtained. This V map can also be expressed as an image in which pixels having pixel values corresponding to the frequency f are distributed on the two-dimensional orthogonal coordinate system.

Next, in this embodiment, road surface shape detection in which the road surface shape detection unit 138 detects the three-dimensional shape of the front road surface of the host vehicle 100 from the V map information (parallax histogram information) generated by the V map generation unit 137. Processing is executed.
FIG. 6 is an image example schematically illustrating an example of a reference image captured by the imaging unit 110A.
FIG. 7 is a V map corresponding to the image example of FIG.
In the image example shown in FIG. 6, a road surface on which the host vehicle 100 is traveling, a preceding vehicle existing in front of the host vehicle 100, and a utility pole existing outside the road are displayed. This image example is obtained by extending a road surface in which the front road surface of the host vehicle 100 is relatively flat, that is, a surface in which the front road surface of the host vehicle 100 is parallel to a road surface portion directly below the host vehicle 100 to the front side of the host vehicle 100. This is a case where it coincides with a virtual reference road surface (virtual reference movement surface). In this case, in the lower part of the V map corresponding to the lower part of the image, the high-frequency points are distributed in a substantially straight line having an inclination such that the parallax value d decreases toward the upper part of the image. Pixels exhibiting such a distribution are pixels that are present at almost the same distance in each row on the parallax image, have the highest occupation ratio, and project a detection object whose distance continuously increases toward the top of the image. It can be said that.

  Since the imaging unit 110A captures an area in front of the host vehicle, the parallax value d of the road surface decreases as the content of the captured image moves upward as shown in FIG. Also, in the same row (horizontal line), the pixels that project the road surface have substantially the same parallax value d. Therefore, the high-frequency points distributed substantially linearly on the V map correspond to the characteristics of the pixels that project the road surface (moving surface). Therefore, it is possible to estimate that the pixels of the points distributed on or near the approximate straight line obtained by linearly approximating high-frequency points on the V map are the pixels displaying the road surface with high accuracy. Further, the distance to the road surface portion projected on each pixel can be obtained with high accuracy from the parallax value d of the corresponding point on the approximate straight line.

  Here, when linearly approximating high-frequency points on the V map, the range of points to be included in the linear approximation processing greatly affects the accuracy of the processing result. In other words, the wider the range included in the straight line approximation process, the more points that do not correspond to the road surface are included, and the processing accuracy is reduced. The narrower the range included in the straight line approximation process is, the smaller the number of points corresponding to the road surface is. However, the processing accuracy is reduced. Therefore, in the present embodiment, the parallax histogram information portion that is the target of the straight line approximation process is extracted as follows.

FIG. 8 is an explanatory diagram showing a V map for explaining extraction conditions in the present embodiment.
When the V map generation unit 137 of the present embodiment receives parallax image data, each parallax pixel data (x, y, d) included in the parallax image data is used as a parallax histogram information constituent element that is three-dimensional coordinate information. To V map information (d, y, f) to generate V map information as parallax histogram information. At this time, the parallax pixel data in which the relationship between the image vertical position y and the parallax value d satisfies a predetermined extraction condition is extracted from the parallax image data, and the above-described conversion is performed on the extracted parallax pixel data. To generate V map information.

  The extraction condition in the present embodiment corresponds to a virtual reference road surface (virtual reference movement surface) obtained by extending a front road surface of the host vehicle 100 parallel to a road surface portion immediately below the host vehicle 100 to the front of the host vehicle. This is a condition that the image belongs to a predetermined extraction range determined based on the relationship between the parallax value d and the image vertical position y. The relationship between the parallax value d corresponding to the reference road surface and the image vertical position y is indicated by a straight line (hereinafter referred to as “reference straight line”) on the V map, as shown in FIG. In the present embodiment, a range of ± δ in the vertical direction of the image around this straight line is defined as the extraction range. This extraction range is set so as to include the fluctuation range of the actual road surface V map element (d, y, f) that changes momentarily according to the situation.

  Specifically, for example, when the road surface ahead of the host vehicle is relatively upwardly inclined, the road surface image portion (moving surface image region) displayed in the captured image is more than when the road surface is relatively flat. Expands to the top of the image. In addition, when comparing road surface image portions projected at the same image vertical direction position y, the parallax value d is greater when the slope is relatively upward than when it is relatively flat. In this case, the V map element (d, y, f) is a straight line that is positioned above the reference straight line and has a large slope (absolute value) on the V map. In the present embodiment, the V map element (d, y, f) of the road surface falls within the extraction range as long as the relative upward inclination on the road surface in front can be assumed.

  Further, for example, when the road surface in front of the host vehicle has a relatively downward slope, the V map element (d, y, f) is located below the reference straight line on the V map, and A straight line having a small slope (absolute value) is shown. In the present embodiment, the V map element (d, y, f) of the road surface falls within the extraction range as long as the relative downward inclination on the road surface ahead can be assumed.

  Further, for example, during acceleration in which the host vehicle 100 is accelerating the speed, a weight is applied to the rear of the host vehicle 100, and the posture of the host vehicle is such that the front of the host vehicle is directed upward in the vertical direction. In this case, as compared with the case where the speed of the host vehicle 100 is constant, the road surface image portion (moving surface image region) displayed in the captured image is shifted to the lower side of the image. In this case, the V map element (d, y, f) is located on the lower side of the reference line on the V map and indicates a straight line substantially parallel to the reference straight line. In the present embodiment, the V map element (d, y, f) of the road surface is within the extraction range as long as the acceleration of the host vehicle 100 can be assumed.

  In addition, for example, when the host vehicle 100 is decelerating at a reduced speed, a load is applied to the front of the host vehicle 100, and the posture of the host vehicle is such that the front of the host vehicle faces downward in the vertical direction. . In this case, compared with the case where the speed of the host vehicle 100 is constant, the road surface image portion (moving surface image region) displayed in the captured image is shifted to the upper side of the image. In this case, the V map element (d, y, f) is located on the upper side of the reference line on the V map and shows a straight line substantially parallel to the reference straight line. In the present embodiment, the V map element (d, y, f) of the road surface is within the extraction range as long as the host vehicle 100 can be decelerated.

  In the road surface shape detection unit 138, when the V map information is generated by the V map generation unit 137, the road surface shape detection unit 138 moves upward in the feature indicated by the set (V map element) of the parallax value and the y-direction position corresponding to the road surface, that is, above the captured image. A process of linearly approximating a high-frequency point on the V map indicating the feature that the parallax value becomes lower as the value becomes smaller. When the road surface is flat, it can be approximated with sufficient accuracy with a single straight line.However, with respect to a road surface in which the slope of the road surface changes in the vehicle traveling direction, sufficient accuracy can be achieved with a single straight line. Approximation is difficult. Therefore, in this embodiment, the V map information (V map information) is divided into two or more parallax value sections according to the parallax value, and linear approximation is performed individually for each parallax value section.

FIG. 9 is a processing block diagram in the road surface shape detection unit 138.
When the road surface shape detection unit 138 of the present embodiment receives the V map information (V map information) output from the V map generation unit 137, the road surface candidate point detection unit 138A first determines a V map element corresponding to the road surface. A high-frequency point on the V map showing the characteristic to be shown, that is, the characteristic that the parallax value becomes lower toward the upper side of the captured image is detected as a road surface candidate point.

  At this time, in the present embodiment, the road surface candidate point detection process in the road surface candidate point detection unit 138A divides the V map information (V map information) into two or more parallax value sections according to the parallax values, and sets each parallax. Road surface candidate points in each parallax value section are determined according to a determination algorithm corresponding to each value section. Specifically, for example, with the parallax value corresponding to a predetermined reference distance as a boundary, the V map is divided into two regions in the X-axis direction (horizontal axis direction), that is, a region having a large parallax value and a region having a small parallax value. Road surface candidate points are detected using different road surface candidate point detection algorithms for each region. Note that a first road surface candidate point detection process described later is performed for a short distance area with a large parallax value, and a second road surface candidate point detection process described below is performed for a long distance area with a small parallax.

Here, the reason for changing the method of road surface candidate point detection processing between the short-distance region having a large parallax and the long-distance region having a small parallax as described above will be described.
As shown in FIG. 6, in the captured image obtained by capturing the front of the host vehicle 100, the area occupied by the road surface image area is large and the number of pixels corresponding to the road surface is large for a short-distance road surface portion. The frequency of On the other hand, for a road surface portion at a long distance, the occupied area in the captured image of the road surface image region is small and the number of pixels corresponding to the road surface is small, so the frequency on the V map is low. That is, in the V map, the frequency value of the point corresponding to the road surface is small at a long distance and large at a short distance. Therefore, if you try to detect road surface candidate points with the same criteria for both areas, for example, using the same frequency threshold, you can properly detect road surface candidate points for short distance areas, but road surface candidate points are appropriate for long distance areas. The road surface detection accuracy in a long-distance area deteriorates. On the contrary, if the short distance area is detected on the basis of sufficiently detecting road surface candidate points in the long distance area, many noise components in the short distance area are detected, and the road surface detection accuracy in the short distance area deteriorates. Therefore, in this embodiment, the V map is divided into a short-distance area and a long-distance area, and road surface detection accuracy in both areas is detected by detecting road surface candidate points using a criterion and a detection method suitable for each area. Is kept high.

  In the first road surface candidate point detection process, for each parallax value d, the frequency value f of each V map element (d, y, f) included in the V map information is changed while changing the position in the y direction within a predetermined search range. A V map element that is larger than the first frequency threshold and has the largest frequency value f is searched, and the V map element is determined as a road surface candidate point for the parallax value d. In this case, it is preferable that the first frequency threshold value is set to be low so that the V map element corresponding to the road surface does not fall out. In the present embodiment, as described above, the V map generation unit 137 extracts the V map element corresponding to the road surface. Therefore, even if the first frequency threshold is set low, the V map does not correspond to the road surface portion. This is because the situation where elements are determined as road surface candidate points is reduced.

  Here, the search range in which the y value is changed for each parallax value d is the extraction range in the above-described V map generation unit 137, that is, ± δ in the image vertical direction centered on the image vertical direction position yp of the reference line. It is a range. Specifically, the range from “yp−δ” to “yp + δ” is set as the search range. Thereby, the range of the y value to be searched is limited, and high-speed road surface candidate point detection processing can be realized.

  On the other hand, the second road surface candidate point detection process is the same as the first road surface candidate point detection process except that a second frequency threshold different from this is used instead of the first frequency threshold. That is, in the second road surface candidate point detection process, for each parallax value d, the frequency value of each V map element (d, y, f) included in the V map information while changing the y-direction position within a predetermined search range. A V map element having the largest frequency value f and f being larger than the second frequency threshold is searched, and the V map element is determined as a road surface candidate point for the parallax value d.

FIG. 10 is a flowchart showing a flow of road surface candidate point detection processing performed by the road surface candidate point detection unit 138A.
For the input V map information, for example, road surface candidate points are detected in descending order of the parallax value d, and a road surface candidate point (y, d) for each parallax value d is detected. When the parallax value d is larger than the reference parallax value corresponding to the predetermined reference distance (Yes in S1), the above-described first road surface candidate point detection process is performed. That is, a search range of y (“yp−δ” to “yp + δ”) corresponding to the parallax value d is set (S2), and the V map element in which the frequency value f in the search range is larger than the first frequency threshold value. (D, y, f) is extracted (S3). Then, among the extracted V map elements, the V map element (d, y, f) having the maximum frequency value f is detected as a road surface candidate point of the parallax value d (S4).

  Then, the first road surface candidate point detection process is repeatedly performed until the parallax value d is equal to or smaller than the reference parallax value (S5). When the parallax value d is equal to or smaller than the reference parallax value (No in S1), this time, Two road surface candidate point detection processing is performed to detect road surface candidate points. That is, in the second road surface candidate point detection process, a search range (“yp−δ” to “yp + δ”) corresponding to the parallax value d is set (S6), and the frequency value f in the search range is the first value. A V map element (d, y, f) larger than the frequency threshold is extracted (S7). Then, among the extracted V map elements, the V map element (d, y, f) having the maximum frequency value f is detected as a road surface candidate point of the parallax value d (S8). This second road surface candidate point detection process is repeated until there is no parallax value d (S9).

  After the road surface candidate point detection unit 138A detects the road surface candidate points (extraction processing target) for each parallax value d in this way, the segmented straight line approximation unit 138B then approximates these road surface candidate points on the V map. A straight line approximation process is performed to obtain a straight line. At this time, if the road surface is flat, it is possible to approximate with sufficient accuracy with a single straight line over the entire range of the parallax value of the V map, but the inclination of the road surface changes in the vehicle traveling direction. Therefore, it is difficult to approximate with sufficient accuracy with a single straight line. Therefore, in the present embodiment, V map information (V map information) is divided into two or more parallax value sections according to the parallax value, and linear approximation processing is individually performed for each parallax value section.

  For the linear approximation process, least square approximation can be used, but other approximation such as RMA (Reduced Major Axis) is preferably used for more accurate execution. The reason for this is that the least square approximation is accurately calculated when there is an assumption that there is no error in the X-axis data and that there is an error in the Y-axis data. However, considering the nature of the road surface candidate points detected from the V map information, the data of each V map element included in the V map information indicates an accurate position on the image with respect to the Y-axis data y. It can be said that the parallax value d which is X-axis data includes an error. In the road surface candidate point detection process, a road surface candidate point is searched along the Y-axis direction and the V map element having the maximum y value is detected as a road surface candidate point. Axial errors are also included. Therefore, the V map element which is a road surface candidate point includes an error in both the X-axis direction and the Y-axis direction, and the premise of least square approximation is broken. Therefore, a regression line (RMA) compatible with the two variables (d and y) is effective.

FIG. 11 is a flowchart showing the flow of the piecewise straight line approximation process performed by the piecewise straight line approximation unit 138B.
When the piecewise straight line approximating unit 138B receives the road surface candidate point data of each parallax value d output from the road surface candidate point detecting unit 138A, first, the first interval of the nearest distance (section with the largest parallax value) is set. (S11). Then, road surface candidate points corresponding to the respective parallax values d in the first section are extracted (S12). At this time, when the number of extracted road surface candidate points is equal to or smaller than a predetermined value (No in S13), the first section is extended by a predetermined parallax value (S14). Specifically, the initial first section and the second section are combined to form a new first section (extended first section). At this time, the initial third section becomes a new second section. Then, the road surface candidate points corresponding to each parallax value d in the extended first section are extracted again (S12), and when the number of extracted road surface candidate points exceeds a predetermined value (S13) Yes), a straight line approximation process is performed on the extracted road surface candidate points (S15). In addition, when extending a section that is not the first section, for example, the second section, the first second section and the third section are combined to create a new second section (extended second section). And

  After performing the straight line approximation process in this way, next, the reliability of the approximate straight line obtained by the straight line approximation process is determined. In this reliability determination, first, it is determined whether or not the slope and intercept of the obtained approximate line are within a predetermined range (S17). If it is not within the predetermined range in this determination (No in S17), the first section is extended by a predetermined parallax value (S14), and the linear approximation process is performed again for the extended first section (S12). ~ 15). If it is determined that it is within the predetermined range (Yes in S17), it is determined whether or not the section on which the straight line approximation process is performed is the first section (S18).

  At this time, when it is determined that it is the first section (Yes in S18), it is determined whether or not the correlation value of the approximate straight line is larger than a predetermined value (S19). In this determination, if the correlation value of the approximate line is larger than a predetermined value, the approximate line is determined as the approximate line of the first section. If the correlation value of the approximate straight line is less than or equal to a predetermined value, the first section is extended by a predetermined disparity value (S14), and the extended first section is again subjected to a linear approximation process (S12 to 15). Reliability determination is performed again (S17 to S19). In addition, about the area which is not a 1st area (No of S18), the determination process (S19) regarding the correlation value of an approximate line is not implemented.

  Thereafter, it is confirmed whether there is a remaining section (S20). If there is no remaining section, the piecewise linear approximation unit 138B ends the piecewise straight line approximation process. On the other hand, when there is a remaining section (Yes in S20), a next section (second section) having a width corresponding to a distance obtained by multiplying a distance corresponding to the width of the previous section by a constant is set (S21). . Then, it is determined whether or not the section remaining after the setting is smaller than the section (third section) to be set next (S22). If it is determined in this determination that the distance is not small, road surface candidate points corresponding to the respective parallax values d in the second section are extracted to perform straight line approximation processing (S12 to S15), and reliability determination processing is performed ( S17 to S19).

  In this way, when the sections are sequentially set and the process of performing the linear approximation process and the reliability determination process in the sections is repeated, the section remaining after the setting is further set in the process step S22. It is determined that it is smaller than the interval to be performed (Yes in S22). In this case, the set section is extended to include the remaining section, and this is set as the last section (S23). In this case, road surface candidate points corresponding to the respective parallax values d in the last section are extracted (S12), and after the straight line approximation process is performed on the extracted road surface candidate points (S15), in the processing section S16, the last section is processed. Since it is determined that there is (Yes in S16), the piecewise line approximation unit 138B ends the piecewise line approximation process.

  Thus, the approximate straight line of each section obtained by the piecewise straight line approximation unit 138B executing the straight line approximation process of each section is not usually continuous at the section boundary. Therefore, in the present embodiment, the approximate straight line output from the segmented straight line approximation unit 138B is corrected by the segment approximate straight line continuation unit 138C so that the approximate straight line of each segment is continuous at the segment boundary. Specifically, for example, both approximate lines are corrected so as to pass the midpoint between the end points of the approximate lines of both sections on the boundary of the sections.

  As described above, when the road surface shape detection unit 138 obtains the information of the approximate straight line on the V map, the road surface height table calculation unit 139 then determines the road surface height (relative to the road surface portion directly below the host vehicle). Road surface height table calculation processing for calculating a table and calculating a table. From the approximate straight line information on the V map generated by the road surface shape detection unit 138, the distance to each road surface portion displayed in each row region (each position in the vertical direction of the image) on the captured image can be calculated. On the other hand, each surface part in the traveling direction of the virtual plane obtained by extending the road surface portion located directly below the traveling vehicle in front of the traveling direction of the vehicle so as to be parallel to the surface is displayed in which row area in the captured image. The virtual plane (reference road surface) is represented by a straight line (reference straight line) on the V map. By comparing the approximate straight line output from the road surface shape detection unit 138 with the reference straight line, the height of each road surface portion ahead of the host vehicle can be obtained. In a simple manner, the height of the road surface portion existing in front of the host vehicle can be calculated from the Y-axis position on the approximate straight line output from the road surface shape detection unit 138 by a distance obtained from the corresponding parallax value. The road surface height table calculation unit 139 tabulates the height of each road surface portion obtained from the approximate straight line for the necessary parallax range.

Note that the height from the road surface of the object projected on the captured image portion corresponding to the point where the Y-axis position is y ′ at a certain parallax value d is the y-axis position on the approximate straight line at the parallax value d. , It can be calculated from (y′−y0). In general, the height H from the road surface of the object corresponding to the coordinates (d, y ′) on the V map can be calculated from the following equation (2). However, in the following formula (2), “z” is a distance (z = BF / (d−offset)) calculated from the parallax value d, and “f” is the focal length of the camera (y′− It is a value converted into the same unit as the unit of y0). Here, “BF” is a value obtained by multiplying the base line length of the stereo camera and the focal length, and “offset” is a parallax value when an object at infinity is photographed.
H = z × (y′−y0) / f (2)

Next, the U map generation unit 140 will be described.
The U map generation unit 140 executes U map generation processing for generating a U map. In the U map generation processing, a set (x, y, d) of the x-direction position, the y-direction position, and the parallax value d in each piece of parallax pixel data included in the parallax image data is x on the X axis, d on the Y axis, Three-dimensional coordinates that are converted to three-dimensional coordinate information (x, d, f) in which the frequency is set on the Z axis, or limited to information that exceeds a predetermined frequency threshold from the three-dimensional coordinate information (x, d, f) Information (x, d, f) is generated as parallax histogram information. The three-dimensional coordinate information (x, d, f) distributed in the XY two-dimensional coordinate system is called a frequency U map.

  More specifically, the U map generation unit 140 calculates a parallax value frequency distribution for each column region of parallax image data obtained by dividing an image into a plurality of left and right directions. Information indicating the parallax value frequency distribution is parallax histogram information. Specifically, when disparity image data having a disparity value distribution is input, the U map generation unit 140 calculates a disparity value frequency distribution that is a distribution of the number of pieces of data of each disparity value for each column. Are output as parallax histogram information. The two-dimensional orthogonal coordinate system in which the parallax value frequency distribution information of each column obtained in this manner is taken with the x-axis position on the parallax image (the left-right position of the captured image) on the x-axis and the parallax value on the y-axis. By expressing it above, a frequency U map can be obtained. This frequency U map can also be expressed as an image in which pixels having pixel values corresponding to the frequency f are distributed on the two-dimensional orthogonal coordinate system.

  In the U map generation unit 140 of the present embodiment, the height H from the road surface is within a predetermined height range (for example, 20 cm to 3 m) based on the height of each road surface portion tabulated by the road surface height table calculation unit 139. The U map is created only for the point (x, y, d) of the parallax image at). In this case, an object existing within the predetermined height range from the road surface can be appropriately extracted. For example, the U map may be created only for the point (x, y, d) of the parallax image corresponding to the lower 5/6 image area of the captured image. In this case, the upper one-sixth of the captured image is because, in most cases, the sky is projected and no object that needs to be recognized is projected.

  Further, the point (x, y, d) of each parallax image is associated with the height H from the road surface based on the height of each road surface portion tabulated by the road surface height table calculation unit 139. Thus, not only the three-dimensional coordinate information (x, d, f) related to the frequency but also the three-dimensional coordinate information (x, d, H) can be obtained. This three-dimensional coordinate information (x, d, H) distributed in an XY two-dimensional coordinate system is called a height U map. This height U map can also be expressed as an image in which pixels having pixel values corresponding to the height H are distributed on the two-dimensional orthogonal coordinate system. In the present embodiment, in the present embodiment, the three-dimensional coordinate information (x, d) is obtained using the maximum height Hmax for each parallax value d from the three-dimensional coordinate information (x, d, f) constituting the frequency U map. , Hmax) is extracted, and a height U map in which the three-dimensional coordinate information (x, d, Hmax) is distributed in an XY two-dimensional coordinate system is used.

FIG. 12 is an image example schematically showing an example of a reference image imaged by the imaging unit 110A.
FIG. 13 is a U map corresponding to the image example of FIG. 12, and is a frequency U map expressed as an image in which pixels having pixel values corresponding to the frequency f are distributed on a two-dimensional orthogonal coordinate system.
FIG. 14 is a U map corresponding to the image example of FIG. 12, and is represented as an image in which pixels having pixel values corresponding to the maximum height Hmax in the real space for each parallax value are distributed on a two-dimensional orthogonal coordinate system. This is a height U map.

  In the image example shown in FIG. 12, guard rails exist on both the left and right sides of the road surface, and there are one preceding vehicle and one oncoming vehicle as other vehicles. At this time, in the frequency U map, as shown in FIG. 13, the high-frequency points corresponding to the left and right guard rails are distributed in a substantially straight line extending upward from the left and right ends toward the center. On the other hand, high-frequency points corresponding to other vehicles are distributed between the left and right guard rails in a line segment extending substantially parallel to the X-axis direction. In addition, in a situation where the side portions of these vehicles are projected in addition to the rear portion of the preceding vehicle or the front portion of the oncoming vehicle, parallax occurs in the image area where the same other vehicle is projected. In such a case, as shown in FIG. 13, a high-frequency point corresponding to another vehicle is a state in which a line segment extending in parallel with the substantially X-axis direction and a line segment inclined with respect to the approximately X-axis direction are connected. The distribution of.

  In addition, as shown in FIG. 14, the height U map shows the same image shape as the frequency U map shown in FIG. 13, but the height corresponding to the left and right guard rails is relatively high in real space. The pixel value is low because it is low, and the pixel value is high because the height corresponding to the other vehicle is relatively high in real space.

Next, the real U map generation unit 141 will be described.
The real U map generation unit 141 converts the X axis in the frequency U map and the height U map generated by the U map generation unit 140 from the x direction position on the parallax pixel data to the x direction position in the real space. Is. That is, the x-direction position on the parallax pixel data is closer to the vanishing point on the parallax pixel data as the distance increases (the upper side of the image) even if the x-direction position in the real space is the same. As for the height U map, those obtained by converting the x-direction position on the parallax pixel data into the x-direction position in the real space are referred to as a real frequency U map and a real height U map.

FIG. 15 is a real frequency U map corresponding to the frequency U map shown in FIG.
In the real frequency U map shown in FIG. 15, the Y axis is a unit of thinning parallax obtained by converting the parallax d in the frequency U map at a thinning rate according to the distance. Since an object at a short distance is projected largely, the resolution with respect to the distance of the calculated parallax value is high, so that the parallax value can be greatly thinned out. In the real frequency U map shown in FIG. 15, the left and right guard rails are distributed in a line segment extending in the vertical direction, and the other vehicles are distributed in a substantially rectangular state. The distribution shape is close to the actual shape when the front region is viewed from directly above. The real U map generation unit 141 generates a real height U map in the same manner.

Next, the isolated region detection unit 142 will be described.
The isolated area detection unit 142 is a process for selecting, for each object, an image area corresponding to each object existing in the imaging area. In the present embodiment, using the real frequency U map, an image region that is close in the left-right direction of the image and that has a close parallax value having a frequency f equal to or greater than a predetermined value is detected ( The object is selected as a candidate area (object candidate area) of the detection target image area that projects the object.

FIG. 16 is a flowchart showing the flow of processing performed by the isolated region detection unit 142.
The isolated region detection unit 142 first performs smoothing processing of the real frequency U map from the information of the real frequency U map generated by the real U map generation unit 141. Since the disparity value has dispersion due to calculation errors and the like, and the disparity value is not calculated for all pixels, the actual real frequency U map is different from the schematic diagram shown in FIG. Is included. Therefore, processing for smoothing the real frequency U map is performed in order to remove noise and to make it easier to detect the detection target (object). In the smoothing process, a smoothing filter (for example, a simple average of 3 × 3 pixels) is applied to the frequency value f in the same manner as the smoothing of the image. Thereby, the frequency f of the point on the real frequency U map considered to be noise decreases, and the frequency f becomes a group higher than the surroundings at the point of the detection target (object). As a result, it becomes easy to detect an isolated region in the subsequent processing.

  Next, a process for binarizing the information of the real frequency U map smoothed in this way is performed. Since each detection object (object) has a difference in height, shape, contrast difference with the background, etc., isolated regions corresponding to each detection object each have a large frequency value f or a small one. There is also. Therefore, there is a possibility that an isolated region that cannot be detected properly by binarization using a single threshold value may occur. In order to prevent this, in the present embodiment, an isolated region is detected using a plurality of binarization thresholds.

  Specifically, first, the binarization threshold is set to the smallest value (S32), and the binarization process of the real frequency U map is performed (S33). As a result of this binarization processing, a region having a frequency f higher than the binarization threshold is “1”, and a region having a frequency equal to or lower than the binarization threshold is “0”. In this binarization processing, points having a value of “1” (coordinates where the frequency value f is higher than the binarization threshold) are labeled based on their connectivity (S34), and one region with the same label is labeled as one. Detect as an isolated region. In the labeling method, for example, when the peripheral coordinates located around the target coordinate are already labeled, the same label as the label of the peripheral coordinates is assigned. If different labels are attached to the surrounding coordinates, the label having the smallest value among the attention coordinates and the surrounding coordinates is assigned.

  When the isolated region is detected in this way, the size determination is performed for one or more detected isolated regions (S35). Since the detection objects include pedestrians and large vehicles, it is determined whether or not the width of the isolated region is within the size range. For the isolated region determined to exceed the size by this size determination (Yes in S35), a new binarization threshold value obtained by incrementing the binarization threshold value by 1 is set (S32) Is binarized again (S33), labeling is performed (S34), and a smaller isolated region is detected. By repeating such a process, an isolated region having a desired size can be detected.

  If an isolated region having a desired size is detected in this way, then a peripheral portion removal process is performed (S36). For a distant object, the accuracy of road surface detection is poor, and the parallax value of the road surface part is also reflected in the real frequency U map, and the object and the road surface may be detected as one isolated region. In this case, since there is a possibility that the left end portion, the right end portion, the upper end portion, or the lower end portion on the real frequency U map is a road surface, the peripheral portion that satisfies a predetermined removal condition is removed in the peripheral portion removal process. If a removed portion is present in the peripheral portion removal processing (Yes in S37), the isolated region after the peripheral portion removal processing is labeled again (S34), and the isolated region is reset.

  Next, the horizontal direction separation process is performed on the isolated region that has undergone the peripheral portion removal process according to the size (width, height, distance) (S38). For example, when objects (a car and a motorcycle, a car and a pedestrian, a car, etc.) are side by side and close to each other, the smoothing process of the real frequency U map causes one part corresponding to a plurality of objects. It may be detected as an isolated region. Further, due to the influence of the parallax interpolation processing of the parallax image data, parallax between different objects may be connected, and this may cause a portion corresponding to a plurality of objects to be detected as one isolated region. The horizontal separation process is a process for detecting such a case and separating one isolated region corresponding to a plurality of objects into isolated regions for each object. When one isolated region is separated by this horizontal separation process (Yes in S40), each isolated region after separation is labeled again (S34), and the isolated region is reset.

  In addition, the vertical direction separation process is performed on the isolated region that has undergone the peripheral part removal process according to its size (width, height, distance) (S38) (S41). For example, when a plurality of preceding vehicles are traveling far away, the dispersion of the parallax values on the parallax image data is likely to increase, and the areas corresponding to the preceding vehicles positioned above and below on the real frequency U map are connected, It may be detected as one isolated region. The vertical direction separation process is a process of detecting such a case and separating one isolated region corresponding to a plurality of objects into isolated regions for each object. When one isolated area is separated by this vertical direction separation processing (Yes in S42), each isolated area after separation is labeled again (S34), and the isolated area is reset.

  Furthermore, in the present embodiment, a height separation process is performed on the isolated area after the peripheral portion removal process (S43). Even when the vertical separation process and the horizontal separation process described above are performed, there are cases where a part corresponding to a plurality of objects is detected and left as one isolated region. As such a case, for example, a case where the host vehicle tries to pass through a side where another vehicle with a low vehicle height such as a passenger car and a high wall or building are located close to each other. In the present embodiment, such a case is detected, and a height separation process is performed in order to separate one isolated region corresponding to a plurality of objects into isolated regions for each object. If one isolated region is separated by this height separation processing (Yes in S44), the isolated regions after the separation are labeled again (S34), and the isolated regions are reset. Details of the height separation process will be described later.

About each isolated area obtained in this way, the detection target imaged in the isolated area calculated from the width (the length in the X-axis direction on the U map) and the minimum parallax value d in the isolated area Using the distance z between the (object) and the host vehicle, the width W of the object shown in the image area corresponding to the isolated area can be calculated from the following equation (3).
W = z × (xmax−xmin) / f (3)
An isolated area in which the width W of the object is within a predetermined range is determined as an object candidate area (S45).

Next, the corresponding region detection unit 143 for parallax images will be described.
For the isolated area determined as the object candidate area by the isolated area detection unit 142, as shown in FIG. 17, when a rectangular area inscribed by the isolated area is set, the width of the rectangular area (on the real frequency U map) The length in the X-axis direction corresponds to the width of the detection target (object) corresponding to the isolated region. Further, the height of the set rectangular area corresponds to the depth (length in the traveling direction of the host vehicle) of the detection target (object) corresponding to the isolated area. On the other hand, the height of the detection target (object) corresponding to each isolated area is obtained from the real height U map generated by the real U map generation unit 141. In order to obtain an accurate height of the corresponding object, the corresponding region detection unit 143 of the parallax image detects a corresponding region on the parallax image corresponding to the isolated region.

FIG. 18 is an explanatory diagram schematically showing a parallax image corresponding to the real frequency U map shown in FIG.
The corresponding region detection unit 143 of the parallax image determines a range (detection width) in which the width of the isolated region, that is, the coordinate in the X-axis direction is from xmin to xmax based on the information on the isolated region output from the isolated region detection unit 142. . Further, the corresponding region detection unit 143 of the parallax image has the height on the parallax image corresponding to the isolated region, that is, the Y-axis direction coordinate ymin (maximum parallax) based on the information of the isolated region output from the isolated region detection unit 142. A range (detection height) from ymax (y coordinate of the road surface corresponding to the maximum parallax dmax) to ymax (y coordinate of the point corresponding to the maximum height from the road surface for dmax) is determined. The parallax image is scanned for the detection width and the detection height determined in this manner, and the height of the rectangular area on the real frequency U map set in the isolated area, that is, the Y-axis direction coordinates of the real frequency U map Pixels having a parallax value in the range of (parallax value) from dmin to dmax are extracted as candidate pixels.

  In the candidate pixel group extracted in this way, a horizontal line in which there are more than a predetermined ratio of candidate pixels in the parallax image X-axis direction with respect to the detection width is determined as an object candidate line. Next, when scanning is performed in the vertical direction and there are other object candidate lines with a predetermined density or more around the object candidate line of interest, the object candidate line of interest is determined as an object line. To do.

  The object region extraction unit 144 determines the lowermost end and the uppermost end of the object line group obtained by searching in the range of the detection width and the detection height corresponding to each isolated region, and the circumscribed rectangle defined thereby is As shown in FIG. 19, it is determined as an object area on the parallax image.

Next, the object type classification unit 145 will be described.
From the height (yomax-yomin) of the object area extracted by the object area extraction unit 144, the detection object (object) displayed in the image area corresponding to the object area is expressed by the following equation (4). The actual height Ho can be calculated. However, “zo” is the distance between the object corresponding to the object area calculated from the minimum parallax value d in the object area and the host vehicle, and “f” is the focal length of the camera (yomax−yomin). The value converted to the same unit as.
Ho = zo × (yomax−yomin) / f (4)

Similarly, from the width (xomax−xomin) of the object area extracted by the object area extraction unit 144, the detection target object (object) displayed in the image area corresponding to the object area is calculated from the following equation (5). The actual width Wo of can be calculated.
Wo = zo × (xomax−xomin) / f (5)

In addition, the depth Do of the detection target (object) projected in the image area corresponding to the object area is expressed by the following equation from the maximum parallax value dmax and the minimum parallax value dmin in the isolated area corresponding to the object area. It can be calculated from (6).
Do = BF × (1 / (dmin−offset) −1 / (dmax−offset)) (6)

  The object type classification unit 145 classifies the object type from the information on the height, width, and depth of the object corresponding to the object area that can be calculated in this way. The table shown in FIG. 20 shows an example of table data for classifying object types. According to this, it becomes possible to distinguish and recognize whether the detection target (object) existing in front of the host vehicle is a pedestrian, a bicycle, a small car, a truck, or the like.

Next, the three-dimensional position determining unit 146 will be described.
The distance to the object corresponding to the detected object area and the distance on the image between the image center of the parallax image and the center of the object area on the parallax image are also grasped, so that the three-dimensional position of the object is determined. Can do. When the center coordinates of the object region on the parallax image are (region_centerX, region_centerY) and the image center coordinates of the parallax image are (image_centerX, image_centerY), the horizontal direction relative to the imaging units 110A and 110B of the detection target (object) is detected. The direction position and the height direction position can be calculated from the following equations (7) and (8).
Xo = Z × (region_centerX-image_centerX) / f (7)
Yo = Z × (region_centerY−image_centerY) / f (8)

Next, the object matching unit 147 will be described.
The object matching unit 147 performs comparison matching with the data list in the object data list 135 where S = 0 for each object region detected from the captured image captured in a certain imaging frame. The object area matched by this comparison matching is classified as “Matched” as the matched object in the object data list 135, that is, the same object as the object detected in the past (for example, the immediately preceding imaging frame). The On the other hand, an object area that has not been matched with an object in the object data list 135 is classified as “NewObject” as a newly detected object. Various pieces of information regarding the object area classified as “NewObject” are newly added to the object data list 135.

  After the comparison matching is executed in this way, the objects in the object data list 135 that are not matched are classified as “Missing”. For objects classified as “Missing”, the number of undetected frames is counted up, and an object whose number of undetected frames has reached a specified value is deleted from the object data list 135.

Next, the height separation process in the isolated region detection unit 142 will be described.
As shown in FIG. 21, the other vehicle (a vehicle traveling in an adjacent lane, a parked vehicle, etc.) existing in a position shifted laterally from the front of the host vehicle traveling direction (the central portion in the left-right direction of the captured image) and the wall are close to each other. In this case, the other vehicle and the wall are detected as one object candidate area, and the other vehicle and / or the wall may not be detected appropriately. Even in such a case, the height separation process executes a process of separating and detecting the other vehicle and the wall into separate object candidate areas.

FIG. 22 is an explanatory diagram showing a real frequency U map corresponding to the image example of FIG.
As shown in the image example of FIG. 21, when the host vehicle tries to pass the road where other vehicles are parked near the wall existing on the side of the road, the image areas for displaying these walls and the other vehicles are shown in FIG. On the real frequency U map shown in FIG. 22, the parallax value d, which is close in the left-right direction of the captured image and has a frequency f equal to or higher than a predetermined value, is close. In such a case, the parallax value of the part from the wall of the other vehicle is connected to the parallax value corresponding to the wall due to the dispersion, and even the vertical separation process and the horizontal separation process in the isolated region detection process described above may be performed. As shown in FIG. 22, the other vehicle and the wall are detected as one isolated region.

FIG. 23 is an explanatory diagram showing a real height U map corresponding to the image example of FIG.
The real height U map shown in FIG. 23 shows the distribution of the real space height on the parallax image. In this embodiment, the real space height is limited to each isolated region shown in FIG. The distribution of depth is shown. In the present embodiment, the distribution of the maximum value Hmax of the real space height corresponding to each parallax value among the distribution of the real space height is shown. In the real height U map shown in FIG. 23, an isolated region corresponding to a wall having a relatively low real space height and a pixel value of an isolated region corresponding to another vehicle having a relatively low real space height. Has a high pixel value. In the height separation process of the present embodiment, a real height U map is used, and an isolated area is separated by a difference in real space height so that other vehicles and walls can be detected as separate isolated areas. ing.

FIG. 24 is a flowchart showing the flow of height separation processing in the present embodiment.
In the height separation process (S43) in the present embodiment, first, it is determined whether or not the position in the left-right direction of the isolated area to be processed is within a predetermined height separation execution area (S43-1). In the present embodiment, since the front (left and right central portion) is outside the height separation execution area, if the left and right position of the isolated area is the front (Yes in S43-1), the height separation process is terminated as it is. To do. If a plurality of objects are close to each other and are side by side, it is possible to image a gap that exists between the two objects. For this reason, the image areas in which these objects are respectively displayed are image areas in which the value d of the parallax value having a frequency f equal to or higher than a predetermined value is close, but there is a certain gap in the left-right direction of the captured image. Even if the above-described parallax interpolation processing is performed, the parallax interpolation process does not become continuous but does not become adjacent. Therefore, in this case, individual isolated regions are detected by the above-described binarization processing and labeling processing, and there is no need to separate by height separation processing. Therefore, in the present embodiment, the front side is outside the height separation execution region.

  Specifically, in the real frequency U map shown in FIG. 22, an isolated region that does not overlap in the region (front) surrounded by two broken lines extending in the vertical direction in the drawing is within the height separation execution region. (No in S43-1), the height separation process is continued. The width of the front area is, for example, 1 m in real space. Note that the height separation execution area set on the left and right ends of the two broken lines is set to an area width of, for example, about 10 m as a distance in real space, and the entire isolated area is included in the height separation execution area. It may be determined that the isolated region in which is contained is in the height separation execution region.

  For the isolated area determined to be within the height separation execution area, next, a search range for searching for the maximum height Hmax is set according to the horizontal position of each isolated area (S43-2). . In this search range setting, an area assumed to correspond to a wall is set as a wall search range, and an area assumed to correspond to a vehicle is set as a vehicle search range. Specifically, for the isolated area located on the left side of FIG. 25, an area with a diagonal line rising to the right in the figure is set as the wall search range. The width of this search range is, for example, ½ of the isolated area width, and the height is set to the same height as the isolated area. In addition, a region with a downward slanting diagonal line is set as a vehicle search range. The width of this search range is, for example, 8 pixels, and the height is set to the same height as the isolated area. The same applies to the isolated region located on the right side of FIG.

  Once the wall and vehicle search range is set in this manner, first, the real height U map information is used to search for a region where the maximum height Hmax exceeds the wall height threshold with respect to the wall search range. It is determined whether or not a wall exists (S43-3). Specifically, as shown in FIG. 26, first, for each row region (Y-axis position on the real height U map) in the wall search range, the maximum height Hmax corresponding to each column region is used for the wall. Compared to the height threshold, the number of parallax values exceeding the wall height threshold for each row region is counted (S43-3-1). Then, the number of row areas in which the count value of each row area is larger than a predetermined wall count threshold is counted (S43-3-2). When the number of row areas counted in this way is larger than a predetermined wall area number threshold, a high wall flag is set as a high wall exists in this search range (S43-3-3).

  Further, using the information of the real height U map, a region where the maximum height Hmax exceeds the vehicle height threshold with respect to the vehicle search range is searched to determine whether or not a wall exists (S43-). 4). Specifically, as shown in FIG. 27, first, for each row region (Y-axis position on the real height U map) in the vehicle search range, the maximum height Hmax corresponding to each column region is set for the vehicle. Compared with the height threshold value, the number of parallax values exceeding the vehicle height threshold value for each row region is counted (S43-4-1). Then, the number of row regions in which the count value of each row region is larger than a predetermined vehicle count threshold is counted (S43-4-2). When the number of row areas counted in this way is larger than a predetermined vehicle area number threshold, a high vehicle flag is set on the assumption that a high vehicle exists in this search range (S43-4-3).

  When the high wall flag is set for the wall search area and the high vehicle flag is not set for the vehicle search area (Yes in S43-5), the maximum height Hmax exceeds the wall height threshold. A new label assignment process for assigning a new label to the disparity value is executed (S43-7). In this way, the isolated area portion to which the new label is assigned is assigned a different label from the original isolated area. As a result, the original isolated area has a parallax value whose maximum height Hmax exceeds the wall height threshold. It is separated into a part corresponding to, and other parts.

  Thus, when an isolated region is separated by the height separation process (Yes in S44), a finer isolated region may be detected for the isolated region after separation. Then, labeling is performed again (S34), and the isolated area is reset.

What was demonstrated above is an example, and there exists an effect peculiar for every following aspect.
(Aspect A)
Based on the parallax image information generating unit such as the parallax image generating unit 132 that generates parallax image information such as parallax image data for the imaging region in front of the host vehicle, and the parallax image information generated by the parallax image information generating unit, A U map generation unit 140 and a real U map generation unit 141 that generate parallax histogram information such as real frequency U map information indicating the frequency distribution of the parallax values d in each row region obtained by dividing the captured image into a plurality of left and right directions. Based on the disparity histogram information generating means such as the disparity histogram information generated by the disparity histogram information generating means, and the disparity value of the disparity value that is close in the left-right direction of the captured image and has a frequency f greater than or equal to a predetermined value. An image area where the value d is close to a detection object image area such as an object area that displays a detection object (object) such as another vehicle. Are selected as candidate regions (isolated regions), and an isolated region detecting unit 142 that specifies a detection target image region from the selected candidate regions according to a predetermined specific condition, a corresponding region detecting unit 143 for parallax images, and an object region extracting unit 144, and an object detection device such as the image processing unit 120 and the image analysis unit 102 that detect the detection target object existing in the imaging region. Based on the parallax image information, the U map generation unit 140, the real U map generation unit 141, etc. that generate height distribution information such as real height U map information indicating the distribution of the real space height on the parallax image. A height distribution information generating unit, wherein the detection object image area specifying unit uses the height distribution information to determine whether the real space height is a predetermined height line. Characterized by selecting an image area satisfying as the candidate region.
According to this, as a result of the selection condition for selecting the candidate area including a condition such that the height in the real space is within a predetermined range, a plurality of objects that project a plurality of objects having significantly different heights in the real space are displayed. Image regions can be selected as separate candidate regions. Therefore, it is possible to suppress a situation in which a plurality of image areas that project a plurality of adjacent objects are erroneously selected as one candidate area.

(Aspect B)
In the aspect A, the detection object image area specifying unit temporarily selects a candidate area (isolated area) based on the parallax histogram information, and then uses the height distribution information to select the temporarily selected candidate area. According to the predetermined height condition, it is separated and selected as individual candidate areas, and the height distribution information generating means is a height distribution indicating a distribution of real space heights within the temporarily selected candidate areas. It is characterized by generating information.
According to this, since the height distribution information is generated only within the temporarily selected candidate region, the processing related to the generation of the height distribution information can be reduced.

(Aspect C)
In the aspect B, the height distribution information generated by the height distribution information generation unit indicates a distribution of the maximum height in real space for each parallax value for each row region in the temporarily selected candidate region. It is characterized by being.
According to this, processing related to generation of height distribution information can be further reduced.

(Aspect D)
In any one of the aspects A to C, the predetermined height condition is different depending on a difference in position on the captured image.
Since it is possible to assume in advance a situation in which a plurality of image areas that project a plurality of objects that are significantly different in height in real space are selected as one candidate area, a captured image of each image area in which each object is projected The upper position can be inferred. According to this aspect, for each region estimated in this way, a plurality of image regions that project each object are set by setting a height condition suitable for each object that is expected to exist in that region. Can be appropriately separated into individual candidate regions.

(Aspect E)
In the aspect D, the predetermined height condition includes a condition that a real space height at the position exceeds a threshold value at the position with respect to a threshold value that differs depending on a position on the captured image. And
According to this, the process which isolate | separates the several image area | region which projects each object into an individual candidate area | region can be performed more simply.

(Aspect F)
In any one of the aspects A to E, the detection object image area specifying unit includes a height selection image such as a height separation execution area that is a part of the captured image set in accordance with a predetermined area setting condition. For the area, the candidate area is selected using the height distribution information, but the candidate area is not used for the other part (front area or the like) of the captured image. It is characterized by sorting.
A situation in which a plurality of image regions that project a plurality of objects having significantly different heights in the real space are selected as one candidate region may occur only at a specific position on the captured image. Specifically, as described above, such a situation is unlikely to occur in the central portion (front region) in the left-right direction of the captured image. According to this aspect, since the candidate area is not selected using the height distribution information for a place where such a situation is unlikely to occur, the process can be simplified.

(Aspect G)
An object detection device that detects a detection target such as another vehicle existing in the imaging region based on parallax image information about the imaging region, with the moving direction front of the moving body such as the host vehicle 100 as an imaging region; In a mobile device control system including mobile device control means for controlling a predetermined device mounted on a mobile based on the detection result of the object detection device, the modes A to F are used as the object detection device. The object detection apparatus according to any one of the aspects is used.
According to this, since it is possible to suppress a situation in which a plurality of image areas that project a plurality of adjacent objects are erroneously selected as one candidate area, it is possible to control a predetermined device mounted on the moving body with higher accuracy. It becomes possible.

(Aspect H)
Parallax image information generating means for generating parallax image information for the imaging area, and in each row area obtained by dividing the captured image into a plurality of left and right directions based on the parallax image information generated by the parallax image information generating means Based on the parallax histogram information generated by the parallax histogram information generated by the parallax histogram information generating unit, and the parallax histogram information generated by the parallax histogram information generating unit. In addition, an image area with a parallax value having a frequency equal to or higher than a predetermined value is selected as a candidate area of a detection target image area that displays the detection target, and the predetermined candidate area is selected from the selected candidate areas. Detection object image area specifying means for specifying a detection object image area, and an object detection for detecting a detection object existing in the imaging area. An object detection program that causes a computer of an apparatus to function as the parallax image information generation unit, the parallax histogram information generation unit, and the detection target image region identification unit, and based on the parallax image information, The computer is caused to function as height distribution information generation means for generating height distribution information indicating the distribution of real space height on the image, and the detection object image area specifying means is the height distribution information. And selecting an image area whose real space height satisfies a predetermined height condition as the candidate area.
According to this, as a result of the selection condition for selecting the candidate area including a condition such that the height in the real space is within a predetermined range, a plurality of objects that project a plurality of objects having significantly different heights in the real space are displayed. Image regions can be selected as separate candidate regions. Therefore, it is possible to suppress a situation in which a plurality of image areas that project a plurality of adjacent objects are erroneously selected as one candidate area.

  This program can be distributed or obtained in a state of being recorded on a recording medium such as a CD-ROM. It is also possible to distribute and obtain signals by placing this program and distributing or receiving signals transmitted by a predetermined transmission device via transmission media such as public telephone lines, dedicated lines, and other communication networks. Is possible. At the time of distribution, it is sufficient that at least a part of the computer program is transmitted in the transmission medium. That is, it is not necessary for all data constituting the computer program to exist on the transmission medium at one time. The signal carrying the program is a computer data signal embodied on a predetermined carrier wave including the computer program. Further, the transmission method for transmitting a computer program from a predetermined transmission device includes a case where data constituting the program is transmitted continuously and a case where it is transmitted intermittently.

DESCRIPTION OF SYMBOLS 100 Own vehicle 101 Imaging unit 102 Image analysis unit 103 Display monitor 106 Vehicle traveling control unit 110A, 110B Imaging part 113A, 113B Image sensor 115A, 115B Image sensor controller 120 Image processing board 131 Parallelization image generation part 132 Parallax image generation part 133 Object tracking unit 134 Object selection unit 135 Object data list 136 Parallax interpolation unit 137 V map generation unit 138 Road surface shape detection unit 139 Table calculation unit 140 U map generation unit 141 Real U map generation unit 142 Isolated region detection unit 143 Corresponding region detection unit 144 Object region extraction unit 145 Object type classification unit 146 3D position determination unit 147 Object matching unit

JP 2014-225220 A

Claims (8)

Parallax image information generating means for generating parallax image information about the imaging region;
Based on the parallax image information generated by the parallax image information generation unit, parallax histogram information for generating parallax histogram information indicating the frequency distribution of parallax values in each row region obtained by dividing the captured image into a plurality of left and right directions Generating means;
Based on the parallax histogram information generated by the parallax histogram information generating unit, an image region that is close in the left-right direction of the captured image and that has a value of a parallax value having a frequency equal to or higher than a predetermined value is A detection object image area specifying means for selecting the detection object image area as a candidate area of the detection object image area for projecting the detection object and specifying the detection object image area according to a predetermined specific condition from the selected candidate areas;
In the object detection apparatus for detecting a detection target existing in the imaging region,
Based on the parallax image information, having height distribution information generating means for generating height distribution information indicating the distribution of real space height on the parallax image;
The detection object image area specifying means uses the height distribution information to select, as the candidate area, an image area whose real space height satisfies a predetermined height condition. The object detection apparatus according to claim 1,
The detection object image region specifying unit temporarily selects candidate regions based on the parallax histogram information, and then uses the height distribution information to separate the temporarily selected candidate regions according to the predetermined height condition. Are selected as individual candidate areas,
The height distribution information generation unit generates height distribution information indicating a distribution of real space heights in the temporarily selected candidate area. The object detection device according to claim 2,
The height distribution information generated by the height distribution information generation means indicates the distribution of the maximum height in real space for each parallax value for each row area in the temporarily selected candidate area. A featured object detection device. The object detection device according to any one of claims 1 to 3,
The object detection apparatus according to claim 1, wherein the predetermined height condition varies depending on a position on the captured image. The object detection device according to claim 4,
The predetermined height condition includes a condition that a real space height at the position exceeds a threshold value at the position with respect to a threshold value that differs depending on a difference in position on the captured image. . In the object detection device according to any one of claims 1 to 5,
The detection object image area specifying unit selects the candidate area using the height distribution information for a height selection image area that is a part of the captured image set according to a predetermined area setting condition. However, the candidate area is selected without using the height distribution information for other parts of the captured image. An object detection device that detects a detection target existing in the imaging region based on parallax image information about the imaging region, with the moving direction front of the moving body as an imaging region;
In a mobile device control system comprising mobile device control means for controlling a predetermined device mounted on the mobile based on the detection result of the object detection device,
A mobile device control system using the object detection device according to any one of claims 1 to 6 as the object detection device. Parallax image information generating means for generating parallax image information for the imaging area, and in each row area obtained by dividing the captured image into a plurality of left and right directions based on the parallax image information generated by the parallax image information generating means Based on the parallax histogram information generated by the parallax histogram information generated by the parallax histogram information generating unit, and the parallax histogram information generated by the parallax histogram information generating unit. In addition, an image area with a parallax value having a frequency equal to or higher than a predetermined value is selected as a candidate area of a detection target image area that displays the detection target, and the predetermined candidate area is selected from the selected candidate areas. Detection object image area specifying means for specifying a detection object image area, and an object detection for detecting a detection object existing in the imaging area. The computer device, the parallax image information generating means, wherein the disparity histogram information generating means, and, a object detection program to function as the detection object image area specifying means,
Based on the parallax image information, the computer is caused to function as height distribution information generation means for generating height distribution information indicating a distribution of real space height on the parallax image.
The detection object image area specifying unit selects an image area whose real space height satisfies a predetermined height condition as the candidate area using the height distribution information.