JP7091686B2 - 3D object recognition device, image pickup device and vehicle - Google Patents

Description

The present invention relates to a three-dimensional object recognition device, an image pickup device, and a vehicle.

In order to recognize the surrounding environment, there is a method of measuring the position of the target object and the distance from the sensor by using an image, a distance measuring sensor, or the like. For example, positioning technology using a stereo camera has been actively developed in recent years for in-vehicle use and the like (for example, Patent Document 1 and Patent Document 2).

In Patent Document 1, after an object is detected for each parallax block based on a parallax image obtained from a stereo camera, an object is specified for each detected object based on an image feature and a time feature.

Further, in Patent Document 2, an object is detected for each type of detection target based on two-dimensional image features such as a luminance image and a parallax image.

However, for example, in Patent Document 1, there is a problem that highly accurate detection is difficult in a scene where many structures other than the detection target are present. That is, since it is a method of discriminating the detection target based on the detection result, the discrimination accuracy depends on the detection accuracy. Therefore, when the objects are adjacent to each other, there is a problem that different objects become one parallax block. Further, in an environment where parallax is difficult to occur, there is a problem that the discrimination performance is deteriorated when there is a possibility that one object is separated and detected.

Further, in Patent Document 2, there is a problem that high-precision detection is difficult when there are various appearances on the image to be detected. For example, when a human is a detection target, the image features differ depending on the various human movements (walking, hip bending, stationary, holding an object, riding in a vehicle) and clothing. Therefore, stable detection was difficult in an environment where movements and clothes were not unified. Further, since the object is detected by the same method from the near field to the distant place, there is a problem that it is difficult to exhibit high accuracy regardless of the distance. In other words, the farther the object is, the smaller it is in the image and the less features it has, so it was difficult to detect a near-field object with a large amount of information and a distant object with a small amount of information with high accuracy using a single method. ..

The present invention has been made in view of the above, and efficiently uses three-dimensional spatial information and two-dimensional brightness image information to detect a non-stationary region whose position changes with time in an image. It is an object of the present invention to provide a three-dimensional object recognition device, an image pickup device, and a vehicle capable of detecting a three-dimensional object with high accuracy and high speed even in a complicated environment.

In order to solve the above-mentioned problems and achieve the object, the present invention includes an acquisition means for acquiring a distance image having distance information for each pixel in a time series and a time-series distance image acquired by the acquisition means. An identification means for identifying a three-dimensional object from the above, a mapping means for mapping a region identified as a three-dimensional object by the identification means to a time-series bird's-eye view map, and a model of the three-dimensional object for a region identified as a three-dimensional object. A discriminating means for discriminating a three-dimensional object included in the distance image in a time series based on the model creating means to be created, the bird's-eye view map mapped by the mapping means, and the model created in advance by the model creating means. The discriminating means is provided with a generation means for generating a plurality of filter images by acting a plurality of different spatial filters on one input image, and a stereoscopic effect on each of the plurality of filter images in advance. A non-stationary region is detected from the input image by calculating the occurrence probability of each part of the input image using a model group including one or more models having parameters expressing the shape of an object. The non-stationary region detecting means includes a region detecting means, and the non-stationary region detecting means sets the occurrence probability of each part of the input image for each model corresponding to the plurality of different spatial filters into one image by combining the corresponding parts. It is a three-dimensional object recognition device that detects a non-stationary region from the input image based on the integrated occurrence probability .

According to the present invention, it is possible to detect a three-dimensional object with high accuracy and high speed even in a complicated environment.

FIG. 1 is a schematic diagram showing a schematic configuration of a vehicle equipped with the three-dimensional object recognition device of the embodiment. FIG. 2 is a hardware block diagram showing an example of the hardware configuration of the three-dimensional object recognition device. FIG. 3 is a diagram showing an example of the positional relationship between the subject and the image pickup lens of each image pickup unit. FIG. 4 is a functional block diagram showing an example of a functional configuration of a three-dimensional object recognition device, and FIG. 4A is a functional block diagram showing an example of a functional configuration that realizes background data processing. FIG. 4B is a functional block diagram showing an example of a functional configuration that realizes a learning model generation process. FIG. 4C is a functional block diagram showing an example of a functional configuration that realizes an object detection process. FIG. 5A is a diagram showing an example of a photographed image captured by the imaging unit. FIG. 5B is a diagram showing an example of a V map corresponding to a captured image. FIG. 6A is a diagram showing an example of a photographed image captured by the imaging unit. FIG. 6B is a diagram showing an example of a bird's-eye view image corresponding to the captured image of FIG. 6A. FIG. 7 is a functional block diagram showing a detailed configuration of the learning model generation unit. FIG. 8 is a flowchart showing an example of the flow of processing performed by the learning unit. FIG. 9 is an explanatory diagram of a model estimation problem that is a premise of processing performed by the learning unit. FIG. 10 is a schematic diagram showing the overall configuration of the algorithm in which the learning model generation unit calculates the integrated score. FIG. 11 is a schematic diagram showing a flow of a modified example of the processing performed by the learning model generation unit. FIG. 12 is a diagram for explaining the labeling process performed by the isolated region detection unit, and FIG. 12 (a) is a diagram showing an example of a state in which a temporary number is assigned. FIG. 12B is a diagram showing an example of a process of replacing the assigned temporary number. FIG. 12 (c) is a diagram showing an example of a state in which the labeling process is completed. FIG. 13 is a diagram showing an example of set values used in the rejection process. FIG. 14 is a diagram showing a specific example of the object detection process, and FIG. 14A is a diagram showing an example of a captured image captured by the imaging unit. FIG. 14 (b) is a diagram showing an example of a bird's-eye view image corresponding to the captured image of FIG. 14 (a). FIG. 14 (c) is a diagram showing an example of the detection result of the unsteady region. FIG. 14D is a diagram showing an example of the detection result of the isolated region.

Hereinafter, the three-dimensional object recognition device 10 of the embodiment will be described in detail with reference to the accompanying drawings. FIG. 1 is a schematic diagram showing a schematic configuration of a vehicle 1 equipped with the three-dimensional object recognition device 10 of the embodiment. As shown in FIG. 1, the three-dimensional object recognition device 10 is provided in a vehicle 1 such as an automobile, which is an example of a moving body. The three-dimensional object recognition device 10 includes an image pickup unit 2 and an analysis unit 3. The three-dimensional object recognition device 10 may be used as the image pickup device 10a. Further, the vehicle 1 includes a control unit 4 and a display unit 5 that operate based on the output of the three-dimensional object recognition device 10.

The image pickup unit 2 is provided near the rearview mirror of the windshield 6 of the vehicle 1 and captures an image of the vehicle 1, for example, the traveling direction. Various data including the image data obtained by the image pickup operation of the image pickup unit 2 are supplied to the analysis unit 3. The analysis unit 3 detects the road surface on which the vehicle 1 is traveling based on various data supplied from the image pickup unit 2, and is a three-dimensional object having a height from the road surface. Analyze three-dimensional objects (objects) such as obstacles.

The control unit 4 gives a warning to the driver of the vehicle 1 via the display unit 5 based on the analysis result of the analysis unit 3. Further, the control unit 4 performs running support such as control of various in-vehicle devices, steering control of the vehicle 1, acceleration control, deceleration control, etc. based on the analysis result.

(Explanation of hardware configuration of 3D object recognition device)
Next, the hardware configuration of the three-dimensional object recognition device 10 (imaging device 10a) will be described with reference to FIG. FIG. 2 is a hardware block diagram showing an example of the hardware configuration of the three-dimensional object recognition device 10. As shown in FIG. 2, the image pickup unit 2 has, for example, a stereo camera configuration including two image pickup units 10A and 10B. That is, the optical axes of the two image pickup units 10A and 10B are arranged so as to be substantially parallel to each other. The two imaging units 10A and 10B have the same configuration. Specifically, the image pickup units 10A and 10B are images composed of image pickup lenses 11A and 11B and, for example, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor in which light receiving elements are arranged in two dimensions. It has sensors 12A and 12B and controllers 13A and 13B for driving the image sensors 12A and 12B for image pickup.

The analysis unit 3 has an FPGA (Field-Programmable Gate Array) 14, a RAM (Random Access Memory) 15, and a ROM (Read Only Memory) 16. Further, the analysis unit 3 has a CPU (Central Processing Unit) 17, a serial interface (serial IF) 18, and a data interface (data IF) 19. The data IF 19s from the FPGA 14 are connected to each other via a data bus line 21 provided inside the analysis unit 3. Further, the image pickup unit 2 and the analysis unit 3 are connected to each other via the data bus line 21 and the serial bus line 20.

The RAM 15 stores a parallax image or the like generated based on the luminance image data supplied from the image pickup unit 2. Various programs such as an operation system and an object detection program are stored in the ROM 16.

The FPGA 14 uses one of the captured images captured by the imaging units 10A and 10B as a reference image and the other as a comparative image. Then, the FPGA 14 calculates the amount of positional deviation between the corresponding image portion on the reference image corresponding to the same point in the imaging region and the corresponding image portion on the comparative image as the parallax value D of the corresponding image portion.

Here, a method of calculating the parallax value D will be described with reference to FIG. FIG. 3 is a diagram showing an example of the positional relationship between the subject 30 and the image pickup lenses 11A and 11B of the image pickup units 10A and 10B. Each image pickup unit 10A and 10B is placed in an XYZ coordinate system in which the direction from the center of the image pickup lens 11A toward the center of the image pickup lens 11B is the X-axis and the optical axis direction of each image pickup unit 10A and 10B is the Z axis. It is assumed that it has been done. Further, it is assumed that the distance b between the image pickup lenses 11A and 11B and the focal length f of the image pickup lenses 11A and 11B are both fixed values. At this time, the amount of deviation of the image formation position Pa of the point P on the subject 30 from the center of the image pickup lens 11A is set to Δ1. Further, the amount of deviation of the image formation position Pb of the point P on the subject 30 from the center of the image pickup lens 11B is set to Δ2. In this case, the FPGA 14 calculates the parallax value D, which is the difference in the imaging position through the imaging lenses 11A and 11B of the imaging units 10A and 10B with respect to the point P on the subject 30, by the equation 1.

D = | Δ1-Δ2 | ... (Equation 1)

The FPGA 14 performs processing that requires real-time performance, such as gamma correction processing and distortion correction processing (parallelization of left and right captured images), on the luminance image data supplied from the image pickup unit 2. Further, the FPGA 14 generates a parallax value D and writes it in the RAM 15 by performing the calculation of the equation 1 using the luminance image data that has been subjected to such processing that requires real-time performance.

The CPU 17 operates based on the operation system stored in the ROM 16 and controls the imaging of the imaging units 10A and 10B. Further, the CPU 17 loads the object detection program from the ROM 16 and executes various processes using the parallax value D written in the RAM 15. Specifically, the CPU 17 obtains CAN (Controller Area Network) information such as vehicle speed, acceleration, steering angle, yaw rate, etc. acquired from each sensor provided in the vehicle 1 via the data IF19 based on the object detection program. refer. Then, the CPU 17 performs recognition processing of a recognition object such as a road surface, a guardrail, a vehicle, and a human being, parallax calculation, calculation of a distance to the recognition target, and the like. In addition, in order to improve the parallax detection accuracy of the recognition object, the vehicle 1 may be stopped during the parallax detection.

Here, the parallax value D can be calculated by the following equation 2 with the distance from each of the image pickup lenses 11A and 11B shown in FIG. 3 to the subject 30 as Zo.

D = (b × f) / Zoo ... (Equation 2)

As can be seen from Equation 2, the distance Zo from each of the imaging lenses 11A and 11B to the subject 30 can be calculated by the following equation 3 using the parallax value D.

Zo = (b × f) / D ... (Equation 3)

The CPU 17 calculates the distance Zo with the subject 30 by using the parallax value D supplied from the image pickup unit 2.

(Explanation of the functional configuration of the three-dimensional object recognition device)
Next, the functional configuration of the three-dimensional object recognition device 10 will be described with reference to FIG. FIG. 4 is a functional block diagram showing an example of the functional configuration of the three-dimensional object recognition device 10.

The three-dimensional object recognition device 10 performs background data processing, learning model generation processing, and object detection processing. In the background data processing, a parallax image D (x, y) (not shown) is generated from the stereo image captured by the image pickup unit 2, and a road surface and an object (stereoscopic) are generated from the generated parallax image D (x, y). It is a process of identifying an object (object) and mapping the identified three-dimensional object to a bird's-eye view image. The learning model generation process is a process of generating a learning model of a three-dimensional object such as a preceding vehicle, a pedestrian, or an obstacle having a height from the road surface in a road environment. The object detection process is a three-dimensional object having a height from the road surface from the stereo images captured by the image pickup unit 2 based on the bird's-eye view image generated by the background data processing and the learning model of the object generated by the learning model generation process. It is a process to detect an object.

The three-dimensional object recognition device 10 executes each of the above-mentioned processes by the cooperation of the FPGA 14, the RAM 15, and the ROM 16. Then, the three-dimensional object recognition device 10 realizes a functional configuration corresponding to each of the above-mentioned processes.

(Explanation of the functional configuration that realizes background data processing)
As shown in FIG. 4A, the background data processing is realized by the stereo image acquisition unit 40, the parallax image generation unit 41, the road surface estimation unit 42, and the bird's-eye view image group generation unit 43.

The stereo image acquisition unit 40 acquires a distance image having distance information for each pixel in time series. More specifically, the stereo image acquisition unit 40 acquires a stereo image captured by the image pickup unit 2. The stereo image acquisition unit 40 is an example of acquisition means.

The parallax image generation unit 41 generates a parallax image D (x, y), which is an example of a distance image having distance information for each pixel, from the stereo image acquired by the stereo image acquisition unit 40. The parallax image D (x, y) is an image having a parallax value D for each pixel.

That is, the parallax image generation unit 41 uses the brightness image data of the imaging unit 10A as the reference image data, the brightness image data of the imaging unit 10B as the comparison image data, and performs the calculation shown in the above equation 1 to perform the reference image data. And the parallax value D of the comparison image data is calculated. Specifically, the parallax image generation unit 41 defines a block composed of a plurality of pixels (for example, 16 pixels × 1 pixel) centered on one pixel of interest for a predetermined “row” of the reference image data. On the other hand, in the same "row" in the comparative image data, a block having the same size as the defined reference image data block is shifted one pixel at a time in the horizontal line direction (X direction). Then, the parallax image generation unit 41 shows the feature amount showing the feature of the pixel value (value stored in the pixel, the brightness value) of the block defined in the reference image data and the feature of the pixel value of each block in the comparison image data. Correlation values indicating the correlation with the feature amount are calculated respectively.

Further, the parallax image generation unit 41 performs a matching process of selecting the block of the comparative image data that has the most correlation with the block of the reference image data among the blocks in the comparative image data based on the calculated correlation value. After that, the amount of positional deviation between the pixel of interest of the block of the reference image data and the corresponding pixel of the block of the comparative image data selected by the matching process is calculated as the parallax value D. A parallax image D (x, y) is obtained by performing such a process of calculating the parallax value D for the entire area of the reference image data or a specific area.

As the feature amount of the block used for the matching process, for example, the value (luminance value) of each pixel in the block can be used. The correlation values include, for example, the value of each pixel in the block of reference image data (brightness value) and the value of each pixel in the block of comparative image data corresponding to these pixels (brightness value). The sum of the absolute values of the differences can be used. In this case, the block with the smallest sum is detected as the most correlated block.

The matching process performed by the parallax image generation unit 41 includes, for example, SSD (Sum of Squared Difference), ZSD (Zero-mean Sum of Squared Difference), SAD (Sum of Absolute Difference), or ZSAD (Zero-mean Sum of Absolute). Methods such as Difference) are well known. When the parallax value D at the sub-pixel level of less than one pixel is required in the matching process, the estimated value is used. As a method for estimating the estimated value, for example, an equiangular straight line method, a quadratic curve method, or the like can be used. However, an error occurs in the estimated parallax value D at the subpixel level. Therefore, a method such as EEC (estimation error correction) that reduces the estimation error may be used.

In the present embodiment, since the parallax value D and the distance value can be treated equivalently, the parallax image D (x, y) is shown as an example of the distance image, but the form of the distance image is not limited to this. For example, a distance image may be generated by fusing the distance information obtained from the millimeter wave radar or the laser radar with the parallax value D generated by the stereo camera.

From the parallax image D (x, y) (distance image) acquired by the acquisition means, the road surface estimation unit 42 obtains a two-dimensional histogram in which the horizontal axis is the parallax value D and the vertical axis is the y coordinate value and the pixel value is the frequency. create. Hereinafter, this two-dimensional histogram is referred to as a "V-Disparity map (V map)". Then, the road surface estimation unit 42 identifies a three-dimensional object from the two-dimensional histogram. The road surface estimation unit 42 is an example of the identification means and the road surface detecting means.

Specifically, in the road surface estimation unit 42, for example, the captured image P1 (x, y) captured by the image pickup unit 2 extends toward the back of the screen as shown in FIG. 5A, and the flat road surface 60 When the electric pole 62 is present on the left side of the vehicle 61 traveling above, the following processing is performed. That is, for the pixels (D, y) having the parallax value D and the y coordinate value in the captured image P1 (x, y), the pixels (D, y) of the corresponding V map V (D, y) are displayed. A frequency (frequency) that counts up one pixel value is given. Then, the road surface estimation unit 42 votes for each pixel (D, y) on the two-dimensional histogram in which the horizontal axis is the parallax value D, the vertical axis is the y coordinate value, and the pixel value is the frequency, so that FIG. 5 ( The V map V (D, y) shown in b) is created.

In the created V map V (D, y), as shown in FIG. 5B, a region representing the road surface 60 is voted as a straight line descending to the right. Then, above the area representing the road surface 60, the area representing the vehicle 61 and the utility pole 62 is voted. The road surface estimation unit 42 specifies a pixel group corresponding to the road surface 60 by specifying a linear pixel group that descends to the right in the V map V (D, y). That is, in the V map V (D, y) shown in FIG. 5B, the parallax value D in the portion below the road surface 60 is not detected. Therefore, the parallax value D corresponding to the region A shown by the diagonal line in FIG. 5B is not counted. In this way, the road surface estimation unit 42 detects the road surface 60 from the time-series V-map V (D, y) acquired by the stereo image acquisition unit 40, and has a height from the road surface 60 and the road surface 60. Identify things.

If parallax values D are detected in a region below the road surface 60 (a portion lower than the road surface 60) due to noise or the like, these parallax values D may not be used in subsequent object detection. In order to detect the height of the object, it is necessary to accurately detect the road surface 60. Therefore, a virtual straight line (a straight line descending to the right in FIG. 5B) corresponding to the road surface 60 detected when the vehicle 1 equipped with the three-dimensional object recognition device 10 is stopped is used to determine from the virtual straight line. V-map V (D, y) (restricted V-map) that maps only pixels (D, y) within a distance, that is, only pixels (D, y) having a predetermined height range from the road surface are used for road surface detection. You may.

Next, the operation of the road surface estimation unit 42 that detects the road surface shape using the generated V-map V (D, y) will be described. The road surface estimation unit 42 detects the road surface 60, which is an example of a reference object that is used as a reference for the height of each object. The road surface estimation unit 42 linearly approximates the position estimated to be the road surface 60 on the V map V (D, y). When the road surface 60 is flat, it is approximated by a single straight line. Further, in the case of the road surface 60 whose slope changes on the way, the V map V (D, y) is divided into a plurality of sections and linear approximation is performed. As a result, even in the case of the road surface 60 whose slope changes on the way, it is possible to perform linear approximation with high accuracy.

Specifically, the road surface estimation unit 42 first detects a road surface candidate point using the V map V (D, y). For the detection of road surface candidate points, the horizontal axis is divided into two, and the detection method of the candidate points is changed in each area. Specifically, the road surface estimation unit 42 detects road surface candidate points by the first candidate point detection method in a short-distance region where the parallax value D is large. Further, the road surface estimation unit 42 detects the road surface candidate point by the second candidate point detection method in a long-distance region where the parallax value D is small.

Here, the reason for changing the detection method of the road surface candidate point between the short-distance region where the parallax value D is large and the long-distance region where the parallax value D is small as described above is as follows. That is, for example, as in the captured image P1 (x, y) shown in FIG. 5A, the area of the road surface 60 is large at a short distance, and the parallax data on the road surface is voted on the V map V (D, y). On the other hand, the area of the road surface 60 becomes small at a long distance, and the frequency of the coordinates representing the road surface 60 becomes relatively small. That is, the frequency value of the points predicted as the road surface 60 on the V map V (D, y) is small at a long distance and large at a short distance. Therefore, if the road surface candidate points are detected with the same criteria, the candidate points of the road surface 60 can be detected at a short distance, but the road surface candidate points at a long distance become difficult to detect, and the road surface detection accuracy at a long distance decreases.

In order to eliminate such a defect, the V map V (D, y) is divided into a region having a large parallax value D and a region having a small parallax value D, and a method and a standard for detecting road surface candidate points are defined in each region. You may change it. This makes it possible to improve the road surface detection accuracy of both short-distance and long-distance.

The bird's-eye view image group generation unit 43 is an example of mapping means, and is a stereo image acquisition unit for a region identified by the road surface estimation unit 42 as a stereoscopic object from the time-series distance images acquired by the stereo image acquisition unit 40. Map to a position different from the installation position of 40, for example, a time-series bird's-eye view map J2 (x, D) looking down on the road surface from directly above. More specifically, the bird's-eye view image group generation unit 43 generates a bird's-eye view image, that is, a two-dimensional histogram in which the actual distance x is taken on the horizontal axis and the parallax value D is taken on the vertical axis, and the frequency d is plotted as the pixel value. The horizontal axis of the two-dimensional histogram is not limited to the actual distance x, and may be the actual pixel position.

The bird's-eye view image group generation unit 43 is a pixel (D,) indicating that the height from the road surface 60 is in a predetermined range, for example, in the range of 20 cm to 3 m, among the points voted for by the V map V (D, y). Select only y). Then, the bird's-eye view image group generation unit 43 maps the selected pixels (D, y) to the bird's-eye view image. The bird's-eye view image generated by the bird's-eye view image group generation unit 43 is input data to the learning model generation unit 44, which will be described later, the unsteady region detection unit 45, and the isolated region detection unit 46.

The bird's-eye view image group generation unit 43 simultaneously creates a bird's-eye view image and a bird's-eye view image of height. This has the actual distance x on the horizontal axis and the parallax value D on the vertical axis, and the maximum height of the parallax value D voted for the pixel (x, D) as the pixel value (in the y direction from the detected road surface 60). It is equivalent to creating a map that records the maximum difference). By making the resolution of the pixels (x, D) the same as the bird's-eye view image, the bird's-eye view image generation process can be easily performed. The created bird's-eye view image is used as input data of the corresponding area detection unit 47, which will be described later. In addition, in order to make it less susceptible to noise or the like when generating a learning model, processing such as smoothing may be performed when generating a bird's-eye view image.

Next, the operation of the bird's-eye view image group generation unit 43 will be specifically described with reference to FIG. FIG. 6A is a diagram showing an example of a photographed image P2 (x, y) captured by the image pickup unit 2. FIG. 6B is a diagram showing an example of a bird's-eye view image J2 (x, D) corresponding to the captured image P2 (x, y) of FIG. 6A. The bird's-eye view image J2 (x, D) is an example of a bird's-eye view map.

In the captured image P2 (x, y) of FIG. 6A, three pedestrians, that is, pedestrians 80a, 80b, 80c stand on the road surface 60 provided with wall-shaped guardrails 70a, 70b on the left and right. It shows the state of being.

The bird's-eye view image group generation unit 43 generates a bird's-eye view image J2 (x, D) from the captured image P2 (x, y). The horizontal axis of the bird's-eye view image J2 (x, D) is the actual distance x. The vertical axis D of the bird's-eye view image J2 (x, D) may be, for example, a parallax value D that has been thinned out using a thinning rate according to the distance. For example, in the case of a long distance of 50 m or more, the bird's-eye view image group generation unit 43 may use the parallax value D that is not thinned out. Further, for example, in the case of a medium distance such as 20 to 50 m, the bird's-eye view image group generation unit 43 may use the parallax value D thinned to 1/2. Further, in the case of a short distance such as 10 to 20 m, the bird's-eye view image group generation unit 43 may use the parallax value D that has been thinned out to 1/3. Further, in the case of a recent distance such as 0 to 10 m, the bird's-eye view image group generation unit 43 may use the parallax value D that has been thinned out to 1/8.

At a distance, the parallax information is small because the three-dimensional object to be recognized is small. Moreover, since the resolution of the distance is large, the thinning process is not performed. On the other hand, in the case of a short distance, since a three-dimensional object is captured in a large size, the parallax information is large and the resolution of the distance is small. Therefore, it is possible to perform a large thinning process. As can be seen from the bird's-eye view image J2 (x, D) of FIG. 6B, the guardrails 70a and 70b are represented by a straight line extending in the vertical direction (depth direction) along the road. Further, the pedestrians 80a, 80b, and 80c are projected at positions corresponding to the actual existing positions.

The bird's-eye view image group generation unit 43 periodically executes the above-mentioned bird's-eye view image J2 (x, D) generation process at a predetermined time interval t0. As a result, bird's-eye view images J2 (x, D, t), J2 (x, D, t + t0), J2 (x, D, t + 2t0), ... Are generated. The bird's-eye view image J2 (x, D, t) represents the bird's-eye view image J2 (x, D) generated at time t. Hereinafter, for the sake of simplicity, the bird's-eye view image J2 (x, D, t) is simply referred to as J2 (x, D). The generated series of bird's-eye view images 51 is stored in, for example, a RAM 15 or a storage device such as an HDD (Hard Disk Drive) (not shown in FIG. 2). This bird's-eye view image group 51 is used in the learning model generation process described later.

(Explanation of the functional configuration that realizes the learning model generation process)
Returning to FIG. 4, the description of the functional configuration of the three-dimensional object recognition device 10 will be continued. As shown in FIG. 4B, the learning model generation process performed by the three-dimensional object recognition device 10 is realized by the learning model generation unit 44. That is, the learning model generation unit 44 creates the learning model 52 of the three-dimensional object for the region identified as the three-dimensional object by the road surface estimation unit 42. The learning model generation unit 44 is an example of a model creation means.

Next, the functional configuration of the learning model generation unit 44 will be described with reference to FIG. 7. FIG. 7 is a functional block diagram showing a detailed configuration of the learning model generation unit 44.

As shown in FIG. 7, the learning model generation unit 44 includes a captured image acquisition unit 90, a generation unit 91 which is an example of generation means, a learning unit 92, and a calculation unit 93 which is an example of unsteady region detection means. The discriminator unit 94, which is an example of the discriminating means, is provided.

In the learning model generation process, a three-dimensional object (object) detected by the three-dimensional object recognition device 10, specifically, a preceding vehicle having a height from the road surface 60 existing on the road surface 60 and whose position changes with time. This is a process for learning in advance so-called unsteady regions such as pedestrians and obstacles. The learning model generation process may be executed independently of the object detection process described later (offline processing), or the learning model generation process may be executed while executing the object detection process (online process). .. That is, the object may be learned in advance, or the object may be learned while the object is detected. When the learning model 52 is generated by online processing, the three-dimensional object recognition device 10 uses the learning model generation unit 44 as the model generation device 44a to operate the learning model generation process at the same time as the object detection process.

The captured image acquisition unit 90 acquires the captured image P1 (x, y) (see FIG. 5A) from the image pickup unit 2. The generation unit 91 causes a plurality of different spatial filters Fi (i = 1, 2, ...) To act on one captured image P1 (x, y), and a plurality of filter images Fi (x,) (not shown). y) is generated. In the present embodiment, by acting the spatial filter Fi, a plurality of filter images Fi (x, y) representing the edge strength having a direction corresponding to each spatial filter Fi in the captured image P1 (x, y). ) Is generated. The number of spatial filters Fi to act on is, for example, 12, but is not limited to this. When 12 spatial filters Fi (i = 1 to 12) are used, for example, 3 scales × 4 edge directions (0 degree direction, 45 degree direction, 90 degree direction, 135 degree direction) and the like. As the filter coefficient, for example, a set of four coefficients represented by the following equations 4 to 7 is used. Each coefficient of the spatial filter Fi (i = 1 to 4) shown in the equations 4 to 7 constitutes a Prewitt filter which is a typical edge detection filter, but other types of filters may be used. good.

Further, as the scale, in addition to the one having the same magnification, for example, the one obtained by reducing the captured image P1 (x, y) to 1/4 times or 1/8 times is used. Then, by applying the above-mentioned spatial filter Fi to the image of each scale and returning the result of applying the spatial filter Fi to the same magnification, a total of 12 filter images Fi (x, y) can be obtained. .. Further, in the present embodiment, the captured image P1 (x, y) in a state where the spatial filter Fi is not acted upon (exceptionally, it may be considered as one aspect of the filter image Fi (x, y)) is added. Then, the learning model 52 is generated using a total of 13 filter images Fi (x, y).

The learning unit 92 learns the object based on a plurality of images of the object prepared in advance. More specifically, the learning unit 92 learns a model group including one or more learning models 52 having parameters expressing the target shape for each spatial filter Fi. That is, in the present embodiment, the learning unit 92 learns 13 filter images Fi (x, y) and 13 model groups corresponding to one-to-one. Then, in the present embodiment, as parameters, the average (pixel average value) μ (x, y) of each pixel value of the plurality of captured images P1 (x, y) and the dispersion (pixel dispersion value) σ ² (x, y) and are adopted. However, the types of parameters are not limited to this.

Hereinafter, an object learning method performed by the learning unit 92 will be described. Here, a case where a learning model 52, which is a model group including a plurality of (K) models, is trained for any one spatial filter Fi will be described as an example. As a model, it is assumed that the pixel values of the captured image P1 (x, y) have a normal distribution, and it is assumed that there are a plurality of (K) such models. Then, it is assumed that the captured image P1 (x, y) is an image generated from any one of the plurality of learning models 52. Here, it is unknown from which of the learning model 52 the image generated is observed, and it is a hidden variable. When the learning of the object (estimation of the learning model 52) is completed, the pixel mean value μ (x, y) and the pixel dispersion value σ ² (x, y) for each model are obtained.

Since hidden variables and parameters cannot be determined at the same time, learning is performed here using an EM algorithm that is effective for estimating parameters when there are hidden variables. Hereinafter, the E step and the M step of the EM algorithm will be described.

Object learning starts from the E step. The learning unit 92 has K models for each pixel (x, y) of the input image (n images corresponding to the above-mentioned one spatial filter Fi (filter image Fi (x, y) including an object)). The Z score Z _nk (x, y) for each is calculated. Here, assuming that the _nth filter image Fi (x, y) which is the input image is the input image In (x, y), _k of the pixel (x, y) of the input image In (x, y). The Z score Z _nk (x, y) for the second model, the model k (k = 1, ..., K), is expressed by the following equation 8.

In Equation 8, μ _k (x, y) and σ ² _k (x, y) are parameters of the pixels (x, y) of the model k. More specifically, μ _k (x, y) is the pixel mean value of the pixel (x, y) of the model k, and σ ² _k (x, y) is the pixel of the pixel (x, y) of the model k. It is a distributed value. The Z score Z _nk (x, y) is a value represented by replacing the position of a certain value in the distribution with a standard normal distribution having a mean value of 0 and a standard deviation of 1. In the present embodiment, the Z score Z _nk (x, y) is used as the feature amount, but if the calculation can be performed using the pixel value as a probabilistic model, the feature amount is the Z score Z _nk (x, y). ) Is not limited.

Next, the learning unit 92 obtains the probability _enk that the _nth input image In (x, y) fits into the model k. The probability _enk can be expressed by the following equation 9. In equation 9, the symbol Π is a symbol indicating an infinite product. In the present embodiment, the learning unit 92 substitutes the Z score Z _nk (x, y) into the formula of the standard normal distribution having an average of 0 and a variance of 1, and obtains the probability density for each pixel (x, y). The product of the probability densities for each pixel (x, y) or each region is calculated to obtain the simultaneous probability. Note that X and Y in _Equation 9 are the number of pixels in the horizontal and vertical directions of the input image In (x, y), respectively. Here, the probability _enk is obtained from the distribution of the pixel values over the entire input image In (x, _y ), not for each pixel (x, y). By doing so, it is possible to appropriately obtain the probability _{enk indicating} which learning model 52 corresponds to while looking at the entire input image In (x, y).

Next, the learning unit 92 uses the probability _{en nk to obtain the burden factor γ nk corresponding to} the expected value of which learning model 52 each of the input images In (x, y) is generated from. .. The burden rate γ _nk can be obtained by the following equation 10. In Equation 10, _N represents the total number of input images In (x, y), and K represents the number of models. The above is the contents of the E step.

After the completion of the E step, the learning unit 92 estimates the parameters of each model k in the M step. More specifically, the learning unit 92 obtains the pixel mean value μ _k (x, y) of each pixel (x, y) of the model k weighted by the burden factor γ _nk . In the example of this embodiment, the pixel mean value μ _k (x, y) can be obtained by the following equation 11.

Further, the learning unit 92 obtains the pixel dispersion value σ ² _k (x, y) of each pixel (x, y) of the model k weighted by the burden factor γ _nk . In the example of this embodiment, the pixel dispersion value σ ² _k (x, y) can be obtained by the following equation 12.

_Nk in the formulas 11 and 12 is obtained by the following formula 13.

After the M step is completed, the learning unit 92 returns to the E step and repeats the process until the parameter fluctuation from the previous time becomes equal to or less than the threshold value (until the convergence condition is satisfied). By repeating the E-step process and the M-step process, the model parameters can be estimated with hidden variables. As an example, the initial value may be a random number for μ _k (x, y) and 1 for σ ² _k (x, y), and the operator of the three-dimensional object recognition device 10 may input an image In (x ₎ . If it is clear which model the input image In (x, _y ) should be discriminated from so that the types can be classified while looking at, y), the input image In (x, _y ) is used as the initial value of the model. The pixel value of x, y) may be μ _k (x, y). In this way, the learning unit 92 learns the parameters (μ _k (x, y), σ ² _k (x, y)) of the model k (k = 1, ..., K).

The algorithm (EM algorithm) including the above-mentioned E step and M step is one of the methods for maximum likelihood estimation of the parameters of the probability model in statistics, and the probability model is an unobservable latent variable. This is the method used when it depends. The EM algorithm is a kind of iterative method, and the calculation proceeds by alternately repeating the expected value (expectation, E) step and the maximization (M) step. In the M step, a parameter that maximizes the expected value of the likelihood obtained in the E step is obtained. The parameters determined in the M step are used to determine the distribution of latent variables used in the next E step.

FIG. 8 is a flowchart showing an example of the flow of processing performed by the learning unit 92. Since the specific contents of each step are as described above, the description thereof will be omitted as appropriate. The processing of each step shown in FIG. 8 is performed for the number of spatial filters Fi to be operated, but here, for convenience of explanation, the processing corresponding to one spatial filter Fi will be described. As shown in FIG. 8, the learning unit 92 has K models for each of the plurality of _pixels (x, y) included in the input image In (x, y) corresponding to the target spatial filter Fi. Z score Z _nk (x, y) is calculated (step S11). Next, the learning model generation unit 44 obtains the probability _enk (step S12). Next, the learning unit 92 obtains the burden rate γ _nk (step S13). Next, the learning unit 92 calculates the parameters (μ _k (x, y), σ ² _k (x, y)) of each model k (step S14). The processing of steps S11 to S13 corresponds to the E step, and the processing of the step S14 corresponds to the M step.

Next, the learning model generation unit 44 determines whether or not the fluctuation of the parameters (μ _k (x, y), σ ² _k (x, y)) from the previous time is equal to or less than the threshold value (step S15). ). If the result of step S15 is negative (step S15: No), the process after step S11 is repeated. If the result of step S15 is affirmative (step S15: Yes), the parameter calculated in step S14 (μ _k (x, y), σ ² _k (x, y)) is determined as the final parameter (step S16). ). The parameters (μ _k (x, y), σ ² _k (x, y)) determined as described above are not included in, for example, RAM 15 or FIG. 2 as the learning model 52 (see FIG. 4 (b)). It is stored in a storage device such as the illustrated HDD.

FIG. 9 is an explanatory diagram of a model estimation problem that is a premise of the processing performed by the learning unit 92. The sample image in FIG. 9 is processed from the subject image of the "Image Application Technology Expert Committee Appearance Inspection Algorithm Contest 2014" (Appearance Inspection Algorithm Contest 2014, sponsored by the Precision Engineering Society Image Application Technology Expert Committee). And use it. FIG. 9 is an example in which there are two types of models k (two types of models included in the model group corresponding to any of the spatial filters Fi), that is, the number of models K is two. In FIG. 9, it is assumed that there are two models in which each pixel (x, y) (pixel value) has a normal distribution, and the observable image is an image generated from any model k. Here, it is unknown from which model k the observed image was generated, and it is a hidden variable. When the learning is completed, that is, the estimation of the model k is completed, the pixel average image and the pixel distributed image for each model shown on the left of FIG. 9 are obtained.

As described above, in the learning process of the present embodiment, the pixel mean value μ _k (x, y) and the pixel dispersion value σ ² _k (x, y) for optimizing the burden rate γ _nk are determined and stored. To. 9 (a) maps the pixel mean value μ _k (x, y) and the pixel dispersion value σ ² _k (x, y) determined in step S16 of FIG. 8 for each pixel (x, y). It is a visualization. In this embodiment, the information shown in FIG. 9A is used to estimate a model, which is a hidden variable, from the observed image based on the probability _{enk and the burden rate γ nk .}

The explanation of the function of the learning model generation unit 44 will be continued again. The calculation unit 93 calculates an integrated score, which is a Z score considering all the models, based on the generated plurality of filter images Fi (x, y) and the trained model group.

First, the calculation unit 93 uses a plurality of filter image Fis (x, y) and a plurality of model groups corresponding to one-to-one, for each pixel (x, y) of the plurality of filter images Fi (x, y). In addition, a score indicating a value corresponding to the difference from the corresponding model group (in the example of the present embodiment, the larger the difference from the model group, the higher the value) is calculated. The calculation unit 93 sets the Z score for each pixel (x, y) of the plurality of filter images Fi (x, y) based on the pixel value of the pixel (x, y) and the parameters of the corresponding model group. Calculate Z _nk (x, y).

Hereinafter, using a model group corresponding to any one filter image Fi (x, y), the Z score Z _nk (x, y) of each pixel (x, y) included in the filter image Fi (x, y) is used. The method of calculating y) will be described. Here, a case where K models are included in the model group corresponding to any one filter image Fi (x, y) will be described as an example, but the present invention is not limited to this, and the model group is not limited to this. May contain only one model (the number of models included in the model group is arbitrary).

The calculation unit 93 obtains a Z score Z _nk (x, y) for each model using Equation 8 for each pixel (x, y) included in the above-mentioned filter image Fi (x, y). Further, the calculation unit 93 obtains the probability _enk using the equation 9. Then, the calculation unit 93 uses the following equation 14 for each pixel (x, y) included in the above-mentioned one filter image Fi (x, y), and the amount of deviation from the model, that is, the amount of deviation _Sn ( x, y) is obtained. In a multi-model in which the model group includes a plurality of models, this deviation amount _Sn (x, y) becomes a Z score Z _nk (x, y) based on the occurrence probability of the learned model. In the example of the present embodiment, the calculation unit 93 sets the dissociation amount _Sn (x, y) of each pixel (x, y) of the above-mentioned filter image Fi (x, y) as the final Z score Z _nk . Calculated as (x, y). That is, in the present embodiment, when the model group corresponding to any of the filter images Fi (x, y) includes a plurality of models, the calculation unit 93 includes a plurality of models included in the filter image Fi (x, y). A unit score indicating a value corresponding to the difference between each of the pixels (x, y) of the pixel (x, y) from each model of the pixel (x, y) (Z score Z _nk for each of K models in the example of this embodiment). The final Z score Z _nk (x, y) of the pixel (x, y) is determined based on (x, y)) and the probability enk that the filter image _fits into each model.

As described above, the calculation unit 93 calculates the Z score Z _nk (x, y) for each pixel (x, y) included in each of the plurality of filter images Fi (x, y). In the following description, the Z score of the pixel (x, y) of the m-th filter image Fi (x, y) may be expressed as Z _m (x, y).

Here, since a normal distribution is assumed for the probability of occurrence of each pixel (x, y), this Z score Z _nk (x, y) considers a model learned by the corresponding pixel of the input image. At times, it shows that what σ (σ is the standard deviation) is the probability of occurrence in the standard normal distribution. In the example of this embodiment, the case of using multiple models has been described, but of course, the same thing may be performed assuming a single model. In that case, the equation 11 and the equation are set at the time of learning with k = 1. The model may be obtained in No. 12, and the Z score Z _nk (x, y) may be calculated by the equation 5 at the time of detection. In addition, although it is assumed here that each pixel (x, y) has a normal distribution, in order to improve the accuracy, the model is used as a mixed Gaussian distribution using the EM algorithm as in the case of the above multi-model. It may be changed. The mixed Gaussian distribution is a model represented by a linear superposition of a plurality of Gaussian distributions.

Further, the calculation unit 93 indicates an integrated score showing the result of integrating the Z scores Z _m (x, y) of each of the plurality of pixels (x, y) corresponding to each other over the plurality of filter images Fi (x, y). Calculate Z _total (x, y). That is, it may be considered that the integrated score Z _total (x, y) of each pixel (x, y) of one image in which a plurality of filter images Fi (x, y) are integrated is calculated. In the example of this embodiment, it is assumed that the number of pixels of the 13 filter images Fi (x, y) is the same, and the pixels (x, y) correspond to each other. Further, here, since the Z score Z _m (x, y) is the standard deviation in the standard normal distribution, the calculation unit 93 has pixels (x, y) corresponding to each other over the plurality of filter images Fi (x, y). , Y) For each of the plurality of _pixels (x, _y ), the integrated score Z _total ( x, y) is calculated. More specifically, the calculation unit 93 calculates the occurrence probability P _m (x, y) corresponding to the Z score Z _m (x, y) by the following formula 15, and the integrated score Z _total is calculated by the following formula 16. (X, y) is calculated.

Here, the joint probability was used to integrate the integrated score Z _total (x, y), but in addition to that, the average value of the Z score Z _m (x, y) or the average value of the Z score Z m (x, y) is used as shown in the following equation. A total value or the like may be used.

As a matter of course, it is desirable to use a normal image (an image of a three-dimensional object as a model without interruptions or gradation abnormalities) at the time of learning, but some abnormalities are mixed with the normal image. Even if this is the case, this method can be applied only by slightly reducing the probability of occurrence of an image mixed with anomalies.

The integrated score Z _total (x, y) described above is a value considering all the model groups. This value shows all the spatial filter Fi, that is, all the elements such as various scales and various edge directions, as a grounded probability of occurrence of standard deviation in the standard normal distribution, so-called production process, etc. It agrees with the value of what σ is often used in. Therefore, by determining the threshold value with this integrated score Z _total (x, y), it is not necessary to set an individual threshold value for each spatial filter Fi.

As described above, the learning model generation unit 44 causes a plurality of different spatial filter Fis to act on one captured image P1 (x, y) to generate a plurality of filter images Fi (x, y). Then, the learning unit 92 has a Z score Z _m (x, y) according to the difference from the corresponding model group for each pixel (x, y) included in each of the generated plurality of filter images Fi (x, y). y) is calculated. Then, an integrated score Z _total (x, y) that integrates the Z scores Z _m (x, y) of each of the plurality of pixels (x, y) corresponding to each other over the plurality of filter images Fi (x, y). Is calculated. The calculated integrated score Z _total (x, y) is stored as a learning model 52 in, for example, a RAM 15 or a storage device such as an HDD (not shown in FIG. 2). FIG. 10 is a schematic diagram showing the overall configuration of an algorithm in which the learning model generation unit 44 calculates the integrated score Z _total (x, y), and is a diagram schematically showing the flow of the above-mentioned processing.

The learning model generation unit 44 may use the Wavelet transform that collectively calculates the filter group. In that case, the learning model generation unit 44 executes the process shown in FIG. FIG. 11 is a schematic diagram showing a flow of a modified example of the processing performed by the learning model generation unit 44.

That is, as shown in FIG. 11, the learning model generation unit 44 first performs _Wavelet transform of the input image In (x, y) to create a multi-layered image (one-to-one with the above-mentioned plurality of spatial filters Fi). It is converted into one image including (may be considered as a plurality of corresponding layers) (step S20). That is, the input image In (x, y) becomes one image obtained by extracting the multi-scale and vertical and horizontal edges by the _Wavelet transform. Next, the image is compared with the learning model 52 obtained in advance by learning (step S21), and the Z score Z _m (x, y) of each pixel (x, y) is the same as in the above-described embodiment. Is calculated (step S22). In addition, one model in this case can be considered as one model in which the model for each filter image Fi (x, y) (model for each layer) is integrated. From a different point of view, one model in this case can be considered to include a plurality of filter images Fi (x, y) and a plurality of models having a one-to-one correspondence.

After that, a threshold value process for determining whether or not the learning model 52 applies is performed (step S23). That is, if the Z score Z _m (x, y) is σ or less, a threshold value is set as a criterion for determining whether the learning model 52 is applicable, and the threshold value processing is performed. Then, by returning to the original one image by the inverse Wavelet transform (step S24), the integrated score Z _total (x, y) in which the Z score Z _m (x, y) for each pixel (x, y) is integrated is obtained. Is obtained. As a result, the integrated score Z _total (x, y) of each _pixel (x, y) included in the input image In (x, y) can be obtained as in the above-described embodiment. According to this form, by using the Wavelet transform, the filter group is collectively calculated, so that there is an advantage that the calculation time can be reduced.

The learning model generation unit 44 may learn each of a plurality of types of objects existing in one captured image P1 (x, y). For example, the preceding vehicle and the pedestrian are simultaneously captured in the captured image P1 (x, y). Further, the learning model generation unit 44 may learn each of the three-dimensional objects having a plurality of states existing in one captured image P1 (x, y). For example, there is a case where a plurality of pedestrians in different poses are shown in the captured image P1 (x, y).

(Explanation of functional configuration that realizes object detection processing)
Returning to FIG. 4, the description of the functional configuration of the three-dimensional object recognition device 10 will be continued. The object detection process performed by the three-dimensional object recognition device 10 includes the unsteady region detection unit 45 shown in FIG. 4C and the isolated region in addition to the functional units described in the background data processing (see FIG. 4A). It is realized by the detection unit 46, the corresponding area detection unit 47, the real space information calculation unit 48, the rejection processing unit 49, and the tracking processing unit 50.

The unsteady region detection unit 45 is a non-stationary region whose position changes with time from the stereo image captured by the imaging unit 2, and which has a height above the road surface, such as a preceding vehicle, a pedestrian, or an obstacle. Etc. are detected.

The isolated region detection unit 46 performs an isolated region detection process that outputs a group of pixels forming a non-stationary region as one group. For example, the three-dimensional object recognition device 10 may output two pedestrians as one unsteady region when two adjacent pedestrians enter the image. The isolated region detection unit 46 checks the adjacent state of the pixels on the bird's-eye view image J2 (x, D) with respect to such a detection result, thereby regrouping each isolated region to improve the separation performance of the three-dimensional object. Improve.

In the isolated region detection process, the inside of the detection rectangle in the bird's-eye view image J2 (x, D) obtained by the non-stationary region detection unit 45 is set as a non-stationary region, and the parallax inside the non-stationary region is changed from the connection of adjacent pixels to each isolated region. It is a process of grouping to. As a method for detecting adjacent pixels, a labeling method often used in image processing is used. In this process, an isolated region is detected for pixels having an adjacent relationship in the vicinity of eight. This may be performed for pixels having an adjacent relationship in the vicinity of four. In a scene where three-dimensional objects are in close contact with each other, processing such as improving the separation performance may be performed by using labeling of pixels in the vicinity of four.

FIG. 12 is a diagram illustrating a labeling process performed by the isolated region detection unit 46, and FIG. 12A is a diagram showing an example of a state in which a temporary number is assigned. FIG. 12B is a diagram showing an example of a process of replacing the assigned temporary number. FIG. 12 (c) is a diagram showing an example of a state in which the labeling process is completed. The labeling process is applied to a binary image, that is, an image in which "1" is stored in a pixel showing a three-dimensional object and "0" is stored in a pixel not showing a three-dimensional object.

Although various labeling algorithms have been proposed, generally, the processing is performed in two steps. First, as shown in FIG. 12A, a tentative number is assigned to a pixel indicating a three-dimensional object, that is, a pixel in which "1" is stored, by performing a raster scan on a target image. In the example of FIG. 12A, the corresponding pixel is assigned a temporary number from 1 to 4. Here, the pixel Q of interest represents the pixel currently being focused on. In the example of FIG. 12A, when the pixel Q is a pixel indicating a three-dimensional object, "5" is stored in the pixel of interest Q.

Then, after performing the process shown in FIG. 12A over the entire target image, a process of replacing the assigned temporary number is performed. At this time, if there is a pixel adjacent to the pixel of interest Q indicating a three-dimensional object, the smallest temporary number assigned to the adjacent pixel is assigned to the pixel Q. In the present embodiment, the pixels adjacent to the pixel of interest Q are a total of 8 pixels vertically and horizontally and diagonally above and below the pixel of interest Q (8 adjacent pixels). Further, a total of 4 pixels on the top, bottom, left, and right of the pixel Q of interest may be used as adjacent pixels (4 adjacent pixels).

That is, in the example of FIG. 12B, since "8, 9" is assigned to the 8 pixels adjacent to the pixel of interest Q, the pixel of interest Q is assigned by the process of FIG. 12A. The temporary number "5" is replaced with "8" as shown in FIG. 12 (c). By performing the above processing on the entire image, a number indicating the number of three-dimensional objects is assigned to each three-dimensional object in the image.

Returning to FIG. 4 again, the explanation will be continued. The corresponding area detection unit 47 is a three-dimensional object to be detected from the parallax image D (x, y) based on the position, width, and minimum parallax of the three-dimensional object detected on the bird's-eye view image J2 (x, D). The candidate region, that is, the horizontal range (xmin, xmax) (not shown) of the three-dimensional object in the captured image P1 (x, y) is determined.

Further, the corresponding area detection unit 47 determines the height and position of the three-dimensional object in the parallax image D (x, y). That is, in the parallax image D (x, y), the vertical position ymin (not shown) corresponding to the height from the road surface 60 in the captured image P1 (x, y) that gives the minimum parallax value Dmin (not shown) of the three-dimensional object. In the figure) and the parallax image D (x, y), the vertical position ymax corresponding to the height from the road surface 60 in the captured image P1 (x, y) that gives the maximum parallax value Dmax (not shown) of the three-dimensional object. (Not shown) and.

Then, the corresponding area detection unit 47 scans the inside of the set candidate area of the three-dimensional object in order to detect the accurate position of the three-dimensional object, and the depth of the detected rectangular area is the minimum parallax value Dmin (not shown). ) And a pixel having a parallax value D in the range of the maximum parallax value Dmax (not shown) are extracted as candidate pixels of a three-dimensional object.

The real space information calculation unit 48 determines the relative lateral position and distance to the image pickup unit 2, the width in the real space, and the depth (size information) from the detection result of the corresponding area detection unit 47.

The rejection processing unit 49 performs "size rejection processing" for determining an object based on the size information of the object calculated by the real space information calculation unit 48. If the detection target is clearly determined, the "shape rejection process", which is a rejection process based on the three-dimensional shape of a three-dimensional object using the parallax image D (x, y) obtained by a stereo camera, or the brightness. You may use "brightness feature processing" using the feature of the image.

The rejection processing unit 49 performs a rejection process for selecting a three-dimensional object to be output based on the size on the image and the size on the real space of the three-dimensional object. In the rejection process using the size on the captured image P1 (x, y), since the way of thinking differs depending on the distance to the three-dimensional object, only the lower limit threshold value is provided. For example, the image width of the i-th three-dimensional object is OPWi (not shown), the image height is OPHi (not shown), the lower limit of the width is THW (not shown), and the lower limit of the height is THH (not shown). Then, the rejection processing unit 49 determines that the three-dimensional object having OPWi <THW or OPHi <THH is not the three-dimensional object to be detected, and rejects the three-dimensional object. In the rejection process using the size in the real space, when classifying the detection target, the three-dimensional object having a size other than the three-dimensional object to be detected is rejected.

For example, in the shape rejection process, when the detection target is a person, the rejection may be determined based on the shape feature of the mountain obtained by the person obtained by the stereo camera. If the detection target can be discriminated by color, the rejection may be determined based on the color difference in the detection result.

The rejection processing unit 49 performs a rejection processing for selecting a three-dimensional object to be output based on the detection result of the corresponding area detection unit 47. The rejection processing unit 49 executes size rejection focusing on the size of the three-dimensional object and overlap rejection focusing on the positional relationship between the three-dimensional objects. For example, FIG. 13 is a diagram showing an example of set values used in the rejection process.

The object information shown in FIG. 13 is stored in a storage device such as a RAM 15. As shown in FIG. 13, for example, the type of object having a size of "less than 1100 mm in width, less than 2500 mm in height, and more than 1000 mm in depth" is defined as a bicycle. Similarly, the type of object having a size of "width less than 1100 mm, height less than 2500 mm, depth 1000 mm or less" is defined as a pedestrian. Similarly, the type of object having a size of "width 1100 mm or less, height less than 2500 mm, depth less than 5000 mm" is defined as a vehicle.

The rejection processing unit 49 identifies the type of the three-dimensional object by comparing the size of the three-dimensional object on the captured image P1 (x, y) with the object information shown in FIG. Then, the rejection processing unit 49 rejects the detection result of the size that does not fall within the size range shown in FIG.

The tracking processing unit 50 performs object detection processing on a new bird's-eye view image J2 (x, D) by using the previous object detection result on the bird's-eye view image group 51 obtained in time series. Specifically, a process of tracking an object (three-dimensional object) detected by an image detection process of a past image pickup frame is executed.

Specifically, the object data list 53 showing the information of the objects detected in the past image detection process is stored in, for example, the RAM 15 or a storage device such as an HDD (not shown in FIG. 2). In the object data list 53, for example, in addition to the latest information (latest position, size, distance, relative speed, parallax information) of the detected object data, the object prediction data (where the object is located in the next imaging frame) Information), the amount of object features used by the non-stationary area detection unit 45 and the tracking processing unit 50, the number of frames in which the object was detected, or the number of detected / undetected frames indicating whether the object was continuously detected. Includes tracking accuracy (stability flag), etc., which indicates whether the object should be tracked.

(Explanation of specific examples of object detection processing)
Next, a specific example of the object detection process will be described with reference to FIG. FIG. 14 is a diagram showing a specific example of the object detection process, and FIG. 14 (a) is a diagram showing an example of the captured image P3 (x, y) captured by the image pickup unit 2. 14 (b) is a diagram showing an example of a bird's-eye view image J3 (x, D) corresponding to the captured image P3 (x, y) of FIG. 14 (a). FIG. 14 (c) is a diagram showing an example of the detection result of the unsteady region. FIG. 14D is a diagram showing an example of the detection result of the isolated region.

In the photographed image P3 (x, y) shown in FIG. 14A, wall-shaped guardrails 70a and 70b provided on the left and right sides of the road surface 60 and two pedestrians 80d and 80e are shown. The bird's-eye view image group generation unit 43 (see FIG. 4) maps the V map V (D, y) (not shown in FIG. 14) generated from the captured image P3 (x, y) to the bird's-eye view image J3 (x, D). do.

Next, the unsteady region detection unit 45 (see FIG. 4) detects the unsteady region from the bird's-eye view image J3 (x, D). In the example of FIG. 14C, a region corresponding to the left guardrail 70a, a region corresponding to the right guardrail 70b, and a region corresponding to the pedestrian 80f are detected.

The isolated region detection unit 46 performs isolated region detection processing on the detection result of the unsteady region (FIG. 14 (c)). As a result, as shown in FIG. 14D, the pedestrian 80d and the pedestrian 80e can be detected separately.

In the example of the present embodiment, in the three-dimensional object recognition device 10, the CPU 17 realizes various processes by software, but a part or all of the three-dimensional object recognition device 10 is realized by hardware such as an IC (Integrated Circuit). You may. The object detection program is a file in an installable format or an executable format in a computer device such as a CD-ROM, a flexible disc (FD), a CD-R, a DVD, a Blu-ray disc (registered trademark), or a semiconductor memory. It may be recorded and provided on a readable recording medium. DVD is an abbreviation for "Digital Versatile Disk". Further, the object detection program may be provided in the form of being installed via a network such as the Internet. Further, the object detection program may be provided by incorporating it into a ROM or the like in the device in advance.

As described above, according to the stereoscopic object recognition device 10 according to the present embodiment, the stereo image acquisition unit 40 (acquisition means) has a parallax image D (x, y) (distance image) having distance information for each pixel. ) Is acquired in time series, and the road surface estimation unit 42 (identification means) identifies a stereoscopic object (object) from the time-series parallax image D (x, y) acquired by the stereo image acquisition unit 40. , The bird's-eye view image group generation unit 43 (mapping means) acquires a stereo image of a region identified as a stereoscopic object by the road surface estimation unit 42 from the parallax image D (x, y) acquired by the stereo image acquisition unit 40. Map to the time-series bird's-eye view map J2 (x, D) viewed from a position different from the installation position of the unit 40. Then, the learning model generation unit 44 (model creation means) creates a learning model 52 (model) of the three-dimensional object in advance for the region identified as a three-dimensional object by the stereo image acquisition unit 40, and the discrimination unit 94 (discrimination). The means) is a time-series parallax image D (x) based on the bird's-eye view map J2 (x, D) mapped by the bird's-eye view image group generation unit 43 and the learning model 52 created in advance by the learning model generation unit 44. , Y) is included in the three-dimensional object. Therefore, it is possible to detect a moving object with high accuracy and high speed even in a complicated environment. In particular, the object detection process (offline process) can be performed using the learning model 52 of the three-dimensional object generated in advance. The stereo image acquisition unit 40 needs to stay at the same position until the series of processes described above is completed, that is, from the acquisition of the distance image to the detection of the stereoscopic object.

Further, according to the stereoscopic object recognition device 10 according to the present embodiment, the parallax image D (x, y) is based on the image information captured by the image pickup unit 2 (stereo camera), and is also a road surface estimation unit. The 42 (identification means) further includes a road surface estimation unit 42 (road surface detection means) that detects the road surface 60 from the time-series parallax image D (x, y) acquired by the stereo image acquisition unit 40 (acquisition means). In preparation, the bird's-eye view image group generation unit 43 (mapping means) sets the parallax value D corresponding to the distance to the stereoscopic object existing at a position higher than the road surface 60 detected by the road surface estimation unit 42 on the bird's-eye view map J2 (x, Map to D). Therefore, after detecting the road surface 60, by limiting the processing range to only the region having a height from the road surface 60, a three-dimensional object is not affected by the reflection of the road surface 60 or the white line drawn on the road surface 60. Can be reliably detected.

Further, according to the three-dimensional object recognition device 10 according to the present embodiment, in the bird's-eye view map J2 (x, D), the horizontal axis is the actual distance x and the vertical axis is the parallax value D, and the pixel value is the position of the pixel. It is a two-dimensional histogram which mapped the occurrence frequency of the parallax value D in. Since the parallax value D is an amount corresponding to the actual distance, therefore, both the vertical axis and the horizontal axis correspond to the actual distance, so that a distant three-dimensional object can be reliably detected.

Further, according to the three-dimensional object recognition device 10 according to the present embodiment, in the discrimination unit 94 (discrimination means), the generation unit 91 (generation means) with respect to one input image In (x, _y ). For a plurality of filter images Fi (x, y) generated by applying a plurality of different spatial filter Fis, the calculation unit 93 (unsteady region detecting means) is applied to each of the filter images Fi (x, y). On the other hand, the input image In (x, _y ) of each model corresponding to the plurality of spatial filters Fi is used by using a model group including one or more learning models 52 having parameters for expressing the shape of the three-dimensional object in advance. The unsteady region is detected based on the result of integrating the occurrence probability P _m (x, y) of each part into one image. Therefore, it is possible to detect a three-dimensional object that matches the learning model 52 by a unified method regardless of the type of the detection target.

Further, according to the three-dimensional object recognition device 10 according to the present embodiment, the calculation unit 93 (unsteady region detecting means) has a probability of occurrence P _m (x, y ₎ of each part of the input image In (x, y). ), The unsteady region is detected based on the integrated score Z _total (x, y). Therefore, when detecting the unsteady region, it is not necessary to set an individual threshold value for each spatial filter Fi.

Further, according to the three-dimensional object recognition device 10 according to the present embodiment, the calculation unit 93 (unsteady region detecting means) has a probability of occurrence P _m (x, y ₎ of each part of the input image In (x, y). ) Is replaced with a standard normal distribution having a mean value of 0 and a standard deviation of 1 as to where a certain value is located in the distribution, based on the mean value of the Z score Z _m (x, y). Detect non-stationary regions. Therefore, it is possible to detect a three-dimensional object that matches the learning model 52 with a uniform standard for various input _{images In} (x, y).

Further, according to the three-dimensional object recognition device 10 according to the present embodiment, the generation unit 91 (generation means) has multiple resolutions and includes a plurality of spatial filters Fi for calculating the edge direction. Therefore, it is possible to reliably detect a three-dimensional object that exists on the road surface 60 and has a luminance difference with the road surface 60.

Further, according to the three-dimensional object recognition device 10 according to the present embodiment, the calculation unit 93 (unsteady region detecting means) is included in the learning model 52 for each of the plurality of learning models 52 expressing the plurality of three-dimensional objects. The pixel mean value μ _k (x, y) and the pixel dispersion value σ ² _k (x, y), which are the feature quantities of a plurality of regions, are calculated, and the input image _{In (} x, y) is a plurality of learning models. The input image In (x, y ₎ is input using a plurality of learning models 52 in which the feature amount is optimized based on the calculated probability _enk by calculating the probability _enk indicating which of 52 is applicable. ) Is compared with a plurality of learning models 52. Therefore, it is possible to relax the constraint conditions regarding the shape and posture of the three-dimensional object to be detected.

Further, according to the three-dimensional object recognition device 10 according to the present embodiment, the calculation unit 93 (unsteady region detecting means) has a probability of occurrence P _m (x, y ₎ of each part of the input image In (x, y). ) Is calculated using a model based on a normal distribution. Therefore, the integrated score Z _total (x, y) calculated based on the calculated probability of occurrence P _m (x, y) agrees with the value of what σ is often used in so-called production processes. .. Therefore, it is necessary to set an individual threshold value for each spatial filter Fi by determining a threshold value with this integrated score Z _total (x, y) when determining whether the three-dimensional object matches the model. Is gone.

Further, according to the stereoscopic object recognition device 10 according to the present embodiment, the stereo image acquisition unit 40 (acquisition means) produces a disparity image D (x, y) (distance image) having distance information for each pixel in a time series. The road surface estimation unit 42 (identification means) identifies a stereoscopic object from the time-series disparity images D (x, y) acquired by the stereo image acquisition unit 40, and the bird's-eye view image group generation unit 43. (Mapping means) installs the stereo image acquisition unit 40 in a region identified as a stereoscopic object by the road surface estimation unit 42 from the time-series parallax image D (x, y) acquired by the stereo image acquisition unit 40. Map to the time-series bird's-eye view map J2 (x, D) viewed from a position different from the position. Then, the learning model generation unit 44 (model creation means) creates a learning model 52 of the three-dimensional object for the region identified as a three-dimensional object by the stereo image acquisition unit 40, and the discrimination unit 94 (discrimination means) determines. The parallax image D (x, y) is included by using the learning model 52 of the three-dimensional object created by the learning model generation unit 44 at any time based on the parallax image D (x, y) acquired by the stereo image acquisition unit 40 in time series. Discriminate a three-dimensional object. Therefore, it is possible to simultaneously execute the learning model generation process and the object detection process (online process) while capturing the captured image P1 (x, y).

Further, in the image pickup apparatus 10a according to the present embodiment, the image pickup unit 2 (imaging means) captures a stereo image, and the stereo image acquisition unit 40 (acquisition means) is imaged by the image pickup unit 2, and then the pixels. The parallax image D (x, y) (distance image) having distance information for each is acquired in time series, and the road surface estimation unit 42 (identification means) acquires the parallax image D (x, y) (distance image) in time series acquired by the stereo image acquisition unit 40. A three-dimensional object is identified from (x, y), and the bird's-eye view image group generation unit 43 (mapping means) selects the three-dimensional object from the time-series disparity image D (x, y) acquired by the stereo image acquisition unit 40. The region identified as a three-dimensional object by the road surface estimation unit 42 is mapped to the time-series bird's-eye view map J2 (x, D) viewed from a position different from the installation position of the stereo image acquisition unit 40. Then, the learning model generation unit 44 (model creation means) creates a learning model 52 of the three-dimensional object in advance for the region identified as a three-dimensional object by the stereo image acquisition unit 40, and the discrimination unit 94 (discrimination means) , Time-series parallax image D (x, y) based on the bird's-eye view map J2 (x, D) mapped by the bird's-eye view image group generation unit 43 and the learning model 52 created in advance by the learning model generation unit 44. Determines the three-dimensional object contained in. Therefore, it is possible to detect a three-dimensional object with high accuracy and high speed even in a complicated environment.

Further, the vehicle 1 according to the present embodiment includes a three-dimensional object recognition device 10 or an image pickup device 10a. Therefore, it is possible to detect a three-dimensional object on the road surface 60 with high accuracy and high speed while traveling. In particular, by stopping the vehicle 1 during parallax detection, the parallax detection accuracy of a three-dimensional object can be improved. If the series of processes described above can be executed in real time (for example, at a substantially video rate), it is possible to detect a three-dimensional object even while the vehicle 1 is traveling.

Although the embodiments of the present invention have been described above, the above-described embodiments are presented as examples and are not intended to limit the scope of the present invention. This novel embodiment can be implemented in various other embodiments. In addition, various omissions, replacements, and changes can be made without departing from the gist of the invention. Further, this embodiment is included in the scope and gist of the invention, and is also included in the scope of the invention described in the claims and the equivalent scope thereof.

1 vehicle
2 Imaging unit (imaging means)
10 3D object recognition device 10a Imaging device
40 Stereo image acquisition unit (acquisition means)
42 Road surface estimation unit (identification means, road surface detection means)
43 Bird's-eye view image group generator (mapping means)
44 Learning model generator (model creation means)
44a model generator
45 Unsteady region detection unit (unsteady region detection means)
51 Bird's-eye view image group
52 Learning model
53 Object data list
60 road surface
91 Generation unit (generation means)
93 Calculation unit (unsteady region detection means)
94 Discrimination unit (discrimination means)
D Parallax value
_enk probability
Fi spatial filter
Number of K models
k model D (x, y) parallax image (distance image)
Fi (x, y) Filter image In (x, y) _Input image J2 (x, D), J3 (x, D), J2 (x, D, t) Bird's-eye view image (bird's-eye view map)
P1 (x, y), P2 (x, y), P3 (x, y) Captured image P _m (x, y) Occurrence probability _Sn (x, y) Deviation amount V (D, y) V map Z _nk (X, y), Z _m (x, y) Z score Z _total (x, y) Integrated score (x, y), (D, y), (x, D) Pixels

Japanese Unexamined Patent Publication No. 2013-003787 Japanese Unexamined Patent Publication No. 2013-210908

Claims (10)

An acquisition means for acquiring a distance image having distance information for each pixel in chronological order,
An identification means for identifying a three-dimensional object from the time-series distance images acquired by the acquisition means,
A mapping means that maps the area identified as a three-dimensional object by the identification means to a time-series bird's-eye view map, and
For the area identified as a three-dimensional object, a model creation means for creating a model of the three-dimensional object, and
A discriminating means for discriminating a three-dimensional object included in the distance image in a time series based on a bird's-eye view map mapped by the mapping means and a model created in advance by the model creating means.
Equipped with
The discrimination means is
A generation means for generating a plurality of filter images by operating a plurality of different spatial filters on one input image.
For each of the plurality of filter images, the generation probability of each part of the input image is calculated by using a model group including one or more models having parameters for expressing the shape of a three-dimensional object in advance. A non-stationary region detection means that detects a non-stationary region from the input image,
In preparation for
The unsteady region detecting means said, based on the probability of occurrence of each part of the input image for each model corresponding to the plurality of different spatial filters, integrated the corresponding parts into one image. Detecting unsteady regions in the input image,
A three-dimensional object recognition device characterized by this. The distance image is based on the image information captured by the stereo camera and is also based on the image information.
The identification means further includes a road surface detecting means for detecting a road surface from the time-series distance images acquired by the acquiring means.
The mapping means maps a parallax value corresponding to a distance to a three-dimensional object existing at a position higher than the road surface detected by the road surface detecting means to the bird's-eye view map.
The three-dimensional object recognition device according to claim 1. The bird's-eye view map is
The horizontal axis is the amount corresponding to the actual distance,
The vertical axis is the amount corresponding to the parallax value.
It is a two-dimensional histogram in which the frequency of occurrence of the parallax value at the position of the pixel is mapped to the pixels constituting the bird's-eye view map.
The three-dimensional object recognition device according to claim 2. The unsteady region detecting means is
The occurrence probability integrated into the one image is calculated based on the joint probability of the occurrence probability of each part of the input image.
The three-dimensional object recognition device according to any one of claims 1 to 3, wherein the three-dimensional object recognition device is characterized. The unsteady region detecting means is
The probability of occurrence of each part of the input image is based on the mean value of the Z score, which is expressed by replacing the position of a certain value in the distribution with a standard normal distribution having a mean value of 0 and a standard deviation of 1. Calculate the probability of occurrence integrated into the above one image,
The three-dimensional object recognition device according to any one of claims 1 to 4, wherein the three-dimensional object recognition device is characterized. The generation means is
It has multiple resolutions and has multiple spatial filters to calculate the edge direction.
The three-dimensional object recognition device according to any one of claims 1 to 5 , wherein the three-dimensional object recognition device is characterized. The unsteady region detecting means is
For each of a plurality of models expressing a plurality of three-dimensional objects, the features of a plurality of regions in the model are calculated and the features are calculated.
Calculate the probability that the input image fits into any of the multiple models.
Using the plurality of models whose features are optimized based on the probability, the input image is compared with the plurality of models.
The three-dimensional object recognition device according to any one of claims 1 to 6 , wherein the three-dimensional object recognition device is characterized. The unsteady region detecting means is
The probability of occurrence of each part of the input image is calculated using a model based on a normal distribution.
The three-dimensional object recognition device according to any one of claims 1 to 7 , wherein the three-dimensional object recognition device is characterized. An imaging means that captures stereo images in chronological order,
An acquisition means for acquiring a distance image having distance information for each pixel from the stereo image captured by the imaging means in chronological order.
An identification means for identifying a three-dimensional object from the time-series distance images acquired by the acquisition means,
A mapping means that maps the area identified as a three-dimensional object by the identification means to a time-series bird's-eye view map, and
For the area identified as a three-dimensional object, a model creation means for creating a model of the three-dimensional object, and
A discriminating means for discriminating a three-dimensional object included in the distance image in a time series based on a bird's-eye view map mapped by the mapping means and a model created in advance by the model creating means.
Equipped with
The discrimination means is
A generation means for generating a plurality of filter images by operating a plurality of different spatial filters on one input image.
For each of the plurality of filter images, the generation probability of each part of the input image is calculated by using a model group including one or more models having parameters for expressing the shape of a three-dimensional object in advance. A non-stationary region detection means that detects a non-stationary region from the input image,
In preparation for
The unsteady region detecting means said, based on the probability of occurrence of each part of the input image for each model corresponding to the plurality of different spatial filters, integrated the corresponding parts into one image. Detecting unsteady regions in the input image,
An imaging device characterized by this . A vehicle provided with the three-dimensional object recognition device according to any one of claims 1 to 8 or the image pickup device according to claim 9 .

Images