WO2020054429A1

WO2020054429A1 - Image processing device

Info

Publication number: WO2020054429A1
Application number: PCT/JP2019/033762
Authority: WO
Inventors: 裕介内田; 稲田　圭介; 野中　進一
Original assignee: 日立オートモティブシステムズ株式会社
Priority date: 2018-09-10
Filing date: 2019-08-28
Publication date: 2020-03-19
Also published as: CN112513572B; JP2020041891A; CN112513572A; JP7066580B2

Abstract

This image processing device is used to compute parallax on the basis of a stereo camera image captured by means of a stereo camera, and is provided with: a condition setting unit for setting conditions for determining a calculation target pixel in the stereo camera image; and a parallax computing unit which, on the basis of the conditions set by the condition setting unit, determines the calculation target pixel within the stereo camera image, and uses information relating to the calculation target pixel to compute the parallax.

Description

Image processing device

The present invention relates to an image processing device.

There is known a position measuring device that measures a distance to an object existing in a camera field of view based on a principle of triangulation based on a stereo image captured by a stereo camera having a pair of cameras. The principle of triangulation is to calculate the distance from a camera to an object using the displacement (parallax) of the image of the same object captured by the left and right cameras. Derivation of parallax is realized by specifying where an image of an object on one image exists on the other image.

導出 Various methods have been proposed for deriving parallax. In the classical technique, block matching is known in which a region having a plurality of pixels in one image is searched for a region having the lowest degree of dissimilarity in the other image. In addition, a method called SGM (Semi-Global Matching) has been proposed as a method to reduce mismatching and improve accuracy by including not only local information in the image but also information of the entire image in the matching process. ing. While such a technique can be expected to have high accuracy, it has a problem that the amount of calculation increases as compared with the classical technique.

Regarding the suppression of the amount of calculation in calculating parallax using a stereo camera, for example, a technique described in Patent Document 1 is known. Patent Literature 1 discloses an obstacle measurement method that measures a distance to an obstacle by parallax calculation using a stereo image. The method measures the parallax in a high-resolution image based on distance information obtained from a low-resolution image. An obstacle measurement method is disclosed in which a sampling interval of a pixel to be subjected to a correlation operation is set to be large at a short distance and small at a long distance.

Japanese Patent Application Publication No. 2008-298533

According to the method of Patent Document 1, by setting an appropriate sampling interval, high-accuracy distance information can be obtained only in a necessary area, unnecessary calculation can be suppressed, and high-accuracy distance information can be obtained only in a necessary area. It is possible to reduce the operation time and the circuit scale of the device. However, since it is necessary to calculate parallax from each of the low-resolution image and the high-resolution image, the overall calculation amount cannot be reduced much, and there is room for further improvement in suppressing the calculation amount.

An image processing apparatus according to the present invention calculates a parallax based on a stereo camera image captured by a stereo camera, and a condition setting unit that sets a condition for determining a calculation target pixel in the stereo camera image. A parallax calculation unit that determines the calculation target pixel in the stereo camera image based on a condition set by the condition setting unit, and calculates the parallax using information on the calculation target pixel. .

According to the present invention, the amount of calculation required for calculating parallax can be suppressed.

1 is a block diagram illustrating a basic configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. 4 is a flowchart illustrating an algorithm of parallax calculation in the image processing apparatus according to the first embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration of an image processing apparatus according to a second embodiment of the present invention. 9 is a flowchart illustrating an algorithm of parallax calculation in the image processing device according to the second embodiment of the present invention. FIG. 6 is a block diagram illustrating a configuration of an image processing apparatus according to a third embodiment of the present invention. FIG. 14 is a diagram showing an example of determining the number of scan lines in the image processing apparatus according to the third embodiment of the present invention. FIG. 9 is a block diagram illustrating a configuration of an image processing apparatus according to a fourth embodiment of the present invention. FIG. 14 is a block diagram illustrating a configuration of an image processing apparatus according to a fifth embodiment of the present invention. Explanatory drawing of pixel thinning used for smoothing calculation in an image processing device according to a fifth embodiment of the present invention Explanatory diagram of pixel arrangement used for smoothing calculation in an image processing device according to a fifth embodiment of the present invention. FIG. 14 is a block diagram illustrating a configuration of an image processing apparatus according to a sixth embodiment of the present invention.

(Basic configuration)
First, the basic configuration of the embodiment of the present invention will be described. FIG. 1 is a block diagram illustrating a basic configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 10 illustrated in FIG. 1 is connected to the camera 20 and the processor 30, and includes a condition setting unit 101 and a parallax calculation unit 102. The image processing device 10 is mounted on a vehicle, for example, and is used for detecting and recognizing an object such as another vehicle or an obstacle existing around the vehicle.

The camera 20 is a stereo camera installed at predetermined intervals on the left and right sides, captures a pair of left and right stereo camera images, and outputs the captured data to the image processing apparatus 10. The image processing apparatus 10 calculates parallax in a stereo camera image using the parallax calculation unit 102 based on the shooting data input from the camera 20. At this time, the parallax calculation unit 102 determines a pixel to be calculated in the stereo camera image based on the condition set by the condition setting unit 101, and uses information on the pixel to be calculated, for example, using a calculation method called SGM Is calculated by the following. The calculation of the parallax may be performed using a calculation method other than the SGM.

結果 The calculation result of the parallax by the parallax calculation unit 102 is output from the image processing device 10 to the processor 30. The processor 30 performs processing such as object detection and object recognition using the information on the parallax input from the image processing apparatus 10.

Next, each embodiment of the present invention using the above basic configuration will be described.

(First embodiment)
FIG. 2 is a block diagram illustrating a configuration of the image processing apparatus according to the first embodiment of the present invention. As shown in FIG. 2, the image processing apparatus 10 according to the present embodiment includes an FPGA 11 and a memory 12. The FPGA 11 is an arithmetic processing device configured by combining a large number of logic circuits, and has, as its functions, the condition setting unit 101 and the parallax calculating unit 102 described with reference to FIG. Note that the condition setting unit 101 and the parallax calculation unit 102 may be implemented in the image processing apparatus 10 by using something other than the FPGA 11, for example, another logic circuit such as an ASIC, software executed by a CPU, or the like. The memory 12 is a storage device configured using a readable and writable recording medium such as a RAM, an HDD, and a flash memory, and has a stereo camera captured image storage unit 103 and a parallax image storage unit 104 as its functions.

The image data representing the stereo camera image input from the camera 20 to the image processing apparatus 10 is stored in the stereo camera image storage unit 103 in the memory 12 on a frame basis. The parallax calculating unit 102 in the FPGA 11 reads the pixel data of the stereo camera image of each frame from the stereo camera captured image storage unit 103, and performs parallax calculation using the read data, thereby obtaining a parallax image representing parallax in pixel units. Is generated for each frame. The parallax image generated by the parallax calculation unit 102 is output to the processor 30 and stored in the parallax image storage unit 104 in the memory 102.

The parallax calculation unit 102 performs smoothing calculation and dissimilarity calculation in the calculation of parallax using a stereo camera image. Smoothing calculation refers to smoothing (averaging) the pixel value of each pixel in a stereo camera image together with the pixel values of a plurality of surrounding pixels, thereby smoothing fluctuations in pixel values between pixels. This is the operation processing. The calculation of dissimilarity is the likelihood that the object shown in one of the stereo camera images and the object shown in the other image are not the same thing, that is, the dissimilarity that indicates how far apart they are. This is the required arithmetic processing.

In the present embodiment, the condition setting unit 101 has the line number determination unit 111. The number-of-lines determining unit 111 reads a parallax image generated by the parallax calculation unit 102 from the stereo camera image of the previous frame and stored in the parallax image storage unit 104, and is executed by the parallax calculation unit 102 based on the parallax image. Determine the number of scan lines used in the smoothing process. That is, the line number determination unit 111 of the present embodiment determines the number of scan lines for scanning the stereo camera image of the current frame based on the parallax calculated by the parallax calculation unit 102 in the past. Then, the parallax calculating unit 102 is notified of the determined number of scan lines. The condition setting unit 101 in the present embodiment performs the condition setting for the parallax calculation unit 102 by notifying the number of scan lines determined by the line number determination unit 111 in this manner.

When the number of scan lines is notified from the line number determination unit 111, the parallax calculation unit 102 determines the number of pixels notified for each pixel with respect to the stereo camera image of the current frame read from the stereo camera captured image storage unit 103. Set the scan line. Then, smoothing calculation is performed using pixels of the stereo camera image on each set scan line as calculation target pixels. Thereafter, the dissimilarity is calculated for each pixel on the stereo camera image after the smoothing calculation, and the parallax of each pixel is calculated based on the calculation result. The parallax calculation unit 102 outputs the parallax calculation result obtained in pixel units thus obtained to the processor 30 as a parallax image, and stores the parallax image in the parallax image storage unit 104.

FIG. 3 is a flowchart showing an algorithm for calculating parallax in the image processing apparatus according to the first embodiment of the present invention. The parallax calculation shown in the flowchart of FIG. 3 is performed in the FPGA 11 when the stereo camera image newly captured by the camera 20 is stored in the stereo camera captured image storage unit 103 as the stereo camera image of the current frame.

In step S10, the parallax calculation unit 102 reads the stereo camera image of the current frame from the stereo camera captured image storage unit 103, and sequentially selects the pixels from the pixels at the screen edge.

In step S20, the line number determination unit 111 reads out the parallax of the pixel selected in step S10 from the parallax image storage unit 104 in the parallax image calculated by the parallax calculation unit 102 in the previous frame, and refers to the read parallax.

In step S30, the number-of-lines determining unit 111 determines whether the parallax referred to in step S20 is larger than a predetermined threshold dth. When the parallax is larger than the threshold value dth, the process proceeds to step S40, and when the parallax is equal to or smaller than the threshold value dth, the process proceeds to step S50.

In step S40, the line number determination unit 111 determines that the pixel selected in step S10 corresponds to a short-distance image, and sets a small number of scan lines for the pixel. In the present embodiment, for example, the number of scan lines is determined to be four.

In step S50, the line number determination unit 111 determines that the pixel selected in step S10 corresponds to a long-distance image, and sets a large number of scan lines for the pixel. In the present embodiment, for example, the number of scan lines is determined to be eight.

When the number of scan lines is determined by the line number determination unit 111 in step S40 or S50, in step S60, the parallax calculation unit 102 calculates the parallax of the pixel selected in step S10 based on the determined number of scan lines. I do. Here, the smoothing calculation based on the determined number of scan lines is performed on the pixel, and the dissimilarity is optimized to calculate the parallax. Thereby, the parallax in the stereo camera image of the current frame is calculated for each pixel.

において In step S70, the parallax calculation unit 102 determines whether the calculation for all pixels has been completed. If there is a pixel for which the calculation has not been performed yet, the process returns to step S10 to select the next pixel, and the above processing is repeated. When the calculation for all pixels is completed, the obtained parallax image of the current frame is stored in the parallax image storage unit 104, and then the parallax calculation shown in the flowchart of FIG.

In the above steps S40 and S50, the number of scan lines to be set has been exemplified as four and eight, respectively, but these are only examples and do not limit the present invention. Alternatively, a plurality of values may be set as the threshold value dth used in the determination in step S30, and the values may be compared with the parallax obtained in the previous frame, so that three or more types of scan lines may be set. In any case, it is preferable to determine that the smaller the value of the parallax obtained in the previous frame, the longer the distance to the image is, and to increase the number of scan lines.

(Second embodiment)
Next, a second embodiment of the present invention will be described. In the present embodiment, an example will be described in which, when the number of scan lines is determined by the line number determination unit 111, a distance range for each area based on the result of object detection performed by the processor 30 is used.

FIG. 4 is a block diagram showing a configuration of an image processing device according to the second embodiment of the present invention. As shown in FIG. 4, the image processing apparatus 10 according to the present embodiment differs from the first embodiment described with reference to FIG. 2 in that the memory 12 does not include the parallax image Is different in that the area information output from is input to the line number determination unit 111. This area information is information set for each object based on the result of the object detection performed by the processor 30, and information indicating which area in the stereo camera image the object corresponds to, Indicating whether the distance is classified into a short distance range or a long distance range.

In the present embodiment, the line number determination unit 111 inputs area information output from the processor 30 in accordance with the parallax calculated from the stereo camera image of the previous frame, and converts the information of the distance range for each area in the area information. Based on this, the number of scan lines used in the smoothing process performed by the parallax calculation unit 102 is determined. That is, the line number determination unit 111 of the present embodiment acquires information on the distance range determined based on the parallax calculated in the past by the parallax calculation unit 102, and based on the acquired information on the distance range, Determine the number. The other points are the same as the first embodiment.

FIG. 5 is a flowchart illustrating an algorithm for calculating parallax in the image processing apparatus according to the second embodiment of the present invention. The parallax calculation shown in the flowchart of FIG. 5 is performed in the FPGA 11 when the stereo camera image newly photographed by the camera 20 is stored in the stereo camera photographed image storage unit 103 as the stereo camera image of the current frame.

In step S <b> 10 </ b> A, the line number determination unit 111 reads from the processor 30 the determination result of the distance range for each region determined by the processor 30 based on the parallax for the stereo camera image of the previous frame. Here, by acquiring the above-described area information output from the processor 30, the distance range of each area corresponding to each object detected from the stereo camera image of the previous frame corresponds to either the short distance or the long distance. Is read.

In step S20A, the parallax calculation unit 102 reads the stereo camera image of the current frame from the stereo camera captured image storage unit 103, and sequentially selects the pixels at the end of the screen.

In step S30A, the line number determination unit 111 determines, based on the determination result of the distance range of the region corresponding to the pixel selected in step S20A, among the determination results of the distance range in the previous frame read in step S10A, It is determined whether or not the object detected in the previous frame is within the short distance area. If the determination result of the distance range is a short distance, it is determined that the object is in the short distance area in the previous frame, and the process proceeds to step S40. If the determination result of the distance range is a long distance, the object is It is determined that it was not in the short distance area in the previous frame, and the process proceeds to step S50.

In step S40, the line number determination unit 111 determines that the pixel selected in step S20A corresponds to a short-distance image, and sets a smaller number of scan lines for the pixel. In the present embodiment, as in the first embodiment, for example, the number of scan lines is determined to be four.

In step S50, the line number determination unit 111 determines that the pixel selected in step S20A corresponds to a long-distance image, and sets a large number of scan lines for the pixel. In the present embodiment, for example, the number of scan lines is determined to be eight as in the first embodiment.

When the number of scan lines is determined by the line number determination unit 111 in step S40 or S50, the parallax calculation unit 102 performs the same processing as that in the flowchart of FIG. 3 described in the first embodiment after step S60. . That is, in step S60, the calculation of the parallax based on the determined number of scan lines is performed on the pixel selected in step S20A, and in step S70, it is determined whether the calculation on all the pixels has been completed. If there is a pixel that has not been calculated, the process returns to step S20A to select the next pixel, and then repeats the above processing. When the calculation for all pixels is completed, the parallax calculation shown in the flowchart of FIG. 5 is completed.

In the present embodiment, as in the first embodiment, the number of scan lines set in steps S40 and S50 is not limited to four or eight. Further, by setting the number of distance ranges of each area represented by the area information output from the processor 30 to three or more, three or more types of scan lines may be set. In any case, it is preferable to increase the number of scan lines set for pixels corresponding to the object as the object exists at a longer distance.

Further, in the above description, an example in which the area information obtained from the stereo camera image of the immediately preceding frame is used as the information used to determine the number of scan lines, but this does not limit the present invention. For example, it is also possible to predict and use the area information of the current frame based on the area information of each of the previous frame and the frame before the previous frame. Further, it is also possible to correct area information predicted in the current frame based on traveling information of a vehicle on which the image processing device 10 is mounted, for example, steering information.

In the above-described first embodiment, the number of scan lines is determined based on parallax information in pixel units or pixel group units around pixels. In the present embodiment, the number of scan lines is determined in the area unit obtained by object detection. Is used to determine the number of scan lines. Therefore, there is an effect that the number of scan lines can be determined based on more global information as compared with the first embodiment.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the present embodiment, an example will be described in which the number of scan lines is determined by the line number determination unit 111 using the result of object detection performed by the processor 30.

FIG. 6 is a block diagram illustrating a configuration of an image processing apparatus according to the third embodiment of the present invention. As shown in FIG. 6, the image processing apparatus 10 according to the present embodiment transmits the information of the object detection result output from the processor 30 to the line number determination unit 111 as compared with the second embodiment described in FIG. The input points are different. This information is information indicating the result of the object detection performed by the processor 30, and includes information indicating in which region in the stereo camera image the object has been detected.

In the present embodiment, the number-of-lines determining unit 111 inputs information of an object detection result output from the processor 30 according to the parallax calculated from the stereo camera image of the previous frame, and based on the information, a parallax calculating unit. The number of scan lines used in the smoothing process performed by 102 is determined. For example, a small number of lines is adopted for a pixel determined to be an image of an already detected object, and a large number of lines is adopted for a pixel determined to be an image for which no object has been detected. That is, the line number determination unit 111 of the present embodiment acquires information on an object detected based on the parallax calculated in the past by the parallax calculation unit 102, and determines the number of scan lines based on the acquired information on the object. decide. The other points are the same as the first and second embodiments.

FIG. 7 is a diagram showing an example of determining the number of scan lines in the image processing device according to the third embodiment of the present invention. As shown in FIG. 7, in the parallax calculation for the pixels in the area of the image 901 of the vehicle in which the object has been detected in the previous frame, the number of lines 911 is set to be small even at a long distance. On the other hand, for the image 902 of the vehicle for which no object is detected at the time of the previous frame, the number of lines 912 is set to be large, so that more accurate parallax calculation is performed. Further, a smaller number of lines 913 is set for the road surface 903 on which the object has been detected.

In the present embodiment, by setting the number of scan lines using the result of object detection in the previous frame, more accurate parallax calculation is performed for an image for which distance information has not been obtained in the past. be able to.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In the present embodiment, an example will be described in which the line number determination unit 111 uses the distance information measured by another sensor when determining the number of scan lines.

FIG. 8 is a block diagram showing a configuration of an image processing apparatus according to the fourth embodiment of the present invention. As shown in FIG. 8, the image processing apparatus 10 according to the present embodiment is an example of a sensor capable of acquiring distance information as compared with the second and third embodiments described with reference to FIGS. The difference is that the distance information output from the LIDAR (Light Detection Detection And Ranging) 40 is input to the line number determination unit 111. Note that the LIDAR 40 in FIG. 8 is an example of another sensor capable of acquiring distance information, and the present invention is not limited to this. Alternatively, distance information output from, for example, an infrared depth sensor, an ultrasonic sensor, a millimeter wave radar, or another camera image may be used.

In the present embodiment, the line number determination unit 111 inputs distance information output from the LIDAR 40 for a range of a stereo camera image captured by the camera 20, and performs the same processing as in the first embodiment based on the distance information. Next, the number of scan lines used in the smoothing process performed by the parallax calculation unit 102 is determined. That is, the distance of each pixel represented by the distance information is compared with a predetermined threshold, and the number of scan lines is set smaller for pixels whose distance is larger than the threshold, and the number of scan lines is set smaller for pixels whose distance is smaller than the threshold. Set a large number of lines. The other points are the same as the first to third embodiments.

(Fifth embodiment)
Next, a fifth embodiment of the present invention will be described. In the present embodiment, instead of the line number determining unit 111 described in the first to fourth embodiments, an image processing apparatus having a sparse density determining unit 112 that determines a sparse density representing a ratio of pixels to be thinned out in a smoothing calculation Will be described.

FIG. 9 is a block diagram showing a configuration of an image processing device according to the fifth embodiment of the present invention. As shown in FIG. 9, in the image processing apparatus 10 according to the present embodiment, the condition setting unit 101 of the FPGA 11 is different from the first embodiment described with reference to FIG. The difference is that a point 112 is provided. The sparse density determination unit 112 reads the parallax image generated by the parallax calculation unit 102 from the stereo camera image of the previous frame and stored in the parallax image storage unit 104, and is executed by the parallax calculation unit 102 based on the parallax image. Determine the sparse density in the smoothing process. That is, the sparse density determining unit 112 of the present embodiment determines the sparse density based on the parallax calculated by the parallax calculating unit 102 in the past. Then, the parallax calculating unit 102 is notified of the determined sparse density. The condition setting unit 101 in the present embodiment sets the condition for the parallax calculation unit 102 by notifying the sparse density determined by the sparse density determination unit 112 in this manner.

When notification of the sparse density is performed from the sparse density determination unit 112, the parallax calculation unit 102 calculates a ratio corresponding to the notified sparse density to the stereo camera image of the current frame read from the stereo camera captured image storage unit 103. To thin out the pixels. Then, smoothing calculation is performed with the pixels of the remaining stereo camera image not thinned out as calculation target pixels. After that, similar to the first embodiment, the dissimilarity calculation is performed for each pixel on the stereo camera image after the smoothing calculation, and the parallax for each pixel is calculated based on the calculation result. The parallax calculation unit 102 outputs the parallax calculation result obtained in pixel units as a parallax image to the processor 30 and stores the parallax calculation result in the parallax image storage unit 104.

FIG. 10 is an explanatory diagram of pixel thinning used in the smoothing calculation in the image processing device according to the fifth embodiment of the present invention. FIG. 10A is a diagram illustrating an example of pixels used in a smoothing calculation of a pixel corresponding to a long-distance image. As for the pixels for which the parallax represented by the parallax image of the previous frame is small and which is determined to correspond to an image at a long distance, pixels on each scan line are not thinned out as shown in FIG. A dense smoothing operation is performed in a narrow range on an image. On the other hand, FIG. 10B is a diagram illustrating an example of a pixel used in a smoothing calculation of a pixel corresponding to an image at a short distance. As for the pixels for which the parallax represented by the parallax image of the previous frame is large and which is determined to correspond to an image at a short distance, pixels on each scan line are thinned out as shown in FIG. Smoothing operation is performed using a small number of pixels in a wide range. As described above, the sparse density determining unit 112 of the present embodiment determines that the larger the value of the parallax obtained in the previous frame is, the shorter the distance to the image is, and increases the ratio of the pixels thinned out from the stereo camera image. Preferably, the sparse density is determined.

In FIG. 10, the example in which the pixels used for the smoothing operation are arranged on the scan line has been described. However, it is not necessary to limit the pixels to be arranged on the line. FIG. 11 is an explanatory diagram of an arrangement of pixels used for smoothing calculation in the image processing device according to the fifth embodiment of the present invention. FIG. 11A shows an example in which pixels arranged side by side on eight scan lines are used for smoothing calculation. On the other hand, FIG. 11B shows an example in which pixels arranged not according to a scan line but arranged according to a predetermined arrangement pattern are used for smoothing calculation. In each of the arrangement examples shown in FIGS. 11A and 11B, it is possible to perform the above-described pixel thinning based on the sparse density determined by the sparse density determination unit 112 and perform a smoothing operation. it can. At this time, which pixels to thin out can be determined in advance. However, the pixel arrangement shown in FIG. 11 is an example, and the pixel arrangement used for the smoothing calculation is not limited to this.

Further, in the present embodiment, the sparse density determination unit 112 determines the number of scan lines in the line number determination unit 111 described in the first embodiment, based on the parallax calculated in the past by the parallax calculation unit 102. Although the example of determining the sparse density has been described, the present invention is not limited to this. For example, similarly to the determination of the number of scan lines in the second embodiment, information on the distance range determined based on the disparity calculated in the past by the disparity calculation unit 102 is acquired from the processor 30 and the acquired information on the distance range is acquired. Sparse density may be determined based on Further, similarly to the determination of the number of scan lines in the third embodiment, information of an object detected based on the disparity calculated in the past by the disparity calculating unit 102 is acquired from the processor 30 and based on the acquired information of the object. Alternatively, the density may be determined. Alternatively, similarly to the determination of the number of scan lines in the fourth embodiment, distance information corresponding to the range of the stereo camera image measured by another sensor is obtained, and the sparse density is determined based on the obtained distance information. You may. Other than this, the sparse density can be determined by an arbitrary method.

(Sixth embodiment)
Next, a sixth embodiment of the present invention will be described. In the present embodiment, an example of an image processing apparatus in which the condition setting unit 101 includes the line number determination unit 111 described in the first to fourth embodiments and the sparse density determination unit 112 described in the fifth embodiment will be described. explain.

FIG. 12 is a block diagram showing a configuration of an image processing apparatus according to the sixth embodiment of the present invention. As shown in FIG. 12, in the image processing apparatus 10 according to the present embodiment, the condition setting unit 101 of the FPGA 11 is different from the first embodiment described with reference to FIG. The difference is that a point 112 is further provided. This sparse density determination unit 112 is the same as that described in the fifth embodiment. Although FIG. 12 illustrates an example in which the memory 12 includes the parallax image storage unit 104, the line number determination unit 111 and the sparse density determination unit 112 use the same method as in the second to fourth embodiments. When the number of scan lines and the sparse density are determined by using, the parallax image storage unit 104 may be omitted without being provided in the memory 12. Further, as described in the fourth embodiment, when determining the number of scan lines or the sparse density using the distance information measured by another sensor, the distance information from the sensor is used as the line number determination unit. 111 or the sparse density determination unit 112.

According to the embodiment of the present invention described above, the following operational effects can be obtained.

(1) The image processing apparatus 10 calculates parallax based on a stereo camera image captured by the camera 20. The image processing apparatus 10 includes: a condition setting unit 101 that sets a condition for determining a calculation target pixel in a stereo camera image; and a calculation target pixel in the stereo camera image based on the condition set by the condition setting unit 101. And a parallax calculating unit 102 that calculates parallax using information of the calculation target pixel. With this configuration, the amount of calculation required for calculating the parallax can be suppressed.

(2) In each of the first to fourth and sixth embodiments, the condition setting unit 101 has a line number determination unit 111 that determines the number of scan lines for scanning a stereo camera image. The parallax calculation unit 102 sets the number of scan lines determined by the line number determination unit 111 for the stereo camera image, and determines pixels of the stereo camera image on the scan line as calculation target pixels. With this configuration, it is possible to appropriately determine the calculation target pixel according to the condition.

(3) In the first embodiment, the line number determination unit 111 determines the number of scan lines based on the parallax calculated by the parallax calculation unit 102 in the past. At this time, the line number determination unit 111 increases the number of scan lines as the value of the parallax is smaller. With this configuration, it is possible to determine the optimal number of scan lines according to the distance for each pixel from past disparity information.

(4) In the second embodiment, the number-of-lines determining unit 111 acquires information on the distance range determined based on the disparity calculated in the past by the disparity calculating unit 102, and based on the acquired information on the distance range. , Determine the number of scan lines. With this configuration, it is possible to determine the optimal number of scan lines according to the distance for each object from the past disparity information.

(5) In the third embodiment, the line number determination unit 111 acquires information on an object detected based on the parallax calculated in the past by the parallax calculation unit 102, and performs scanning based on the acquired information on the object. Determine the number of lines. With this configuration, it is possible to determine the optimal number of scan lines for each object from past disparity information.

(6) In the fourth embodiment, the number-of-lines determining unit 111 acquires distance information for a range of a stereo camera image, and determines the number of scan lines based on the acquired distance information. Thus, even when parallax cannot be calculated from a past stereo camera image, it is possible to determine the optimal number of scan lines according to the distance for each pixel.

(7) In each of the fifth and sixth embodiments, the condition setting unit 101 includes a sparse density determination unit 112 that determines a sparse density representing a ratio of pixels to be thinned out from a stereo camera image. The parallax calculation unit 102 determines the remaining pixels thinned out from the stereo camera image based on the sparse density determined by the sparse density determination unit 112 as calculation target pixels. With this configuration, it is possible to appropriately determine the calculation target pixel according to the condition.

(8) In the fifth and sixth embodiments, the sparse density determination unit 112 can determine the sparse density based on the parallax calculated by the parallax calculation unit 102 in the past. At this time, the sparse density determining unit 112 determines the sparse density such that the larger the value of the parallax, the greater the proportion of pixels thinned out from the stereo camera image. With this configuration, it is possible to determine the optimal sparse density according to the distance for each pixel from the past disparity information.

(9) In each of the fifth and sixth embodiments, the sparse-density determining unit 112 obtains information on the distance range determined based on the parallax calculated in the past by the parallax calculating unit 102, and obtains the information on the obtained distance range. The sparse density can also be determined based on the information. Further, information on an object detected based on the parallax calculated in the past by the parallax calculation unit 102 may be acquired, and the sparse density may be determined based on the acquired information on the object. Furthermore, it is also possible to acquire distance information with respect to the range of the stereo camera image and determine the sparse density based on the acquired distance information. This makes it possible to determine the optimum sparse density according to the situation.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various changes can be made without departing from the scope described in the claims. For example, in each of the embodiments described above, the image processing apparatus 10 used by being mounted on a vehicle has been described, but the present invention is also applicable to an image processing apparatus used in another system.

The embodiments and modifications described above are merely examples, and the present invention is not limited to these contents as long as the features of the invention are not impaired. Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other embodiments that can be considered within the scope of the technical concept of the present invention are also included in the scope of the present invention.

The disclosure of the following priority application is incorporated herein by reference.
Japanese patent application 2018-168882 (filed on September 10, 2018)

DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus, 11 ... FPGA, 12 ... Memory, 20 ... Camera, 30 ... Processor, 40 ... LIDAR, 101 ... Condition setting part, 102 ... Parallax calculation part (SGM calculation), 103 ... Stereo camera photography image storage part , 104: parallax image storage unit, 111: line number determination unit, 112: sparse density determination unit

Claims

An image processing apparatus that calculates parallax based on a stereo camera image captured by a stereo camera,
A condition setting unit for setting a condition for determining a calculation target pixel in the stereo camera image,
A parallax calculating unit that determines the calculation target pixel in the stereo camera image based on the condition set by the condition setting unit, and calculates the parallax using information on the calculation target pixel. Processing equipment.
The image processing apparatus according to claim 1,
The condition setting unit has a line number determination unit that determines the number of scan lines to scan the stereo camera image,
The parallax calculation unit sets the number of the scan lines determined by the line number determination unit for the stereo camera image, the pixels of the stereo camera image on the scan line, the calculation target pixel Image processing device to determine.
The image processing apparatus according to claim 2,
The image processing device, wherein the number-of-lines determining unit determines the number of the scan lines based on the parallax calculated by the parallax calculating unit in the past.
The image processing apparatus according to claim 3,
The image processing device, wherein the line number determining unit increases the number of the scan lines as the value of the parallax is smaller.
The image processing apparatus according to claim 2,
The line number determining unit obtains information on a distance range determined based on the parallax calculated by the parallax calculating unit in the past, and determines the number of the scan lines based on the obtained information on the distance range. Image processing device.
The image processing apparatus according to claim 2,
The line number determination unit obtains information on an object detected based on the parallax calculated in the past by the parallax calculation unit, and determines the number of the scan lines based on the obtained information on the object. Processing equipment.
The image processing apparatus according to claim 2,
The image processing device, wherein the number-of-lines determining unit obtains distance information for a range of the stereo camera image, and determines the number of the scan lines based on the obtained distance information.
The image processing apparatus according to any one of claims 1 to 7,
The condition setting unit includes a sparse density determination unit that determines a sparse density representing a ratio of pixels to be thinned out from the stereo camera image,
The image processing device, wherein the parallax calculation unit determines remaining pixels thinned out from the stereo camera image based on the sparse density determined by the sparse density determination unit as the calculation target pixels.
The image processing device according to claim 8,
The image processing device, wherein the sparse density determination unit determines the sparse density based on the parallax calculated by the parallax calculation unit in the past.
The image processing apparatus according to claim 9,
The image processing device, wherein the sparse density determining unit determines the sparse density such that the larger the value of the parallax, the greater the proportion of pixels to be thinned out from the stereo camera image.
The image processing device according to claim 8,
The sparse density determining unit acquires information on a distance range determined based on the parallax calculated by the parallax calculating unit in the past, and determines the sparse density based on the acquired information on the distance range. Processing equipment.
The image processing device according to claim 8,
An image processing apparatus that obtains information on an object detected based on the parallax calculated in the past by the parallax calculating unit, and determines the sparse density based on the obtained information on the object. .
The image processing device according to claim 8,
The image processing device, wherein the sparse density determining unit acquires distance information for a range of the stereo camera image, and determines the sparse density based on the acquired distance information.