JP7066580B2 - Image processing equipment - Google Patents

Description

The present invention relates to an image processing apparatus.

A position measuring device for measuring the distance to an object existing in the field of view of a camera based on a stereo image taken by a stereo camera having a pair of cameras by the principle of triangulation is known. The principle of triangulation is to calculate the distance from the camera to the object by using the positional deviation (parallax) of the images of the same object taken by the left and right cameras. Parallax derivation is achieved by identifying where an image on one image of an object is on the other image.

Various methods have been proposed for deriving parallax. In the classical method, block matching is known in which a region consisting of a plurality of pixels in one image is searched for a region having the lowest dissimilarity in the other image. Furthermore, as a method of reducing mismatching and improving accuracy by including not only local information in the image but also information of the entire image in the matching process, for example, a method called SGM (Semi-Global Matching) has been proposed. ing. While such a method can be expected to have high accuracy, there is a problem that the amount of calculation increases as compared with the classical method.

For example, the technique described in Patent Document 1 is known for suppressing the amount of calculation in the calculation of parallax using a stereo camera. Patent Document 1 is an obstacle measuring method for measuring the distance to an obstacle by parallax calculation using a stereo image, and is used for calculating the misalignment in a high-resolution image based on the distance information obtained from the low-resolution image. Disclosed is an obstacle measuring method characterized in that the sampling interval of a pixel to be a target of a correlation calculation is set to be large at a short distance and small at a long distance.

Japanese Unexamined Patent Publication No. 2008-298533

In the method of Patent Document 1, by setting an appropriate sampling interval, highly accurate distance information can be acquired only in a necessary area, unnecessary calculation can be suppressed, and highly accurate distance information can be acquired only in a necessary area. At the same time, it is possible to shorten the calculation time and reduce the circuit scale of the device. However, since it is necessary to calculate the parallax from the low-resolution image and the high-resolution image, the total amount of calculation cannot be reduced so much, and there is room for further improvement in suppressing the amount of calculation.

The image processing device according to the present invention calculates the disparity based on the stereo camera image taken by the stereo camera , and for each pixel of the stereo camera image, a plurality of calculation target pixels corresponding to the pixel. Based on the condition setting unit for setting the condition for determining the condition and the condition set by the condition setting unit, the plurality of calculation target pixels are determined for each pixel of the stereo camera image, and the plurality of calculation target pixels are determined. The stereo camera image is provided with a parallax calculation unit that calculates the parallax for each pixel of the stereo camera image using the information of the calculation target pixel, and the parallax calculation unit includes the pixel value of each pixel of the stereo camera image and the pixel. A smoothing calculation is performed to average each pixel value of the corresponding plurality of calculation target pixels, and a dissimilarity calculation is performed for each pixel of the stereo camera image in which the smoothing calculation is performed. The difference is calculated based on the calculation result of the dissimilarity .

According to the present invention, the amount of calculation required for parallax calculation can be suppressed.

A block diagram showing a basic configuration of an image processing apparatus according to an embodiment of the present invention. The block diagram which shows the structure of the image processing apparatus which concerns on 1st Embodiment of this invention. A flowchart showing an algorithm for parallax calculation in the image processing apparatus according to the first embodiment of the present invention. The block diagram which shows the structure of the image processing apparatus which concerns on 2nd Embodiment of this invention. A flowchart showing an algorithm for parallax calculation in the image processing apparatus according to the second embodiment of the present invention. The block diagram which shows the structure of the image processing apparatus which concerns on 3rd Embodiment of this invention. The figure which shows the example of the scan line number determination in the image processing apparatus which concerns on 3rd Embodiment of this invention. The block diagram which shows the structure of the image processing apparatus which concerns on 4th Embodiment of this invention. The block diagram which shows the structure of the image processing apparatus which concerns on 5th Embodiment of this invention. Explanatory drawing of thinning out of pixel used for smoothing calculation in image processing apparatus which concerns on 5th Embodiment of this invention Explanatory drawing of arrangement of pixel used for smoothing calculation in image processing apparatus which concerns on 5th Embodiment of this invention A block diagram showing a configuration of an image processing apparatus according to a sixth embodiment of the present invention.

(Basic configuration)
First, the basic configuration of the embodiment of the present invention will be described. FIG. 1 is a block diagram showing a basic configuration of an image processing apparatus according to an embodiment of the present invention. The image processing device 10 shown in FIG. 1 is connected to the camera 20 and the processor 30, and includes a condition setting unit 101 and a parallax calculation unit 102. The image processing device 10 is used, for example, by being mounted on a vehicle to detect and recognize an object such as another vehicle or an obstacle existing around the vehicle.

The camera 20 is a stereo camera installed at a predetermined interval on the left and right, and captures a pair of left and right stereo camera images and outputs the captured data to the image processing device 10. The image processing device 10 calculates the parallax in the stereo camera image by the parallax calculation unit 102 based on the shooting data input from the camera 20. At this time, the parallax calculation unit 102 determines the calculation target pixel in the stereo camera image based on the condition set by the condition setting unit 101, and uses the information of the calculation target pixel to perform a calculation method called, for example, SGM. The parallax is calculated by. The parallax may be calculated by using a calculation method other than SGM.

The parallax calculation result by the parallax calculation unit 102 is output from the image processing device 10 to the processor 30. The processor 30 executes processing such as object detection and object recognition using the parallax information input from the image processing device 10.

Subsequently, each embodiment of the present invention using the above basic configuration will be described.

(First Embodiment)
FIG. 2 is a block diagram showing a configuration of an image processing apparatus according to the first embodiment of the present invention. As shown in FIG. 2, the image processing apparatus 10 according to the present embodiment includes an FPGA 11 and a memory 12. The FPGA 11 is an arithmetic processing device configured by combining a large number of logic circuits, and has a condition setting unit 101 and a parallax calculation unit 102 described with reference to FIG. 1 as its functions. The condition setting unit 101 and the parallax calculation unit 102 may be mounted on the image processing device 10 by using other logic circuits such as ASIC, software executed by the CPU, or the like other than the FPGA 11. The memory 12 is a storage device configured by using a readable / writable recording medium such as a RAM, an HDD, or a flash memory, and has a stereo camera shooting image storage unit 103 and a parallax image storage unit 104 as its functions.

The shooting data representing the stereo camera image input from the camera 20 to the image processing device 10 is stored in the stereo camera shooting image storage unit 103 in the memory 12 in frame units. The parallax calculation unit 102 in the FPGA 11 reads the pixel data of the stereo camera image of each frame from the stereo camera image storage unit 103, and calculates the parallax using this, so that the parallax image represents the parallax in pixel units. Is generated frame by frame. The parallax image generated by the parallax calculation unit 102 is output to the processor 30 and stored in the parallax image storage unit 104 in the memory 102.

The parallax calculation unit 102 performs a smoothing calculation and a dissimilarity calculation in the parallax calculation using the stereo camera image. The smoothing calculation smoothes (averages) the pixel value of each pixel in the stereo camera image together with the pixel values of a plurality of pixels around it, thereby smoothing the fluctuation of the pixel value between pixels. It is the arithmetic processing to make. The calculation of dissimilarity is the certainty that the object shown in one of the stereo camera images and the object shown in the other image are not the same, that is, the degree of dissimilarity that indicates how far apart they are. It is the required arithmetic processing.

In the present embodiment, the condition setting unit 101 has a line number determination unit 111. The line number determination unit 111 reads out the parallax image generated by the parallax calculation unit 102 from the stereo camera image of the previous frame and stored in the parallax image storage unit 104, and the parallax calculation unit 102 executes the parallax image based on the parallax image. Determine the number of scanlines used in the smoothing process. That is, the line number determination unit 111 of the present embodiment determines the number of scan lines for scanning the stereo camera image of the current frame based on the parallax calculated by the parallax calculation unit 102 in the past. Then, the number of the determined scan lines is notified to the parallax calculation unit 102. The condition setting unit 101 in the present embodiment sets conditions for the parallax calculation unit 102 by notifying the number of scan lines determined by the line number determination unit 111 in this way.

When the number of scan lines is notified from the line number determination unit 111, the parallax calculation unit 102 notifies the stereo camera image of the current frame read from the stereo camera shooting image storage unit 103 for each pixel. Set the scan line. Then, the smoothing calculation is performed using the pixels of the stereo camera image on each set scan line as the calculation target pixels. After that, the dissimilarity is calculated for each pixel on the stereo camera image after the smoothing calculation, and the parallax of each pixel is calculated based on the calculation result. The parallax calculation unit 102 outputs the parallax calculation result for each pixel thus obtained to the processor 30 as a parallax image, and stores it in the parallax image storage unit 104.

FIG. 3 is a flowchart showing an algorithm for parallax calculation in the image processing apparatus according to the first embodiment of the present invention. The disparity calculation shown in the flowchart of FIG. 3 is performed in the FPGA 11 when the stereo camera image newly taken by the camera 20 is stored in the stereo camera shooting image storage unit 103 as the stereo camera image of the current frame.

In step S10, the parallax calculation unit 102 reads out the stereo camera image of the current frame from the stereo camera shooting image storage unit 103, and selects in order from the pixel at the edge of the screen.

In step S20, the line number determination unit 111 reads out the parallax of the pixel selected in step S10 from the parallax image calculated by the parallax calculation unit 102 in the previous frame from the parallax image storage unit 104 and refers to it.

In step S30, the line number determination unit 111 determines whether or not the parallax referred to in step S20 is larger than the predetermined threshold value dth. If the parallax is larger than the threshold value dth, the process proceeds to step S40, and if the parallax is equal to or less than the threshold value dth, the process proceeds to step S50.

In step S40, the line number determination unit 111 determines that the pixel selected in step S10 corresponds to a short-distance image, and sets a small number of scan lines for the pixel. In this embodiment, for example, the number of scan lines is determined to be four.

In step S50, the line number determination unit 111 determines that the pixel selected in step S10 corresponds to a long-distance image, and sets a large number of scan lines for the pixel. In this embodiment, for example, the number of scan lines is determined to be eight.

When the number of scan lines is determined by the line number determination unit 111 in step S40 or S50, the parallax calculation unit 102 in step S60 calculates the parallax based on the determined number of scan lines for the pixels selected in step S10. I do. Here, the smoothing calculation based on the determined number of scan lines is performed on the pixel, and the dissimilarity is optimized to calculate the parallax. As a result, the parallax in the stereo camera image of the current frame is calculated in pixel units.

In step S70, the parallax calculation unit 102 determines whether or not the calculation for all pixels has been completed. If there is a pixel for which the calculation has not been performed yet, the process returns to step S10 to select the next pixel, and then the above process is repeated. When the calculation for all pixels is completed, the obtained parallax image of the current frame is stored in the parallax image storage unit 104, and then the parallax calculation shown in the flowchart of FIG. 3 is completed.

In steps S40 and S50, the number of scan lines to be set is illustrated as 4 and 8, respectively, but these are examples and do not limit the present invention. Further, by setting a plurality of values for the threshold value dth used for the determination in step S30 and comparing them with the parallax obtained in the previous frame, it is possible to set the number of three or more types of scan lines. In any case, it is preferable to increase the number of scan lines by determining that the distance to the image is longer as the parallax value obtained in the previous frame is smaller.

(Second embodiment)
Next, a second embodiment of the present invention will be described. In this embodiment, an example of using a distance range for each region based on the result of object detection performed by the processor 30 when determining the number of scan lines in the line number determination unit 111 will be described.

FIG. 4 is a block diagram showing a configuration of an image processing apparatus according to a second embodiment of the present invention. As shown in FIG. 4, the image processing apparatus 10 according to the present embodiment is not provided with the parallax image storage unit 104 in the memory 12 and the processor 30 as compared with the first embodiment described with reference to FIG. The difference is that the area information output from is input to the line number determination unit 111. This area information is information set for each object based on the result of object detection performed by the processor 30, information indicating which area in the stereo camera image the object corresponds to, and up to the object. It contains information indicating whether the distance of is classified into a short distance or a long distance range.

In the present embodiment, the line number determination unit 111 inputs the area information output from the processor 30 according to the parallax calculated from the stereo camera image of the previous frame, and uses the area information as the information of the distance range for each area. Based on this, the number of scan lines used in the smoothing process performed by the parallax calculation unit 102 is determined. That is, the line number determination unit 111 of the present embodiment acquires the information of the distance range determined based on the parallax calculated by the parallax calculation unit 102 in the past, and based on the acquired distance range information, the scan line Determine the number. Other than this, it is the same as the first embodiment.

FIG. 5 is a flowchart showing an algorithm for parallax calculation in the image processing apparatus according to the second embodiment of the present invention. The disparity calculation shown in the flowchart of FIG. 5 is performed in the FPGA 11 when the stereo camera image newly captured by the camera 20 is stored in the stereo camera captured image storage unit 103 as the stereo camera image of the current frame.

In step S10A, the line number determination unit 111 reads from the processor 30 the determination result of the distance range for each region determined by the processor 30 based on the parallax with respect to the stereo camera image of the previous frame. Here, by acquiring the above-mentioned area information output from the processor 30, the distance range of each area corresponding to each object detected from the stereo camera image of the previous frame corresponds to either a short distance or a long distance. Read the information indicating that.

In step S20A, the parallax calculation unit 102 reads out the stereo camera image of the current frame from the stereo camera shooting image storage unit 103, and selects the stereo camera image of the current frame in order from the pixel at the edge of the screen.

In step S30A, the line number determination unit 111 determines the distance range of the region corresponding to the pixel selected in step S20A among the distance range determination results in the previous frame read in step S10A. It is determined whether or not the object detected in the previous frame is within the short-distance region. If the determination result of the distance range is a short distance, it is determined that the object was in the short distance region in the previous frame, and the process proceeds to step S40. If the determination result of the distance range is a long distance, the object is determined. It is determined that the previous frame was not within the short distance region, and the process proceeds to step S50.

In step S40, the line number determination unit 111 determines that the pixel selected in step S20A corresponds to a short-distance image, and sets a small number of scan lines for the pixel. In this embodiment, as in the first embodiment, for example, the number of scan lines is determined to be four.

In step S50, the line number determination unit 111 determines that the pixel selected in step S20A corresponds to a long-distance image, and sets a large number of scan lines for the pixel. In this embodiment, as in the first embodiment, for example, the number of scan lines is determined to be eight.

When the number of scan lines is determined by the line number determination unit 111 in step S40 or S50, the parallax calculation unit 102 performs the same processing as the flowchart of FIG. 3 described in the first embodiment in step S60 and subsequent steps. .. That is, in step S60, the parallax is calculated based on the determined number of scan lines for the pixels selected in step S20A, and in step S70, it is determined whether or not the calculation for all the pixels is completed. If there is a pixel for which the calculation has not been performed yet, the process returns to step S20A, the next pixel is selected, and then the above processing is repeated. When the calculation for all pixels is completed, the parallax calculation shown in the flowchart of FIG. 5 is completed.

In this embodiment as well, as in the first embodiment, the number of scan lines set in steps S40 and S50 is not limited to four and eight, respectively. Further, by setting the number of classifications of the distance range of each area represented by the area information output from the processor 30 to 3 or more, the number of scan lines of 3 or more types may be set. In any case, it is preferable to increase the number of scan lines set for the pixels corresponding to the object as the object exists at a longer distance.

Further, in the above description, an example is shown in which the area information obtained from the stereo camera image of the immediately preceding frame is used as the information used for determining the number of scan lines, but this is not limited to the present invention. For example, it is also possible to predict and use the area information of the current frame based on the area information of each of the previous frame and the previous frame. Further, it is also possible to correct the area information predicted in the current frame based on the traveling information of the vehicle on which the image processing device 10 is mounted, for example, the steering information.

In the above-mentioned first embodiment, the number of scan lines is determined based on the parallax information of the pixel unit or the pixel group unit around the pixel, whereas in the present embodiment, the area unit obtained by object detection is used. The number of scan lines is determined using the information in the distance range of. Therefore, as compared with the first embodiment, there is an effect that the number of scan lines can be determined based on more global information.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In this embodiment, an example of using the result of object detection performed by the processor 30 when determining the number of scan lines in the line number determination unit 111 will be described.

FIG. 6 is a block diagram showing a configuration of an image processing apparatus according to a third embodiment of the present invention. As shown in FIG. 6, in the image processing apparatus 10 according to the present embodiment, the information of the object detection result output from the processor 30 is sent to the line number determination unit 111 as compared with the second embodiment described with reference to FIG. The points to be entered are different. This information is information representing the result of object detection performed by the processor 30, and includes information indicating to which region in the stereo camera image the object was detected.

In the present embodiment, the line number determination unit 111 inputs information on the object detection result output from the processor 30 according to the parallax calculated from the stereo camera image of the previous frame, and the parallax calculation unit is based on the information. Determine the number of scanlines used in the smoothing process performed by 102. For example, a small number of lines is adopted for a pixel determined to be an image of an object that has already been detected, and a large number of lines is adopted to a pixel determined to be an image not detected by an object. That is, the line number determination unit 111 of the present embodiment acquires the information of the object detected based on the parallax calculated by the parallax calculation unit 102 in the past, and determines the number of scan lines based on the acquired object information. decide. Other than this, it is the same as the first and second embodiments.

FIG. 7 is a diagram showing an example of determining the number of scan lines in the image processing apparatus according to the third embodiment of the present invention. As shown in FIG. 7, in the parallax calculation for the pixels in the region of the image 901 of the vehicle whose object has been detected in the front frame, the number of lines 911 is set to be small even at a long distance. On the other hand, for the image 902 of the vehicle whose object is not detected at the time of the front frame, the number of lines 912 is set to be large so that the parallax calculation can be performed with higher accuracy. Further, for the road surface 903 in which the object has been detected, a smaller number of lines 913 is set.

In this embodiment, by setting the number of scan lines using the result of object detection in the previous frame, more accurate parallax calculation is performed for an image for which distance information has not been obtained in the past. be able to.

(Fourth Embodiment)
Next, a fourth embodiment of the present invention will be described. In this embodiment, an example of using the distance information measured by another sensor when determining the number of scan lines in the line number determination unit 111 will be described.

FIG. 8 is a block diagram showing a configuration of an image processing apparatus according to a fourth embodiment of the present invention. As shown in FIG. 8, the image processing apparatus 10 according to the present embodiment is an example of a sensor capable of acquiring distance information as compared with the second and third embodiments described with reference to FIGS. 4 and 6, respectively. The difference is that the distance information output from the LIDAR (Light Detection And Ranging) 40 is input to the line number determination unit 111. The LIDAR 40 in FIG. 8 is an example of another sensor capable of acquiring distance information, and the present invention is not limited to this. In addition, distance information output from, for example, an infrared depth sensor, an ultrasonic sensor, a millimeter wave radar, or another camera image may be used.

In the present embodiment, the line number determination unit 111 inputs distance information output from the LIDAR 40 with respect to the range of the stereo camera image captured by the camera 20, and based on this distance information, the same as in the first embodiment. In addition, the number of scan lines used in the smoothing process performed by the parallax calculation unit 102 is determined. That is, the distance of each pixel represented by the distance information is compared with a predetermined threshold value, the number of scan lines is set small for pixels whose distance is larger than the threshold value, and scans are performed for pixels whose distance is less than or equal to the threshold value. Set a large number of lines. Other than this, it is the same as the first to third embodiments.

(Fifth Embodiment)
Next, a fifth embodiment of the present invention will be described. In the present embodiment, instead of the line number determination unit 111 described in the first to fourth embodiments, an image processing apparatus having a sparse density determination unit 112 that determines the sparse density representing the ratio of pixels to be thinned in the smoothing calculation. An example of is described.

FIG. 9 is a block diagram showing a configuration of an image processing apparatus according to a fifth embodiment of the present invention. As shown in FIG. 9, in the image processing apparatus 10 according to the present embodiment, the condition setting unit 101 of the FPGA 11 determines the sparse density instead of the line number determination unit 111, as compared with the first embodiment described with reference to FIG. The difference is that it has a portion 112. The sparse density determination unit 112 reads out the parallax image generated by the parallax calculation unit 102 from the stereo camera image of the previous frame and stored in the parallax image storage unit 104, and the parallax calculation unit 102 executes the parallax image based on the parallax image. Determine the sparse density in the smoothing process. That is, the sparse density determination unit 112 of the present embodiment determines the sparse density based on the parallax calculated by the parallax calculation unit 102 in the past. Then, the determined sparse density is notified to the parallax calculation unit 102. The condition setting unit 101 in the present embodiment sets the conditions for the parallax calculation unit 102 by notifying the sparse density determined by the sparse density determination unit 112 in this way.

When the sparse density determination unit 112 notifies the sparse density, the parallax calculation unit 102 indicates the ratio of the notified sparse density to the stereo camera image of the current frame read from the stereo camera captured image storage unit 103. The pixels are thinned out with. Then, the smoothing calculation is performed using the remaining pixels of the stereo camera image that have not been thinned out as the calculation target pixels. After that, as in the first embodiment, the dissimilarity is calculated for each pixel with respect to the stereo camera image after the smoothing calculation, and the parallax for each pixel is calculated based on the calculation result. The parallax calculation unit 102 outputs the parallax calculation result for each pixel thus obtained to the processor 30 as a parallax image, and stores it in the parallax image storage unit 104.

FIG. 10 is an explanatory diagram of pixel thinning used for smoothing calculation in the image processing apparatus according to the fifth embodiment of the present invention. FIG. 10A is a diagram showing an example of pixels used in a pixel smoothing calculation corresponding to a long-distance image. For pixels that are judged to correspond to a long-distance image because the parallax represented by the parallax image of the previous frame is small, as shown in FIG. 10A, the pixels on each scan line are not thinned out. Make sure that dense smoothing operations are performed over a narrow area on the image. On the other hand, FIG. 10B is a diagram showing an example of pixels used in the pixel smoothing calculation corresponding to a short-distance image. For pixels that are judged to correspond to a short-distance image because the parallax represented by the parallax image of the previous frame is large, the pixels on each scan line are thinned out on the image as shown in FIG. 10 (b). The smoothing operation is performed using a small number of pixels in a wide range of. As described above, in the sparse density determination unit 112 of the present embodiment, the larger the parallax value obtained in the front frame, the closer the distance to the image is determined to be, and the proportion of pixels thinned from the stereo camera image increases. In addition, it is preferable to determine the sparse density.

Although the example in which the pixels used for the smoothing calculation are arranged on the scan line has been described with reference to FIG. 10, it is not necessary to limit the pixels to those arranged on the line. FIG. 11 is an explanatory diagram of the arrangement of pixels used for smoothing calculation in the image processing apparatus according to the fifth embodiment of the present invention. FIG. 11A shows an example in which each pixel arranged side by side on eight scan lines is used for smoothing calculation. On the other hand, FIG. 11B shows an example in which each pixel arranged according to a predetermined arrangement pattern, which is not arranged on the scan line, is adopted for the smoothing calculation. In both the arrangement examples of FIGS. 11 (a) and 11 (b), the pixels can be thinned out as described above based on the sparse density determined by the sparse density determination unit 112, and the smoothing operation can be performed. can. At this time, which pixel to be thinned out can be determined in advance. However, the arrangement of the pixels shown in FIG. 11 is an example, and the arrangement of the pixels used in the smoothing calculation is not limited to this.

Further, in the present embodiment, the sparse density determination unit 112 determines the number of scan lines in the line number determination unit 111 described in the first embodiment, based on the parallax calculated in the past by the parallax calculation unit 102. Although an example of determining the sparse density has been described, the present invention is not limited thereto. For example, similar to the determination of the number of scan lines in the second embodiment, the distance range information determined based on the parallax calculated in the past by the parallax calculation unit 102 is acquired from the processor 30, and the acquired distance range information is obtained. The sparse density may be determined based on. Further, similarly to the determination of the number of scan lines in the third embodiment, the information of the object detected based on the parallax calculated in the past by the parallax calculation unit 102 is acquired from the processor 30, and the information of the acquired object is obtained. The sparse density may be determined. Alternatively, similar to the determination of the number of scan lines in the fourth embodiment, the distance information corresponding to the range of the stereo camera image measured by another sensor is acquired, and the sparse density is determined based on the acquired distance information. You may. In addition to this, the sparse density can be determined by any method.

(Sixth Embodiment)
Next, a sixth embodiment of the present invention will be described. In the present embodiment, an example of an image processing apparatus in which the condition setting unit 101 has the line number determination unit 111 described in the first to fourth embodiments and the sparse density determination unit 112 described in the fifth embodiment. explain.

FIG. 12 is a block diagram showing a configuration of an image processing apparatus according to a sixth embodiment of the present invention. As shown in FIG. 12, in the image processing apparatus 10 according to the present embodiment, as compared with the first embodiment described with reference to FIG. 2, the condition setting unit 101 of the FPGA 11 determines the sparse density in addition to the line number determination unit 111. The difference is that it further has a portion 112. The sparse density determination unit 112 is the same as described in the fifth embodiment. Although FIG. 12 shows an example in which the memory 12 has the parallax image storage unit 104, the line number determination unit 111 and the sparse density determination unit 112 have the same methods as those in the second to fourth embodiments. When determining the number of scan lines and the sparse density, the parallax image storage unit 104 may be omitted without being provided in the memory 12. Further, as described in the fourth embodiment, when the number of scan lines and the sparse density are determined by using the distance information measured by another sensor, the distance information from the sensor is used as the line number determination unit. It may be input to 111 or the sparse density determination unit 112.

According to the embodiment of the present invention described above, the following effects can be obtained.

(1) The image processing device 10 calculates the parallax based on the stereo camera image taken by the camera 20. The image processing device 10 has a condition setting unit 101 for setting conditions for determining a calculation target pixel in the stereo camera image, and a calculation target pixel in the stereo camera image based on the conditions set by the condition setting unit 101. Is provided, and a parallax calculation unit 102 is provided for calculating the misparity using the information of the calculation target pixel. Since this is done, the amount of calculation required for parallax calculation can be suppressed.

(2) In each of the first to fourth and sixth embodiments, the condition setting unit 101 has a line number determination unit 111 that determines the number of scan lines for scanning a stereo camera image. The parallax calculation unit 102 sets the number of scan lines determined by the line number determination unit 111 for the stereo camera image, and determines the pixels of the stereo camera image on the scan lines as the calculation target pixels. Since this is done, it is possible to appropriately determine the calculation target pixel according to the conditions.

(3) In the first embodiment, the line number determination unit 111 determines the number of scan lines based on the parallax calculated by the parallax calculation unit 102 in the past. At this time, the line number determination unit 111 increases the number of scan lines as the parallax value becomes smaller. Since this is done, it is possible to determine the optimum number of scan lines according to the distance for each pixel from the past parallax information.

(4) In the second embodiment, the line number determination unit 111 acquires information on the distance range determined based on the parallax calculated in the past by the parallax calculation unit 102, and based on the acquired distance range information. , Determine the number of scanlines. Since this is done, it is possible to determine the optimum number of scan lines according to the distance of each object from the past parallax information.

(5) In the third embodiment, the line number determination unit 111 acquires information on an object detected based on the parallax calculated in the past by the parallax calculation unit 102, and scans based on the acquired object information. Determine the number of lines. Since this is done, the optimum number of scan lines can be determined for each object from the past parallax information.

(6) In the fourth embodiment, the line number determination unit 111 acquires distance information with respect to the range of the stereo camera image, and determines the number of scan lines based on the acquired distance information. Since this is done, even if the parallax cannot be calculated from the past stereo camera image, the optimum number of scan lines according to the distance for each pixel can be determined.

(7) In each of the fifth and sixth embodiments, the condition setting unit 101 has a sparse density determining unit 112 that determines the sparse density representing the ratio of pixels thinned out from the stereo camera image. The parallax calculation unit 102 determines the remaining pixels thinned out from the stereo camera image based on the sparse density determined by the sparse density determination unit 112 as the calculation target pixels. Since this is done, it is possible to appropriately determine the calculation target pixel according to the conditions.

(8) In each of the fifth and sixth embodiments, the sparse density determination unit 112 can determine the sparse density based on the parallax calculated by the parallax calculation unit 102 in the past. At this time, the sparse density determination unit 112 determines the sparse density so that the proportion of pixels thinned out from the stereo camera image increases as the parallax value increases. Since this is done, the optimum sparse density according to the distance for each pixel can be determined from the past parallax information.

(9) In each of the fifth and sixth embodiments, the sparse density determination unit 112 acquires information on the distance range determined based on the parallax calculated in the past by the parallax calculation unit 102, and the acquired distance range. It is also possible to determine the sparse density based on the information. It is also possible to acquire information on an object detected based on the parallax calculated by the parallax calculation unit 102 in the past, and determine the sparse density based on the acquired information on the object. Further, it is also possible to acquire distance information with respect to the range of the stereo camera image and determine the sparse density based on the acquired distance information. In this way, the optimum sparse density can be determined according to the situation.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope described in the claims. For example, although the image processing device 10 mounted on the vehicle and used in each of the above embodiments has been described, the present invention can also be applied to the image processing device used in other systems.

The embodiments and modifications described above are merely examples, and the present invention is not limited to these contents as long as the features of the invention are not impaired. Further, although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects considered within the scope of the technical idea of the present invention are also included within the scope of the present invention.

10 ... Image processing device, 11 ... FPGA, 12 ... Memory, 20 ... Camera, 30 ... Processor, 40 ... LIDAR, 101 ... Condition setting unit, 102 ... Parallax calculation unit (SGM calculation), 103 ... Stereo camera shooting image storage unit , 104 ... Parallax image storage unit, 111 ... Line number determination unit, 112 ... Sparse density determination unit

Claims (13)

An image processing device that calculates parallax based on a stereo camera image taken by a stereo camera.
For each pixel of the stereo camera image, a condition setting unit for setting a condition for determining a plurality of calculation target pixels corresponding to the pixel, and a condition setting unit.
Based on the conditions set by the condition setting unit, the plurality of calculation target pixels are determined for each pixel of the stereo camera image, and the pixels of the stereo camera image are used using the information of the plurality of calculation target pixels. It is equipped with a parallax calculation unit that calculates the misalignment for each .
The parallax calculation unit performs a smoothing calculation that averages the pixel value of each pixel of the stereo camera image and each pixel value of the plurality of calculation target pixels corresponding to the pixel, and performs the smoothing calculation. An image processing device that calculates the dissimilarity for each pixel of the stereo camera image and calculates the disparity based on the calculation result of the dissimilarity . In the image processing apparatus according to claim 1,
The condition setting unit has a line number determination unit that determines the number of scan lines that scan the stereo camera image in the smoothing calculation .
The parallax calculation unit sets the number of scan lines determined by the line number determination unit for each pixel of the stereo camera image, and sets each pixel of the stereo camera image on the scan line to the stereo camera image. An image processing device that determines the pixel to be calculated. In the image processing apparatus according to claim 2,
The line number determination unit is an image processing device that determines the number of scan lines based on the parallax calculated in the past by the parallax calculation unit. In the image processing apparatus according to claim 3,
The line number determining unit is an image processing device that increases the number of scan lines as the parallax value becomes smaller. In the image processing apparatus according to claim 2,
The line number determination unit acquires information on a distance range determined based on the parallax calculated in the past by the parallax calculation unit, and determines the number of scan lines based on the acquired information on the distance range. Image processing device. In the image processing apparatus according to claim 2,
The line number determination unit acquires information on an object detected based on the parallax calculated in the past by the parallax calculation unit, and determines the number of scan lines based on the acquired information on the object. Processing device. In the image processing apparatus according to claim 2,
The line number determination unit is an image processing device that acquires distance information with respect to the range of the stereo camera image and determines the number of scan lines based on the acquired distance information. The image processing apparatus according to any one of claims 1 to 7.
The condition setting unit has a sparse density determining unit that determines the sparse density representing the ratio of pixels thinned from the stereo camera image in the smoothing calculation .
Based on the sparse density determined by the sparse density determination unit, the parallax calculation unit determines a pixel to be thinned out among the pixels existing around the pixel for each pixel of the stereo camera image. An image processing device that determines the remaining pixels obtained by thinning out the pixels to be thinned out as the calculation target pixels. In the image processing apparatus according to claim 8,
The sparse density determination unit is an image processing device that determines the sparse density based on the parallax calculated by the parallax calculation unit in the past. In the image processing apparatus according to claim 9,
The sparse density determining unit is an image processing device that determines the sparse density so that the proportion of pixels thinned out from the stereo camera image increases as the value of the parallax increases. In the image processing apparatus according to claim 8,
The sparse density determination unit acquires information on a distance range determined based on the parallax calculated in the past by the parallax calculation unit, and determines the sparse density based on the acquired information on the distance range. Processing device. In the image processing apparatus according to claim 8,
The sparse density determination unit acquires information on an object detected based on the parallax calculated in the past by the parallax calculation unit, and determines the sparse density based on the acquired information on the object. .. In the image processing apparatus according to claim 8,
The sparse density determining unit is an image processing device that acquires distance information with respect to the range of the stereo camera image and determines the sparse density based on the acquired distance information.

Images