JP2021131377A - Generation device, method for generation, and program - Google Patents

Abstract

To reduce a memory band necessary to calculate distance information.SOLUTION: The present invention relates to a generation device for generating distance information showing a distance to an object, and the generation device includes: generation means for generating a distance image on the basis of a pair of taken images having a parallax; and correction means for correcting a distortion in a distance image resulting from an optical system of imaging means which generated a pair of taken images for a region with a reliability of at least a threshold value of the distance image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a distance measuring technique for calculating a distance to a subject.

In recent years, distance measurement technology has been used for various purposes such as automatic driving and FA (Factory Automation), and there are various distance measurement technologies. As a distance measuring technique, for example, there is a technique of using an image captured by a stereo camera. In this method, a stereo camera is used to simultaneously image a subject from different directions by a plurality of imaging devices, and distance information to the subject is calculated based on the parallax of the obtained plurality of captured images (Patent Document 1).

As another distance measuring technology, there is a technology in which pixels having a distance measuring function (distance measuring pixels) are arranged in a part or all of the pixels of one image sensor, and the distance to the subject is calculated by using a phase difference method. There is (Patent Document 2). In this method, a plurality of photodiodes are arranged for one distance measuring pixel, separate captured images (called A image and B image, respectively) are generated, and the subject is reached based on the amount of positional deviation of the captured images. Calculate the distance of.

There is also a distance measuring method that does not use an image sensor. For example, in the TOF (Time Of Flight) method, the distance to the subject is measured by irradiating the subject with light and detecting the time until the light is reflected and returned.

In a distance measuring device using these distance measuring techniques, distortion due to an optical factor occurs. For example, in a distance measuring device using an image sensor, distortion occurs in the captured image due to distortion of the lens, and when the distance is calculated based on the captured image including the distortion and a distance image is generated, the distance image is also distorted. Will occur. Therefore, it is common to perform distortion correction on the captured image and generate a distance image based on the image after distortion correction, or perform distortion correction on the generated distance image.

There are various methods for distortion correction, but in each method, it is necessary to refer to the pixels of the uncorrected image corresponding to each pixel of the corrected image and the pixel values around it.

JP-A-2018-105682 Japanese Patent No. 5192096

However, the memory bandwidth when reading the pixel value of the image before correction from the memory for distortion correction increases as the correction method has higher accuracy, so that the memory bandwidth required for calculating the distance information with high accuracy increases. However, there is a problem of causing a memory bandwidth shortage.

The present invention is a generation device that generates a distance image representing a distance to a subject, and is a generation means that generates a distance image based on a pair of captured images having a parallax, and a reliability of the distance image is a predetermined threshold value. It is characterized in that the above-mentioned region is provided with a correction means for correcting distortion in a distance image caused by the optical system of the imaging means that generated a pair of captured images.

INDUSTRIAL APPLICABILITY The present invention can reduce the memory bandwidth required for calculating the distance information.

It is a figure which shows the structure of the distance measuring apparatus in Embodiment 1. FIG. It is a figure which shows the processing flow in Embodiment 1. FIG. It is a figure which shows the example of the received image in Embodiment 1. FIG. It is a figure which shows the example of the band information necessary for the distortion correction method in Embodiment 1. It is a figure explaining the distortion correction method in Embodiment 1. FIG. It is a figure which shows the structure of the distance measuring apparatus in Embodiment 2. It is a figure which shows the processing flow in Embodiment 2. It is a figure which shows the structure of the distance measuring apparatus in Embodiment 3. FIG. It is a figure which shows the structure of the cache in Embodiment 3. It is a figure explaining the reliability in Embodiment 3. It is a figure explaining the distortion correction process in Embodiment 3. It is a figure which shows the structure of the distance measuring apparatus in Embodiment 4. It is a figure which shows the example of the distortion correction method and the band information of an additional process in Embodiment 4. It is a figure which shows the structure of the distance measuring apparatus in Embodiment 5. It is a figure which shows an example of the processing flow in Embodiment 5. It is a figure explaining the image shape deformation processing method in Embodiment 5. It is a figure which shows the structure of the distance measuring apparatus which concerns on Embodiment 6. It is a figure which shows another example of the structure of the distance measuring apparatus which concerns on Embodiment 6. It is a figure which shows an example of the processing flow in Embodiment 6. It is a figure which shows an example of the processing flow in Embodiment 7.

Hereinafter, embodiments of the ranging device of the present invention will be described with reference to the drawings. In the following embodiment, as the distance measuring method, a method of arranging a plurality of photodiodes for one distance measuring pixel and calculating the distance using the phase difference method and a method using a stereo camera will be described as an example. The distance measuring method to which the present invention can be applied is not limited to the above. It can be applied to all distance measuring devices that generate distortion due to an optical system such as TOF. Further, in the present embodiment, the storage destination of the captured image and the distance image is set to DRAM (Dynamic Random Access Memory), but other memories may be used. In that case, it is necessary to provide the band information corresponding to the storage destination memory of the distance image as the memory band information required for each distortion correction described later.

(Embodiment 1)
The distance measuring device according to the first embodiment of the present invention will be described. In the first embodiment, a plurality of photodiodes are arranged at intervals of several μm for one ranging pixel, and a pair of captured images having parallax generated by an imaging device having at least one common optical system On the other hand, a method of calculating the distance using the parallax method is applied. When two pairs of captured images are acquired from one optical system in this way, the amount of distortion of the two captured images is close, so distortion correction is performed after first generating a distance image from the two captured images. The configuration is suitable. By doing so, since the distortion correction needs to be performed only for one distance image, the DRAM for reading the pixel value accompanying the distortion correction is compared with the case where the distortion correction is performed for each of the pair of captured images. The access band (DRAM band) can be reduced.

FIG. 1 is a hardware configuration diagram of the distance measuring device according to the first embodiment. Reference numeral 100 denotes a distance measuring device of the present embodiment, which generates a distance image of a subject to which distortion correction has been performed. Reference numeral 101 denotes an image pickup unit, which captures an image of a subject and generates image data. Reference numeral 102 denotes a distance image generation unit, which receives image data from the imaging unit 101 and generates a distance image.

Next, the internal configuration of the imaging unit 101 will be described. Reference numeral 103 denotes a lens, which collects the light reflected from the subject and guides it to the image pickup device 104 to form an image on the image pickup surface. Reference numeral 104 denotes an image sensor, which is an image sensor such as a CMOS sensor or a CCD sensor. The image sensor 104 converts the light received through the lens 103 into an electric signal. Reference numeral 105 denotes an image transmission unit, which transmits the electric signal generated by the image sensor 104 to the distance image generation unit 102 as image data.

Next, the internal configuration of the distance image generation unit 102 will be described. Reference numeral 106 denotes an image receiving unit, which receives image data transmitted from the imaging unit 101. Reference numeral 107 denotes an image correction unit, which performs necessary correction processing on the image data received by the image receiving unit 106 as a preprocessing for generating a distance image. At this time, in the image data caused by the optical system of the imaging unit 101. Distortion correction to correct distortion is not included. Reference numeral 108 denotes a parallax calculation unit, which calculates the parallax for each pixel of the corrected image corrected by the image correction unit 107. Reference numeral 109 denotes a distance calculation unit, which converts the parallax data calculated by the parallax calculation unit 108 into a distance image. Reference numeral 110 denotes a DRAM, and in the present embodiment, the distance image generated by the distance calculation unit 109 is stored. Here, the distance image generated by the distance calculation unit 109 and stored in the DRAM 110 is a distance image before distortion correction in the present embodiment. Further, it is assumed that the DRAM 110 stores in advance information on the coordinate values of the uncorrected image (hereinafter referred to as coordinate conversion map information) corresponding to each pixel in the corrected image in the distortion correction. Reference numeral 111 denotes a reliability calculation unit, which calculates the reliability for each arbitrary pixel unit in the corrected image corrected by the image correction unit 107. In FIG. 1, the reliability calculation unit 111 acquires the image data via the DRAM 110, but the image data may be acquired directly from the image correction unit 107 without going through the DRAM 110. Reference numeral 112 denotes SRAM, which stores the reliability calculated for each pixel in the present embodiment. The reliability may be calculated not for each pixel but for each of a plurality of pixels, and the storage destination of the reliability does not necessarily have to be SRAM and may be another memory. Reference numeral 113 denotes a memory access processing content selection unit. In the present embodiment, the distortion correction method is based on the reliability calculated by the reliability calculation unit 111 and the memory access band required by each preset distortion correction method. Select. In this embodiment, selection is made based on the DRAM band. Reference numeral 114 denotes a distortion correction necessity determination unit, which determines whether or not to perform distortion correction according to the reliability of the pixels of the uncorrected image corresponding to each pixel of the image after distortion correction. At this time, the coordinate values of the pixels of the uncorrected image corresponding to each pixel of the image after distortion correction are acquired by referring to the coordinate conversion map stored in the DRAM 110 in advance. Reference numeral 115 denotes a distortion correction unit, which performs distortion correction of pixels determined to require distortion correction by the distortion correction necessity determination unit 114 by the method selected by the memory access processing content selection unit 113, and the corrected pixels. Output the value. Reference numeral 116 denotes a mask unit, and a predetermined value is set for the pixels of the image after distortion correction determined by the distortion correction necessity determination unit 114 to be output.

Next, a detailed explanation of each part will be described with reference to the flowchart. FIG. 2 is a flowchart showing a processing flow of the distance image generation unit 102 in the distance measuring device 100 of the present embodiment. The distance image generation unit 102 can be realized by an information processing device, and configurations other than the DRAM 110 and the SRAM 112 can be realized by executing the program by one or more processors. The same applies to the distance image generation unit of other embodiments described later.

In S201, the image receiving unit 106 receives the image data from the imaging unit 101. In this embodiment, it is assumed that an image as shown in FIG. 3A is received.

In S202, the image correction unit 107 performs necessary correction processing other than distortion correction on the received image. Examples of the correction process include scratch correction when the sensor has scratches, and shading correction for correcting brightness unevenness caused by a decrease in brightness as the distance from the center of the lens increases. However, the correction performed here is not limited to the above.

In S203, the parallax calculation unit 108 calculates the parallax for each pixel of the image corrected in S202. Here, a parallax calculation method called template matching will be described. It is assumed that the image pickup unit 101 has an image pickup element 104 having a plurality of photodiodes in one pixel, and one of the captured images (A image, B image) obtained from the plurality of photodiodes in one pixel is used as a reference image. And the other is the reference image. Then, an area of an arbitrary size (called a collation area) centered on the point of interest is set in the reference image, and the images in the collation area are sequentially moved in the reference image to have the highest degree of similarity in the reference image. Identify high areas. The distance between the point of interest in the collation area in the reference image and the center point of the area having the highest degree of similarity to the collation area in the reference image is calculated as parallax. Here, any value can be set for the size of the collation area and the number of steps (unit movement amount) of the collation area in the reference image. In addition, any method can be applied to the method of obtaining the similarity between images. For example, SSD (Sum of Squared Difference), which evaluates similarity by the sum of squares of pixel value differences, and SAD (Sum of Absolute Difference), which evaluates similarity by the sum of absolute values of pixel value differences, can be used. Is.

In S204, the distance calculation unit 109 calculates the distance for each pixel based on the parallax calculated in S203. The distance is calculated from the parallax by the principle of triangulation, and a distance image before distortion correction is generated.

In S205, the reliability calculation unit 111 calculates the reliability for each predetermined pixel unit in the image corrected in S202. In the present embodiment, the reliability is calculated in units of one pixel, but the reliability may be calculated in units of any pixel. Further, in the present embodiment, the reliability is calculated by 1 bit, but the reliability may be calculated by any number of bits. The reliability is calculated based on factors that affect the distance measurement accuracy, but in the calculation of the distance using the phase difference method in the present embodiment, the reliability is calculated based on the amount of change in contrast. The amount of change in contrast can be calculated from the dispersion of the pixel values of the point of interest and the pixels around it. In the present embodiment, with respect to the input image of FIG. 3 (a), the region 300 (the region corresponding to the sky) of FIG. 3 (b) is calculated as the reliability 0, and the other regions are calculated as the reliability 1. It is assumed that the reliability calculation algorithm has been adjusted. Here, the threshold value is set to 1, and 0 below the threshold value means low reliability, and 1 above the threshold value means high reliability.

Further, in the present embodiment, the calculated reliability is stored in SRAM (Static Random Access Memory) in association with the coordinate values so that it can be read at high speed, but it may be stored in other memory. .. Further, in the present embodiment, the region 300 having a reliability of 0 is about 14% of the region of the entire image.

In S206, the memory access processing content selection unit 113 selects the distortion correction method based on the reliability calculated in S204 and the DRAM band required for each of the preset distortion correction methods. In the present embodiment, the distortion correction method is selected from the three distortion correction methods of nearest neighbor, bilinear, and bicubic according to the ratio of the region having low reliability. However, the types of distortion correction methods are not limited to these. Further, any method may be used as the method of giving the information of the DRAM band required for each distortion correction method. For example, an input unit by the user may be prepared, or may be stored in a memory such as DRAM or SRAM in advance. In this embodiment, the band information as shown in the table shown in FIG. 4 is given, but this format is just an example, and any format may be used as long as the band required for each distortion correction method is known. The 400 columns in the table shown in FIG. 4 represent the types of distortion correction methods.

Here, three distortion correction methods of nearest neighbor, bilinear, and bicubic will be described. FIG. 5A is a diagram illustrating distortion correction.

Reference numeral 500 denotes an image after distortion correction, and it is assumed that the pixel at the point of interest 503 is now corrected. Therefore, first, the coordinate value 504 of the pixel of the uncorrected image corresponding to the point of interest 503 (hereinafter, referred to as the uncorrected coordinate value) is referred to. In the present embodiment, the uncorrected coordinate value 504 is acquired by referring to the value corresponding to the corrected coordinate value from the coordinate conversion map 502 stored in the DRAM 110 in advance, but is stored in a memory other than the DRAM 110. You may keep it. The coordinate conversion map 502 is calculated in advance based on the coordinate conversion formula according to the distortion of the lens 103, but the method of acquiring the coordinate value 504 before correction is not limited to the above method. The method of acquiring the coordinate value 504 before correction may be a method of calculating the coordinate value 504 before correction one by one in the distance measuring device 100 by the above coordinate conversion formula.

Reference numeral 501 denotes a pre-correction image, and the coordinate position of the pre-correction coordinate value 504 corresponding to the point of interest 503 is represented by 505 (hereinafter, referred to as a pre-correction coordinate position). However, the uncorrected coordinate value 504, which is generally obtained by the coordinate conversion formula, includes a decimal point. Therefore, in Nearest Neighbor, as shown in FIG. 5B, the pixel value of the pixel 508, which is the pixel closest to the uncorrected coordinate position 505 in the uncorrected image 501, is set as the corrected pixel value at the point of interest 503. .. On the other hand, in bilinear, the value interpolated from the pixel values of the pixels 506, 507, 508, and 509 at the four points around the uncorrected coordinate position 505 in the uncorrected image 501 is used as the corrected pixel value at the point of interest 503. Further, in bicubic, the value interpolated from the 16 pixel values around the uncorrected coordinate position 505 in the uncorrected image 501 is used as the corrected pixel value of the point of interest 503. Therefore, in the nearest neighbor, only one pixel value needs to be read from the DRAM 110, but in bilinear, it is necessary to read four pixel values, and in bicubic, it is necessary to read 16 pixel values, which is distorted. The DRAM band increases depending on the type of correction method. On the other hand, the accuracy of distortion correction is higher in bilinear than in near rest neighbors and in bicubic than in bilinear.

Next, the column 401 of FIG. 4 represents an example of the required band of the DRAM 110 of each distortion correction method. As described above, since it is necessary to read the pixel values of 4 times the number of pixels for bilinear and 16 times the number of pixels for bicubic with respect to the nearest neighbor, when all the pixel values are read from the DRAM 110 each time, the required bandwidth is also required. It will be 4 times and 16 times, respectively. However, for example, some of the pixel values of the four uncorrected images 501 used in the distortion correction of a certain pixel in bilinear are likely to be referred to in the distortion correction of the adjacent pixel. Therefore, in general, the DRAM 110 is provided with a cache, and the pixel values once read from the DRAM 110 are temporarily stored in the cache for a predetermined period. As a result, when the pixel value of the same pixel is read again within a certain period of time, the DRAM band is reduced by reading from the cache without accessing the DRAM 110. How much bandwidth can be reduced depends on the hit rate of the cache. In the present embodiment, it is assumed that the DRAM 110 is accessed in tile units (area units composed of arbitrary numbers of pixels in the vertical and horizontal directions), and the distortion of the lens and the pixels referred to in each distortion correction method are used. The cache hit rate is estimated from the area. The required band 401 of the DRAM 110 shown in FIG. 4 exemplifies a value calculated based on such an estimation.

The column 402 represents the band increase rate of each distortion correction method with respect to the nearest neighbor. Looking at 402, the bilinear band rise rate is 12%, and the bicubic band rise rate is 40%. Here, the memory access processing content selection unit 113 first refers to the reliability calculated in S204 to obtain the ratio of the region where the reliability is 0 with respect to the entire image. Here, it is assumed that it is calculated as 14%. Although the details will be described later, the region where the reliability is 0 is determined as a pixel that does not require distortion correction in the subsequent step. Therefore, the fact that the region where the reliability is 0 is 14% means that the number of pixels for distortion correction can be reduced by 14%, and the required band for reading the pixel value from the DRAM 110 can be reduced by 14% accordingly. That is, when the ratio of the region with zero reliability is 14%, the required bands of nearest neighbor, bilinear, and bicubic are 4.3MB / Frame, 4.8MB / Frame, and 6.0MB / Frame, respectively.

Next, the band increase rate of each distortion correction method in FIG. 4 is referred to, and which distortion correction method is applicable is determined using the band that can be reduced. In the present embodiment, the required band of 5.0 MB / Frame of the nearest neighbor when performing distortion correction in the entire area is set as the maximum allowable band, and the required band is the maximum allowable band or less (the maximum allowable access band or less). The correction method shall be adopted. As described above, when the band reduction rate is 14%, the bilinear is 4.8 MB / Frame and the bicubic is 6.0 MB / Frame, so the required band is 5.0 MB / Frame or less and the most accurate distortion correction method. The bilinear distortion correction method is selected. For example, when the band reduction rate is less than 10%, the required band of bilinear exceeds 5.0 MB / Frame, so the nearest neighbor distortion correction method is selected. When the bandwidth reduction rate is 28% or more, the required bandwidth of bicubic is 5.0 MB / Frame or less, so that the bicubic distortion correction method is selected.

As described above, in the present embodiment, the distortion correction method is selected based on the DRAM band required for each distortion correction method and the predetermined maximum allowable band in the region excluding the region having low reliability as described above.

In S207, the distortion correction necessity determination unit 114 determines whether or not it is necessary to perform distortion correction for each pixel of the distance image generated in S204. In S207, first, the uncorrected coordinate value 504 for the pixel of interest in the corrected image is read from the DRAM 110, as in the case of calculating the reliability described above. Next, the reliability corresponding to the pixel closest to the uncorrected coordinate position 505 corresponding to the read pixel of interest is read from the SRAM 112. If the reliability is 0, it is determined that distortion correction is unnecessary and the process proceeds to S209. If the reliability is 1, it is determined that distortion correction is necessary and the process proceeds to S208.

Next, in S208, the distortion correction unit 115 performs distortion correction on the pixel of interest determined in S207 that distortion correction is necessary by the distortion correction method selected in S206. In the present embodiment, since the bilinear distortion correction method is selected, the bilinear distortion correction is performed. At this time, since the distortion correction unit 115 receives the uncorrected coordinate value 504 read from the DRAM 110 by the distortion correction necessity determination unit 114, the distortion correction unit 115 does not need to read the uncorrected coordinate value 504 from the DRAM 110.

On the other hand, in S209, the mask unit 116 assigns a predetermined value to the pixel determined in S207 that distortion correction is unnecessary without performing distortion correction. Since the distortion correction is not performed on the pixel determined that the distortion correction is unnecessary, the process of reading the pre-correction pixel value from the DRAM 110 for the distortion correction becomes unnecessary, and the DRAM band can be reduced. Further, the allowable DRAM band, that is, the DRAM band reduced from 5.0 MB / Frame in the case of the present embodiment, can be used for the distortion correction in S208.

Next, in S210, the distortion correction unit 115 or the mask unit 116 outputs a distortion-corrected pixel value or a pixel value masked with a predetermined value. In the present embodiment, the distortion correction unit 115 and the mask unit 116 are configured to output the distortion-corrected pixel value or the masked pixel value to the outside of the distance measuring device 100, but the memory inside the distance measuring device 100, for example, It may be stored in the DRAM 110.

In this way, from S207 to S210, the pixel value for one pixel in the distance image can be generated and output.

In S211 it is determined whether distortion correction or masking has been performed on all the pixels of the distance image, and if it still remains, the process returns to S207. By repeatedly performing such a step for all the pixels, it is possible to output a distance image after distortion correction.

Next, in S212, if reception of the image is not stopped based on, for example, user input, the process returns to S201 and the next image is received. The above processing is always executed until the image reception is stopped.

In the case of acquiring two captured images from one optical system in this way, it is preferable to apply this embodiment with a configuration in which a distance image is first generated from the two captured images and then distortion correction is performed. By doing so, by performing the distortion correction only on one distance image instead of the two captured images, the pixel value read for the distortion correction is halved, so that the DRAM band can be reduced. In addition to this, since the distortion correction processing is not performed in the region with low reliability, the DRAM band can be further reduced. As a result, it is possible to select the distortion correction method of the region to be distortion-corrected according to the band reduction amount. As a result, the reduced band can be assigned to the distortion correction of the region to be distortion-corrected, and more accurate distortion correction becomes possible.

Further, in the present embodiment, the reliability is set to 2 values of 0 or 1, but may have a value of 3 values or more. For example, when the reliability has three values of 0, 1, and 2, the distortion correction is not performed when the reliability is 0, and the distortion is corrected by different distortion correction methods when the reliability is 1 and 2. You may.

(Embodiment 2)
In the first embodiment, the distance image is generated before the distortion correction is performed, and then the distance image is subjected to the distortion correction. However, in the second embodiment, the distortion correction is performed before the distance image is generated, and then the distance image is generated. Generate. In the second embodiment, a distance measuring device using a stereo camera as an imaging device will be described. When acquiring captured images using multiple separate optical systems that are separated by several centimeters to a dozen centimeters, such as in a stereo camera, the amount of distortion of the captured image differs for each optical system, so there are multiple. It is preferable to perform distortion correction on the captured image of the above and then generate a distance image. If the parallax is calculated between captured images with different amounts of distortion, the error will be large. Therefore, by first performing the distortion correction of the captured image and then calculating the parallax to generate the distance image, a highly accurate distance image can be obtained. can get.

FIG. 6 is a configuration diagram of a distance measuring device showing the second embodiment. Reference numeral 600 denotes a distance measuring device of the present embodiment, which generates a distance image of a subject. The distance measuring device 600 includes two imaging units 601a and 601b and a distance image generation unit 602. Since the processing of the imaging units 601a and 601b is the same as that of the imaging unit 101 of the first embodiment, the details will be omitted.

The distance image generation unit 602 receives a pair of captured images composed of image data a and image data b having a predetermined parallax from the two imaging units. The received captured image data is received by the two image receiving units 606a and 606b, and then various image corrections including distortion correction are performed by the two image correction units 607a and 607. Since the processing of the image receiving units 606a and 606b and the image correction units 607a and 607 is the same as that of the first embodiment, the details will be omitted.

The two image correction units 607a and 607 obtain a pre-distortion image captured image a and a pre-distortion image image b, and these pre-distortion image images are passed to the DRAM 610 and the reliability calculation unit 611. The two pre-distortion captured images a and b passed to the DRAM 610 are referred to at the time of distortion correction. Further, it is assumed that the coordinate conversion map information for distortion correction is stored in the DRAM 610 in advance. Here, since the amount of distortion of each captured image obtained by the two imaging units is different, two coordinate conversion maps corresponding to the amount of distortion of each captured image are stored.

On the other hand, the reliability calculation unit 611 calculates the reliability for each of the two received images before distortion correction. Since the reliability calculation unit 611 is the same as the reliability calculation unit 111 of the first embodiment, the details will be omitted. The calculated reliability data is stored in SRAM 612.

Subsequent processing of the memory access processing content selection unit 613, the distortion correction necessity determination unit 614, the distortion correction unit 615, and the mask unit 616 processes the two captured images. The processing for each captured image by these is the same as the memory access processing content selection unit 113, the distortion correction necessity determination unit 114, the distortion correction unit 115, and the mask unit 116 of the first embodiment, and thus the details will be omitted.

Next, a detailed explanation of each part will be described with reference to the flowchart. FIG. 7 is a flowchart showing a processing flow of the distance image generation unit 602 in the distance measuring device 600 of the present embodiment.

Each step other than S710 has a different arrangement order from the flowchart of FIG. 2, but performs the same processing as each corresponding step. S701 and S702 correspond to S201 and S202, S703 to S709 correspond to S205 to S211, S711 and S712 correspond to S203 and S204, and S713 corresponds to S212. In the present embodiment, S710 is added to perform distortion correction on the image captured before distortion correction a and the image captured before distortion correction b.

In these processes, by not performing distortion correction in a region with low reliability as in the first embodiment, the pixel value to be read is reduced to reduce the DRAM band, and the accuracy is higher within the limited maximum allowable band. Distortion correction can be performed.

In this way, two distortion-corrected images obtained with distortion correction are obtained with respect to the captured images obtained from the two imaging units, and the parallax calculation unit 617 obtains the parallax from these distortion-corrected images. Further, a distance image is obtained by the distance calculation unit 618. Since the processing contents of the parallax calculation unit 617 and the distance calculation unit 618 are the same as those of the parallax calculation unit 108 and the distance calculation unit 109 of the first embodiment, the details will be omitted.

In this way, in a distance measuring device that acquires a plurality of captured images and generates a distance image using a plurality of separate optical systems such as a stereo camera, the amount of distortion of the captured image differs for each optical system. It is preferable to perform distortion correction before generating a distance image as in the present embodiment. By doing so, the parallax can be calculated between the captured images correctly corrected for distortion, and a highly accurate distance image can be obtained. In addition, the DRAM band can be reduced by not performing distortion correction in a region having low reliability, and a distortion correction method applied to the region to be strain-corrected can be selected according to the amount of bandwidth reduction. As a result, the reduced band can be assigned to the distortion correction of the region to be distortion-corrected, and more accurate distortion correction becomes possible.

(Embodiment 3)
In the first embodiment, the reliability is calculated for each pixel, but in the third embodiment, one reliability is generated for each of a plurality of pixels according to the cache size. By doing so, the hit rate of the cache can be improved and the DRAM bandwidth can be reduced. Also, the size of the SRAM that stores the reliability can be reduced. In the third embodiment, as in the first embodiment, a method of arranging a plurality of photodiodes for one ranging pixel and calculating the distance by using the phase difference method is applied. Therefore, the distortion is corrected after the distance image is generated. However, even in the configuration in which the distance image is generated after the distortion is corrected as in the second embodiment, one reliability can be generated for each of a plurality of pixels according to the cache size.

FIG. 8 is a block diagram of the third embodiment. On the configuration diagram, a cache 817 is added to the configuration of FIG. An example of the structure of the cache 817 will be described with reference to FIG. The cache 817 has a configuration having a plurality of tile-shaped buffers 900 (hereinafter referred to as tile buffers). Further, one tile buffer can have data for a total of 16 pixels of 4 vertical pixels and 4 horizontal pixels, and the cache is replaced in units of tile buffers. 901 means one pixel. The sizes of 4 pixels in the vertical direction and 4 pixels in the horizontal direction are examples, and the vertical and horizontal sizes and the number of tile buffers can be set to any number.

Next, the processing flow of the present embodiment will be described with reference to the flow chart of FIG.

Since S201 to S204 are the same as those in the first embodiment, the details will be omitted.

Next, in S205, the reliability calculation unit 111 calculates the reliability for each pixel size (referred to as tile buffer size) region of one tile buffer in the cache 817. In this embodiment, one reliability is calculated in a region of 4 vertical pixels and 4 horizontal pixels. There are various methods for calculating the reliability for each tile buffer size area, but in this embodiment, after calculating all the reliability for each pixel, even one within the tile buffer size area has the reliability. If there is a pixel with 1, the reliability of that area is 1. FIG. 10 shows an image of the reliability generated when the input image is shown in FIG. 3A. One small square area 1000 in FIG. 10 is one tile buffer size area, and one reliability is generated for each of these areas. In the present embodiment, the region surrounded by the thick line such as 1001 is the region with zero reliability. Here, by calculating the reliability for each tile buffer size, the amount of reliability data can be reduced, and the SRAM size required for storing the reliability can be reduced.

Since S206 is the same as that of the first embodiment, the details will be omitted.

Next, S207 to S209 will be collectively described with reference to FIG. Reference numeral 1100 is an image after distortion correction, and first, distortion correction of the point of interest 1 (1101) is performed. The uncorrected coordinate value 1104 at the point of interest 1 is read from the coordinate conversion map 1103. Then, the reliability of the tile buffer size area corresponding to the uncorrected coordinate value 1104 is read. Here, it is assumed that the reliability is 1. If Nearest Neighbor is selected as the distortion correction method, the pixel value on the pre-distortion image 1106 closest to the pre-correction coordinate position 1107 corresponding to the pre-correction coordinate value 1104 is read from the DRAM 110. .. At this time, if the corresponding pixel value is not contained in the cache 817, the data of the tile buffer size area 1109 including the corresponding pixel value is stored in the tile buffer of the cache 817, and the required pixel value is read from the tile buffer of the cache 817. ..

Next, distortion correction is performed for the point of interest 2 (1102). Here, the reliability of the tile buffer size area corresponding to the coordinate value 1105 before correction at the point of interest 2 is read, but since the point of interest 2 is next to the point of interest 1, it corresponds to the point of interest 1 and the point of interest 2. The tile buffer size area may be the same. Here, it is assumed that the tile buffer size area corresponding to the point of interest 1 and the point of interest 2 are the same, and the reliability of the point of interest 2 in the same tile buffer size area 1109 as that of the point of interest 1 is read, and 1 is set as the reliability. obtain. Since the reliability is 1, the distortion correction unit 115 tries to read the pixel value closest to the pre-correction coordinate position 1108 corresponding to the pre-correction coordinate value 1105 in order to perform distortion correction. Since the reliability in the tile buffer size area 1109 which is the same as the point of interest 1 is referred to here, the pixel value to be read is also in the tile buffer size area 1109 which is the same as the point of interest. Then, since the tile buffer size area 1109 is already in the tile buffer of the cache 817, the pixel value can be read from the cache 817 without accessing the DRAM 110 (cache hit).

In the above, the distortion correction method of the nearest neighbor has been described, but even in bilinear or bicubic, since the pixel values in the vicinity of the coordinate position before correction are collectively read into the cache 817, the possibility of a cache hit is also high. In this way, by generating the reliability for each tile buffer size area of the cache, the cache hit rate is increased, and the DRAM data can be read efficiently. Further, by calculating the reliability for each tile buffer size, the amount of reliability data can be reduced, and the SRAM size required for storing the reliability can be reduced.

In FIG. 8, the reliability calculation unit 111 acquires the image data via the DRAM 110, but the image data may be acquired directly from the image correction unit 107 without going through the DRAM 110.

(Embodiment 4)
In the first to third embodiments, the DRAM band associated with the distortion correction is reduced by not performing the distortion correction for the pixel having the reliability of 0, and the reduced band can be used for the distortion correction of the pixel with high reliability. .. In the fourth embodiment, an example of performing additional processing for increasing the added value of the distance measuring device using the reduced band will be described.

FIG. 12 is a configuration diagram of a distance measuring device showing the embodiment of the fourth embodiment. In FIG. 1 (Embodiment 1), since the distortion correction method was selected by the memory access processing content selection unit 113, the selection result was passed to the distortion correction unit 115. On the other hand, in FIG. 12 (Embodiment 4), since the memory access processing content selection unit 1213 selects additional processing different from the distortion correction, it does not need to be connected to the distortion correction unit 115. Instead, the additional processing to be executed is performed according to the preset DRAM band required for distortion correction, the DRAM band required for the additional processing selected by the memory access processing content selection unit 1213, and the ratio of the area having zero reliability. select.

FIG. 13A shows band information required for the distortion correction method used. In the present embodiment, the bilinear distortion correction method is used, but other distortion correction methods may be used. Next, the ratio of the area with zero reliability stored in the SRAM is calculated. Here, it is assumed that 23% is a region with zero reliability. Then, it is possible to reduce the bandwidth of 5.6 MB / Frame, which is the required bandwidth (1301) of the DRAM in bilinear distortion correction, that is, 1.3 MB / Frame.

Next, information on additional processing selected by the memory access processing content selection unit 1213 is shown in FIG. 13 (b). Here, as additional processing 1302, two types of object recognition processing and object tracking processing are described, and the required bands 1303 of these DRAMs are 1MB / Frame and 3MB / Frame, respectively.

Next, select additional processing that can be performed in the reduceable bandwidth. In this embodiment, since the band of 1.3MB / Frame can be reduced, only object recognition is selected. If the bandwidth can be reduced by 4MB / Frame or more, both object recognition and object tracking are selected.

In the present embodiment, there are only two types of additional processing of the selected candidate, but there may be three or more types. In addition, the algorithm for determining which additional processing is selected may be any method. Further, any method may be used as the distortion correction method or the method of giving the DRAM band information required for the additional processing. For example, an input unit by the user may be prepared so that the information can be acquired from the user input, or may be stored in a memory such as DRAM or SRAM in advance. In the present embodiment, the band information is given in the format shown in FIGS. 13 (a) and 13 (b), but this format is just an example and may be another format.

In this way, the object recognition process selected according to the bandwidth reduction amount is executed. By combining the result of object recognition and the generated distance image, it is possible to link the distance information between a specific object (human, animal, vehicle, etc.) and the object. In addition, the trajectory and speed of the object can be calculated by handling the acquired distance information and the recognition result in time series between a plurality of frames.

As described above, the DRAM band can be reduced by not performing the distortion correction in the region with low reliability, and the reduced band can be used to execute additional processing for increasing the added value of the distance measuring device.

In FIG. 12, the reliability calculation unit 111 acquires the image data via the DRAM 110, but the image data may be acquired directly from the image correction unit 107 without going through the DRAM 110.

(Embodiment 5)
In the first to fourth embodiments so far, the DRAM band associated with the distortion correction is reduced by not performing the distortion correction for the pixels having a reliability of 0 or the pixels having a low reliability when the reliability is multi-valued. ..

On the other hand, in the fifth embodiment, control using reliability is performed even in processing involving DRAM access other than distortion correction, such as enlargement / reduction of a display area, cutting out a part of the area, and multi-screen display in which a plurality of areas are specified. This reduces the DRAM bandwidth used.

FIG. 14 is a diagram showing a configuration of a distance measuring device according to the fifth embodiment. Functions that overlap with the above-described embodiments are designated by the same reference numerals.

Reference numeral 1400 is a distance measuring device according to the present embodiment, which displays a designated image area and generates a distance image of a subject to which distortion correction has been performed. Reference numeral 1401 is a distance image generation unit, which receives image data from the imaging unit 101 and generates a distance image. The distance image generation unit 1401 outputs an image (hereinafter referred to as a display image) obtained by performing not only distortion correction but also processing such as cropping and enlargement / reduction on the distance image so that a predetermined image area of the distance image is output. Can be output.

The internal configuration of the distance image generation unit 1401 will be described. The image correction unit 107 performs necessary correction processing on the image data received by the image reception unit 106. For example, it performs a luminance unevenness correction process due to a drop in the amount of peripheral light that occurs when the lens 103 is passed through, and a process for correcting characteristic variations such as sensitivity unevenness for each pixel of the image sensor 104. It is a parallax calculation unit, and calculates the parallax for each pixel of the corrected image corrected by the image correction unit 107. Reference numeral 109 denotes a distance calculation unit, which converts the parallax data calculated by the parallax calculation unit 108 into a distance image. Reference numeral 110 denotes a DRAM, which stores a distance image generated by the distance calculation unit 109.

Reference numeral 1402 is a reliability calculation unit, which calculates the reliability for each arbitrary pixel unit in the image from the image correction unit 107. Reference numeral 112 denotes SRAM, which stores the reliability calculated for each pixel in the present embodiment. The reliability may be calculated not for each pixel but for each of a plurality of pixels, and the storage destination of the reliability does not necessarily have to be SRAM and may be another memory.

Reference numeral 1403 is a distance image reading determination unit. The coordinate values of the distance image corresponding to each pixel of the display image are generated by referring to the information on the cutout position and the degree of enlargement / reduction of the distance image corresponding to the display image (hereinafter referred to as display area information). Further, it is determined whether or not to read the distance image from the DRAM 110 according to the display area information and the reliability of each pixel of the distance image. When performing distortion correction, the distance image readout determination unit 1403 refers to the coordinate conversion map that associates the coordinate values of the images before and after the distortion correction with each pixel of the image after the distortion correction, as in the other embodiment described above. Generates the coordinate values of the image before distortion correction corresponding to.

Reference numeral 1404 is an image shape deformation unit, and image shape deformation processing is performed on the pixels determined by the distance image reading determination unit 1403 that the distance image needs to be read out, and the pixel values after the image shape deformation are output. .. The image shape deformation process reads the pixel value of the corresponding distance image from the DRAM 110 with reference to the coordinate value from the distance image reading determination unit 1403. As a method of calculating the pixel value, it is preferable to use an interpolation method typified by bilinear or bicubic using peripheral pixel values, but in the case of image cropping without scaling, the distance image can be read out as it is according to the coordinate values. good. The image shape deformation process performed by the image shape deformation unit 1404 includes distortion correction in addition to cutting out, enlargement / reduction, and the like.

Reference numeral 1405 is a mask unit, and when the distance image reading determination unit 1403 determines that the reading of the distance image is unnecessary, a predetermined value is set for the pixels of the display image and output.

As described above, the hardware configuration of the distance measuring device according to the fifth embodiment has been described with reference to FIG.

In addition, even in the case of performing distortion correction as in the above-described embodiment, the same configuration can be realized. In this case, a coordinate conversion map for associating the coordinates before and after the distortion correction is stored in the DRAM 110 in advance, and the coordinate values of the pixels of the image before the distortion correction corresponding to each pixel of the image after the distortion correction are generated based on the coordinate conversion map. Hereinafter, the processing flow in the case of performing the image shape deformation processing including the distortion correction using the coordinate conversion map and the display area information will be described with reference to the flowchart.

FIG. 15 is an example of a flowchart showing a processing flow of the distance image generation unit 1401 in the distance measuring device 1400 of the present embodiment. Functions that overlap with the above-described embodiments are designated by the same reference numerals.

Since S201 to S204 are the same as those in the above-described embodiment, the description thereof will be omitted.

In S1501, the reliability calculation unit 1402 calculates the reliability for each predetermined pixel unit in the image corrected in S202. In the present embodiment, the reliability is calculated in units of one pixel, but the reliability may be calculated in units of any pixel. Further, in the present embodiment, the reliability is calculated by 1 bit, but the reliability may be calculated by any number of bits. The reliability is calculated based on factors that affect the distance measurement accuracy, but in the calculation of the distance using the phase difference method in the present embodiment, the reliability is calculated based on the amount of change in contrast. The amount of change in contrast can be calculated from the dispersion of the pixel values of the point of interest and the pixels around it. The calculated reliability is stored in SRAM or the like in association with the coordinates of the image before distortion correction. The processes of S1501 and S203 to S204 are independent and may be processed in parallel, or S1501 may be performed before the process of S203 and the calculated reliability may be stored in the SRAM in advance.

In S1502, the distance image reading determination unit 1403 refers to the display area information and generates the coordinate values of the distortion-corrected image corresponding to each pixel of the display image.

In S1503, the distance image reading determination unit 1403 refers to the coordinate conversion map and reads out the coordinate values of the image before distortion correction corresponding to the coordinate values of the image after distortion correction generated in S1502 from the DRAM 110.

Here, S1502 and S1503 will be described with reference to FIG. FIG. 16 is a diagram illustrating a method of acquiring the coordinate value of the image before distortion correction from the coordinate value of the display image 1600 via the image 1601 after distortion correction in S1502 and S1503 of the image shape deformation processing in the fifth embodiment. .. The display image 1600 cuts out and displays two image areas (image cutout area A1603 and image cutout area B1604) in the distortion-corrected image 1601.

The display area information is information that specifies the coordinate values of each cutout area of the distortion-corrected image 1601 in the display image 1600, and may have coordinate values for each pixel of the display image 1600, or indicates a rectangular area. It may be information. For example, when the rectangular area is a rectangle, the configuration may have the start coordinates (X1, Y1) of the upper left vertex and the end coordinates (X2, Y2) of the lower right vertex of the rectangle.

The distance image reading determination unit 1403 first refers to the display area information in S1502, and acquires the distortion-corrected coordinate position 1606 corresponding to the pixel of interest 1605 of the display image 1600. Next, the distance image reading determination unit 1403 refers to the coordinate conversion map in S1503 and generates the coordinate position 1607 before distortion correction corresponding to the coordinate position 1606 after distortion correction. In this way, the distance image reading determination unit 1403 can acquire the coordinate value before distortion correction 1607 corresponding to the pixel of interest 1605 of the display image 1600, that is, the coordinate value of the image before distortion correction.

In S1504, the reliability corresponding to the pixel closest to the read out coordinate position before distortion correction 1607 is read from the SRAM 112. If the reliability is 0, it is determined that the reading of the distance image is unnecessary, and the process proceeds to S1506. If the reliability is 1, it is determined that the distance image needs to be read, and the process proceeds to S1505.

Next, in S1505, the image shape deforming unit 1404 reads out the pixel value of the pre-distortion coordinate position 1607 on the pre-distortion image determined in S1504 to read the distance image. Then, the image shape deformation processing is performed in which the pixel value read by the image shape deformation unit 1404 is set as the pixel value of the point of interest 1605 on the display image.

On the other hand, in S1506, the image shape deforming unit 1404 does not read out the pixel value of the pre-distortion coordinate position 1607 on the pre-distortion image determined in S1504 that the reading of the distance image is unnecessary, and does not perform the image shape deformation processing. .. Instead, the mask unit 1405 performs mask processing in which a predetermined value is applied to the pixels of the point of interest 1605 on the display image.

In S211 it is determined whether or not the pixel value is added by the image shape deformation processing or the mask processing in all the pixels of the display image 1600, and if there is a pixel to which the pixel value is not given, the process returns to S1502.

By repeatedly performing such steps on all the pixels of the display image, it is possible to output a display image that has been subjected to distortion correction, image cropping, and enlargement / reduction processing.

In S212, for example, based on the user input, if the reception of the image is not stopped, the process returns to S201 to receive the next image, and the processing after S202 is executed.

As described above, according to the present embodiment, in addition to the distortion correction processing, the reliability is also used in the processing involving DRAM access such as enlargement / reduction of the display area, cutting out of a part area, and multi-screen display in which a plurality of areas are specified. By performing control, the DRAM bandwidth used can be reduced.

(Embodiment 6)
In the sixth embodiment, the DRAM band used in the process of detecting a predetermined subject from the captured image and outputting the distance image to the subject is reduced.

FIG. 17 is a diagram showing a configuration of a distance measuring device according to the sixth embodiment. Functions that overlap with the above-described embodiments are designated by the same reference numerals, and description thereof will be omitted.

Reference numeral 1702 is an object detection processing unit, which receives one of a pair of captured images output from the image correction unit 107 as an input to recognize and identify a subject such as character recognition of a vehicle number or a sign and shape identification of a part. Performs processing and acquires object position information. There are various methods for recognition and identification processing. It may be a method of calculating the similarity by comparing the pattern with the object stored in advance, machine learning that finds and estimates the relevance from a lot of data, or a multi-layered structure algorithm such as a neural network. Estimates may be made.

The object position information generated by the object detection processing unit 1702 is information related to coordinates indicating an image area corresponding to a subject to be recognized and identified. Reference numeral 1703 is an SRAM, and in the present embodiment, the object position information calculated for each pixel is stored. The object position information is not for each pixel, but may be information indicating a rectangular area including a subject. For example, when the rectangular area is a rectangle, the start coordinates (X1, Y1) of the upper left vertex and the end coordinates (X2, Y2) of the lower right vertex of the rectangle may be held for each rectangular area. The storage destination of the object position information does not necessarily have to be SRAM, and may be another memory.

Reference numeral 1704 is a distance image reading determination unit. Depending on the above-mentioned display area information, object position information about the subject to be identified, and reliability, there is an image area corresponding to the subject to be recognized and identified in the display area, and the reliability is 1. If, it is determined that the display area in the distance image needs to be read out. When there is no image area corresponding to the subject in the display area or the reliability is 0, it is determined that the reading of the distance image is unnecessary.

Next, as another example of the present embodiment, a configuration is shown in which object detection is performed using a distortion-corrected image and a distance image of an image region portion corresponding to the detected subject is output. FIG. 18 is a diagram showing another configuration of the distance measuring device according to the sixth embodiment, in which an image shape deformation unit for distortion correction is added to the front stage of the object detection processing unit separately from the image shape deformation unit 1404. .. Functions that overlap with the above-described embodiment shown in FIG. 17 are designated by the same reference numerals, and description thereof will be omitted.

Reference numeral 1803 is a DRAM, which stores image data of one of a pair of captured images output from the image correction unit 107. In the present embodiment, of the image data a and the image data b having the parallax, the image data a is used, and the corrected image corrected by the image correction unit 107 with respect to the image data a is referred to as the image data A. And.

Reference numeral 1804 is an image shape deforming unit, which generates coordinate values of pixels of the uncorrected image corresponding to each pixel of the image after distortion correction by referring to a coordinate conversion map stored in DRAM 1803 in advance. The image shape deformation unit 1804 corrects the distortion of the image data A according to the coordinate conversion map, and generates the image data A'after the distortion correction.

Reference numeral 1802 is an object detection processing unit, which uses image data A'after distortion correction to perform subject recognition and identification processing such as character recognition of vehicle numbers and signs and shape identification of parts, and obtains object position information. do. The recognition and identification processing methods are as described above. Object position information may be stored in SRAM (not shown).

The DRAM 1803 that stores the image data A before distortion correction and the DRAM 110 that stores the distance image may use the same DRAM. In this case, the same coordinate transformation map can be referred to. Further, the DRAM 1803 may store the coordinate conversion map information in which the coordinate values are set in advance so as to be resized to the image data size handled by the object detection processing unit 1802.

Reference numeral 1805 is a distance image reading determination unit. With reference to the object position information generated by the object detection processing unit 1802 and the reliability calculated by the reliability calculation unit 1402, the image area corresponding to the subject to be recognized and identified and the reliability is 1. For example, it is determined that it is necessary to read out the pixel values of the coordinates indicated by the object position information on the distance image.

FIG. 19 is a flowchart showing an example from the distance image reading determination by the distance image reading determination unit 1805 to the image shape deformation processing of the distance image in the distance measuring device 1800 of the present embodiment.

In S1901, the object position information generated by the object detection processing unit 1802 is acquired. The object position information is information indicating the coordinates of the image area corresponding to the subject. The object position information may be read from an SRAM (not shown) with 1 bit (1 if there is a subject, 0 if there is no subject) for each pixel, or indicates a rectangular area including an image area corresponding to the subject. It may be information. For example, when the rectangular area is a rectangle, the information has the start coordinates (X1, Y1) of the upper left vertex and the end coordinates (X2, Y2) of the lower right vertex of the rectangle.

In S1902, the coordinate conversion map of the distance image is referred to from the DARM 110, the pixel of interest is selected from the pixels corresponding to the coordinates indicated by the object position information, and the coordinate position before distortion correction corresponding to the selected pixel of interest is generated.

In S1903, the reliability corresponding to the pixel closest to the read out coordinate position before distortion correction is read from the SRAM 112. If the reliability is 0, it is determined that the reading of the distance image is unnecessary, and the process proceeds to S1907. If the reliability is 1, it is determined that the pixel value of the pixel of interest on the distance image needs to be read, and the process proceeds to S1906.

Next, in S1906, the image shape deforming unit 1404 reads out the pixel value on the corresponding uncorrected image and performs the image shape deformation processing on the pixel of interest on the distance image determined to be read in S1903. .. On the other hand, in S1905, a mask process in which a predetermined value is given by the mask unit 1405 to the pixels determined in S1903 that the reading of the distance image is unnecessary without reading the image before distortion correction and performing the image shape deformation processing. I do.

In S1908, the pixel value of the distance image processed in S1906 or S1907 is output. By repeatedly executing the processing flow described with reference to FIG. 19 for all the coordinates indicated by the object position information, a distance image of only the subject to be identified is output.

As described above, according to the present embodiment, it is possible to reduce the DRAM band used in the process of detecting the image area corresponding to the subject in the predetermined display area and outputting the distance image only for the subject. This is because the present embodiment can limit the pixel value of the distance image read from the DRAM by using the object position information and the reliability of the subject.

Even if the object position information of the subject is acquired from the outside of the distance measuring device, the pixel value of the distance image read from the DRAM can be obtained by using the object position information of the subject and the reliability of each pixel as in the above description. Can be limited. In the present embodiment, the object position information is generated, but the object position information may be generated by an external device. For example, the image transmission unit 105 or the image correction unit 107 outputs the captured image to an external device (GPU, etc.), and the detection result of the image region corresponding to the subject on the captured image by the external device is used as object position information for a distance image. The read determination units 1705 and 1805 may acquire the image.

(Embodiment 7)
In the seventh embodiment, two distance images with overlapping distance measuring regions generated by the two distance measuring devices, that is, two distance images generated based on the captured images with overlapping imaging regions are combined to form one composite distance. The DRAM band used in the image output process is reduced.

FIG. 20 is a diagram showing a configuration of a distance measuring device according to the seventh embodiment.

Reference numeral 2001 denotes a first image pickup unit including a lens and an image pickup device, which outputs image data generated by the image pickup element.

Reference numeral 2002 denotes a first captured image processing unit, which includes an image correction unit, a parallax calculation unit, and a distance calculation unit that receive and process image data from the first image pickup unit 2001. The first captured image processing unit 2002 generates a distance image (distance image 1) using the captured image data, and writes the distance image 1 to DRAM 2007.

Reference numeral 2003 denotes a first reliability calculation unit, which calculates the reliability of the distance (reliability 1) from the first captured image processing unit based on the image data after image correction. The method of calculating the reliability is as described in the above-described embodiment.

Reference numeral 2004 denotes a second image pickup unit including a lens and an image pickup device, which outputs image data generated by the image pickup element.

Reference numeral 2005 denotes a second image processing unit, which includes an image correction unit, a parallax calculation unit, and a distance calculation unit that receive and process image data from the second image processing unit 2004. The second captured image processing unit 2005 generates a distance image (distance image 2) using the captured image data, and writes the distance image 2 to DRAM 2007.

Reference numeral 2006 denotes a second reliability calculation unit, which calculates the reliability of the distance (reliability 2) from the second captured image processing unit based on the image data after image correction. The method of calculating the reliability is as described in the above-described embodiment.

Reference numeral 2008 denotes a distance image reading determination unit. The coordinate values of the distance image 1 and the distance image 2 corresponding to each pixel of the display image are generated. Specifically, the coordinate values of each pixel are generated by referring to the coordinate conversion map that associates the coordinates of the distance images before and after the distortion correction corresponding to each of the distance image 1 and the distance image 2 stored in the DRAM 2007 in advance. The distance image reading determination unit 2006 acquires the reliability corresponding to the pixel closest to the generated coordinate value from each of the first reliability calculation unit and the second reliability calculation unit.

When the distance image 1 and the distance image 2 have overlapping regions, the distance image readout determination unit 2008 determines the magnitude (high or low reliability) of each reliability, and the distance image having the higher reliability. The distance information synthesis unit 2009 is instructed to read out the pixel value of. When the reliability of any of them is 0, it is determined that the reading of the distance image is unnecessary, and the mask unit 2010 performs mask processing (setting a predetermined value to the pixel of the display image and outputting). If each reliability has the same value other than 0, it may be instructed to read the pixel value of either distance image. However, in such a case, the influence of optical distortion is caused by reading out the pixel value of the pixel whose reliability is calculated to be closer to the center of the optical axis of the first imaging unit 2001 or the second imaging unit 2004. Can be reduced.

The distance information synthesis unit 2009 is any one of the first imaging unit 2001 and the first imaging image processing unit 2002, the second imaging unit 2004, and the second imaging image processing unit 2005 in response to an instruction from the distance image reading determination unit 2008. The distance image generated by one of them is read out. The distance information synthesis unit 2009 may perform interpolation processing for improving image quality as described in the above-described embodiment.

As described above, as the hardware configuration of the distance measuring device according to the seventh embodiment, a configuration including two imaging units 2001 and 2004 as shown in FIG. 20 has been illustrated, but the same applies when three or more imaging units are used. It is applicable to.

As described above, according to the present embodiment, when the distance images from a plurality of distance-measurable cameras are combined and displayed, the pixel value of the distance image read from the DRAM is limited based on the reliability of each pixel. The DRAM band used can be reduced. When synthesizing a distance image, it is also possible to perform distortion correction for correcting optical distortion, and similarly, the DRAM band used can be reduced. Note that the distortion correction for the overlapping region of the plurality of distance images to be combined is performed only on the distance image having the highest reliability, so that the DRAM to be used is compared with the case where the distortion correction is performed on all the distance images. Bandwidth can be reduced.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Further, in the above-described embodiment, as the distance measuring method, a method of arranging a plurality of photodiodes for one distance measuring pixel and calculating the distance by using the phase difference method and a method using a stereo camera have been described. However, the present invention is not limited to these distance measuring methods, and can also be applied to shape deformation processing when visualizing and displaying distance information acquired by TOF, millimeter-wave radar, LiDAR (Light Detection and Ranger), or the like.

100 Distance measuring device 101 Imaging unit 102 Distance image generation unit 110 DRAM
111 Reliability calculation unit 115 Distortion correction unit 1404 Image shape deformation unit 1702 Object detection processing unit

Claims (19)

A generator that generates a distance image that represents the distance to the subject.
A generation means for generating the distance image based on a pair of captured images having parallax, and
A correction means for correcting distortion in the distance image caused by the optical system of the imaging means that generated the pair of captured images in a region of the distance image whose reliability is equal to or higher than a predetermined threshold value.
A generator characterized by comprising. A generator that generates a distance image that represents the distance to the subject.
A correction means for correcting distortion in the pair of captured images due to the optical system of the imaging means that generated the pair of captured images in a region of the pair of captured images having parallax whose reliability is equal to or higher than a predetermined threshold value. When,
A generation means for generating a distance image based on a pair of captured images whose distortion has been corrected by the correction means, and a generation means.
A generator characterized by comprising. The generator according to claim 1 or 2, wherein the correction means calculates the reliability based on the pixel values of the pixel of interest and the pixels around the pixel of interest. The generator according to any one of claims 1 to 3, wherein the reliability is an amount of change in contrast calculated based on the dispersion of pixel values of the pixel of interest and pixels around the pixel of interest. The generation according to any one of claims 1 to 4, wherein the correction means applies a different predetermined distortion correction method according to the ratio of the region where the reliability is less than the predetermined threshold value. Device. The generator according to any one of claims 1 to 5, further comprising a masking means for setting a predetermined value for pixels in a region where the reliability is less than the predetermined threshold value. A process involving access to the storage means that is equal to or less than the predetermined access band based on at least the ratio of the predetermined access band to the storage means for storing the distance image and the area where the reliability is equal to or higher than the predetermined threshold value. The generator according to any one of claims 1 to 6, further comprising a selection means for selecting the above. The selection means selects one distortion correction method based on the ratio of regions whose reliability is less than a predetermined threshold value and the access band to the storage means required for each of the plurality of predetermined distortion correction methods. ,
The generator according to claim 7, wherein the correction means applies the strain correction method selected by the selection means. The selection means includes the predetermined access band, the proportion of the region whose reliability is less than a predetermined threshold, the access band to the storage means required by the selected distortion correction method, and a plurality of predetermined additional processes. 8. The generator according to claim 8, wherein one is selected from the plurality of predetermined additional processes based on the access band to the storage means required in the above. 8. The method according to claim 8 or 9, wherein the selection means selects one distortion correction method from the plurality of predetermined distortion correction methods having different pixels to be referred to when interpolating pixel values in distortion correction. Generator. The storage means has a buffer capable of storing pixel values for a predetermined pixel, and includes a cache means for exchanging pixel values in buffer units.
The method according to any one of claims 7 to 9, wherein the correction means calculates the reliability in units of pixels corresponding to the buffer, and obtains a pixel value for distortion correction from the cache means. Generator. Further provided with a designation means for designating a part of the distance image as a cutout area,
The generation device according to claim 1, wherein the correction means corrects distortion based on the cutout region and the region whose reliability is equal to or higher than a predetermined threshold value. The generator according to claim 12, wherein the cutout region is an image region corresponding to a predetermined object in the captured image. Further provided is a synthesizing means for generating a composite distance image by synthesizing a plurality of the distance images having overlapping regions generated based on a plurality of pairs of the captured images having overlapping imaging regions.
The synthesizing means has the highest reliability of the overlapping region among the pixel values of the plurality of distance images in the overlapping region with respect to the pixel value of the region corresponding to the overlapping region of the composite distance image. Using the pixel value of the distance image,
The generator according to any one of claims 1 to 13. The correction means corrects the distortion of the distance image having the highest reliability in the overlapping region in the overlapping region among the regions in which the reliability of each of the plurality of distance images is equal to or higher than the threshold value. ,
The generator according to claim 14. It is a generation method that generates a distance image showing the distance to the subject.
A generation step of generating the distance image based on a pair of captured images having parallax, and
A correction step for correcting distortion in the distance image caused by the optical system of the imaging means that generated the pair of captured images in a region of the distance image whose reliability is equal to or higher than a predetermined threshold value.
A production method characterized by having. It is a generation method that generates a distance image showing the distance to the subject.
A correction step for correcting distortion in the pair of captured images due to the optical system of the imaging means that generated the pair of captured images in a region of the pair of captured images having parallax whose reliability is equal to or higher than a predetermined threshold value. When,
A generation step of generating a distance image based on a pair of captured images whose distortion has been corrected by the correction step, and a generation step.
A production method characterized by having. On the computer
It is a generation method that generates a distance image showing the distance to the subject.
A generation step of generating the distance image based on a pair of captured images having parallax, and
A correction step for correcting distortion in the distance image caused by the optical system of the imaging means that generated the pair of captured images in a region of the distance image whose reliability is equal to or higher than a predetermined threshold value.
A program comprising performing each step of a method having. On the computer
It is a generation method that generates a distance image showing the distance to the subject.
A correction step for correcting distortion in the pair of captured images due to the optical system of the imaging means that generated the pair of captured images in a region of the pair of captured images having parallax whose reliability is equal to or higher than a predetermined threshold value. When,
A generation step of generating a distance image based on a pair of captured images whose distortion has been corrected by the correction step, and a generation step.
A program comprising performing each step of a method having.

Images