JP2019113434A - Calibrator, calibration method, and calibration program - Google Patents

Abstract

To provide a calibrator, a calibration method and a calibration program with which it is possible to perform automatic adjustment for the displacement of a roll, etc., arising from the installed state of a stereo camera mounted in a vehicle, etc., easily without incurring an increase in computation time.SOLUTION: Provided is a calibrator for calibrating displacement arising from the installed state of a stereo image capturing device, the calibrator comprising: a luminance information extraction unit for finding the number of points satisfying a given condition, among points in a reference image captured by the stereo image capturing device, the luminance of which changes, as a goal parallax number; a parallax computation unit for calculating a parallax using comparison and reference images captured by the stereo image capturing device; a parallax inspection unit for finding the effective number of parallaxes among the parallaxes as an effective parallax number; and a determination unit for determining the occurrence of a displacement on the basis of a difference between the goal parallax number and the effective parallax number.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a calibration apparatus, calibration method, and calibration program that automatically calibrates a shift that occurs due to the installation state of a stereo camera mounted on a vehicle or the like.

  In recent years, automobiles equipped with a collision prevention device, an inter-vehicle distance control device, and the like have been marketed as devices for assisting in the improvement of automobile safety, and research and development aimed at achieving automatic driving are being carried out actively. . These devices automatically measure the distance to the car ahead etc., alert the driver when the distance between vehicles becomes short, etc., and control the brakes and steering etc. for collision avoidance when the distance between them becomes even shorter. It is a thing. That is, in these devices, automatic measurement of the inter-vehicle distance is essential, and is also an essential processing in automatic driving.

  Radars, stereo cameras, etc. are used in automatic measurement of inter-vehicle distance, but stereo cameras detect the size, position, speed, etc. of a plurality of three-dimensional objects in an instant, and further, side wall which becomes the boundary of traveling area, It has an advantage that it is possible to accurately detect road surface marks such as road shoulders and white lines.

  A stereo camera is a combination of a plurality of (usually two) cameras into one, which captures an object with a plurality of lenses, and measures the distance to the object using a plurality of captured images.

Here, measurement of distance by a stereo camera will be described. FIG. 1 is a view for explaining the principle of distance measurement by a stereo camera composed of two cameras installed in parallel equilibria. The cameras 11 and 12 are arranged at equal distances parallel to each other by a distance B (i.e. the base length is B), and the focal lengths are the same f. When the subject A is placed at a position away from the installation surface of the cameras 11 and 12 by the distance Z in the optical axis direction, the image of the subject A is an optical center O _{1 at the} camera 11. and the P ₁ is an intersection of the straight line and the imaging surface 13 connecting the object a, respectively imaged on P ₂ is an intersection of the straight line and the imaging surface 14 connecting the camera 12 in the optical center O ₂ and the object a. Then, if a straight line parallel to the straight line AP ₂ and passing through the optical center O ₁ is drawn and the point of intersection of the straight line and the imaging surface 13 is P ₂ ′, P ₂ ′ is at the same position as P ₂ at the camera 12 The disparity between P ₁ and P ₂ ′ is parallax. Usually, parallax means a difference in direction and is often expressed by an angle, but here, the distance D between P ₁ and P ₂ ′ is called parallax. By measuring the distance D (parallax D), the distance Z to the subject A can be determined by the following equation ₁ because the triangle AO ₂ O ₁ and the triangle O ₁ P ₂ ′ P ₁ are similar.

In order to measure the distance by the stereo camera in this manner, the intersection points P ₁ and P ₂ must be searched for on the imaging surfaces 13 and 14. In order to perform this search at high speed, the cameras 11 and 12 are usually installed such that the optical axes are parallel and the directions of the scanning lines of the imaging plane coincide. Accordingly, the deviation of the subject in the imaging plane will be generated only in the direction of the scan lines, only will be may be searched intersections P ₁ and P ₂ direction of the scanning lines, faster search can be achieved. In order to realize this, the heights of the corresponding scanning lines of the left and right images must match with high accuracy, for example, with an accuracy of 0.1 pixel or less over the entire imaging surface. However, in real stereo cameras, it is difficult to keep the positions of the two cameras constant with such high accuracy. Therefore, it is necessary to monitor the positional relationship between the two cameras during operation of the stereo camera for a shift, and to automatically adjust the position when the shift is detected. There are six degrees of freedom in displacement due to translation and rotation of the left and right cameras, but frequently occurring displacements are three displacements due to rotation. That is, rotational deviation due to rotation (roll) about the optical axis, vertical deviation due to vertical movement (pitch), and lateral deviation due to horizontal movement (yaw) Has been proposed.

  For example, Naoki Nishikubo, Kotaro Oshida, Yuhei Oka, Keiji Jikichi, "Auto-calibration of stereo cameras with FPGA", Proceedings of the 29th Annual Conference of the Robotics Society of Japan, Tokyo, September 2011 (Non-Patent Document 1). Regions with strong edges in two directions perpendicular to each other are detected as feature points, and searching is performed by two-dimensional matching to calibrate vertical deviation and rotational deviation.

Naoki Nishikubo, Kotaro Oshida, Yuhei Oka, Keiji Jikichi, "Auto-calibration of stereo camera with FPGA", Proceedings of the 29th Annual Conference of the Robotics Society of Japan, Tokyo, September 2011

  However, in the method of Non-Patent Document 1, the search range is two-dimensional, and the search in the area is continued until the feature point is found, and after the feature point is detected, two-dimensional matching is performed by the area base method. Time may increase.

  The present invention has been made under the circumstances as described above, and an object of the present invention is to automatically shift a shift due to a roll or the like caused due to an installation state of a stereo camera (stereo image pickup device) mounted on a vehicle or the like. It is an object of the present invention to provide a calibration device, a calibration method and a calibration program which can be easily adjusted while suppressing an increase in calculation time.

  The present invention relates to a calibration device that calibrates a shift that occurs due to an installation state of a stereo image pickup device, and the above object of the present invention is to set a point at which luminance changes in a reference image picked up by the stereo image pickup device. Among them, a luminance information extraction unit that obtains the number of points satisfying a certain condition as a target parallax number, a parallax operation unit that calculates parallax using the comparison image captured by the stereo image pickup device and the reference image, A disparity inspection unit which obtains the number of effective parallaxes among the parallaxes as an effective parallax number; and a deviation judgment unit which judges that the deviation is occurring based on a difference between the target parallax number and the effective parallax number It is achieved by providing.

  The above object of the present invention is that the luminance information extraction unit sets, as the target number of parallaxes, the number of pixels having a luminance difference larger than a threshold with a pixel adjacent to the reference image, or the parallax inspection unit The ratio of the number of effective parallaxes to the target number of parallaxes is a value close to 1 by setting the number of parallaxes excluding the parallaxes having a large difference between the surrounding parallaxes and the value as the effective number of parallaxes In this case, it is determined that the deviation does not occur, or the screen for the reference image and the comparison image is divided into a plurality of areas, and the luminance information extraction unit determines the target number of parallaxes for each area. The parallax calculating unit calculates the parallax for each area, and the parallax inspection unit uses the parallax for each area to calculate the effective parallax number for each area. Another effective parallax The deviation determining unit determines that the deviation is occurring based on the difference between the target disparity count by region and the effective disparity number by region, or the shift determining unit, If the ratio by region, which is the ratio of the number of effective parallax numbers by region to the number of target parallaxes by region, is a value close to 1 at all, it is determined that the deviation does not occur, and the ratio by region is all approximately the same value When the value is farther than 1, it is determined that a shift due to the pitch occurs, and when the area-based ratio having a value close to 1 and the area-based ratio having a value farther than 1 coexist, a shift due to a roll occurs. If it is determined that the shift is made, or if the shift determination unit determines that a shift due to the pitch is occurring, the reference image or the comparison image is shifted in the direction of the pitch at a predetermined interval, Calculating the effective parallax number by area in the central area located at the center of the screen by the parallax calculation unit and the parallax inspection unit, and determining whether the area ratio in the central area is closest to 1 or 2. By setting the shift amount of the comparison image as the shift amount due to the pitch, or when the shift determination unit determines that the shift due to the roll is generated, the reference image or the comparison image is set to the pitch While shifting by a predetermined interval in the direction, the disparity calculating unit and the disparity checking unit determine the number of effective parallaxes by region in the central region located at the center of the screen, and the ratio by region in the central region is 1 at most An amount obtained by shifting the reference image or the comparison image in the case of being close to the above is the shift amount due to the pitch, and the reference image or the shift made by the shift amount. While the comparative image is rotated at a predetermined angle with the center of the screen as the origin, the parallax computing unit and the parallax inspection unit determine the number of effective parallaxes by region as the number of effective parallaxes by region, and By setting the angle at which the reference image or the comparison image is rotated when the ratio of the number of effective rotation parallax numbers to all becomes a value close to 1 as the amount of rotation by the roll, or the deviation determination unit If the ratio by region which is the ratio of the number of effective parallax numbers by region to the number of target parallaxes by region is all values close to 1, it is determined that the deviation does not occur, and the ratio by regions is all 1 with substantially the same value. In the case of a more distant value, or in the case where there is a mixture of the area ratio having a value close to 1 and the area ratio having a value farther than 1, by determining that the deviation has occurred, or When the shift determination unit determines that the shift is occurring, the parallax calculator and the parallax inspection unit shift the screen while shifting the reference image or the comparison image at a predetermined interval in the pitch direction. The effective parallax number by area in the central area located at the center of the area is determined, and the amount by which the reference image or the comparison image is deviated when the area specific ratio in the central area is closest to 1 is the deviation amount by the pitch And the parallax calculating unit and the parallax inspecting unit rotate the effective parallax count by area by rotating the reference image or the comparison image shifted by the shift amount at a predetermined angle with the center of the screen as the origin. The reference image or the comparison image is obtained as the number of effective rotation parallaxes, and the ratio of the number of effective rotation parallaxes to the target number of parallaxes by region all has a value close to 1 By setting the angle of rotation as the amount of rotation by the roll or after the deviation determination unit determines the amount of rotation by the roll, the reference image or the comparative image rotated by the amount of rotation is the direction of the pitch The disparity calculating unit and the disparity checking unit determine the number of effective parallaxes by region as the number of effective parallaxes by the parallax calculating unit and the parallax inspection unit, and the ratio of the number of effective parallaxes to the number of target parallaxes by region is one. This is achieved more effectively by updating the shift amount due to the pitch by the shift amount of the reference image or the comparison image in the case of close to.

  Further, the present invention relates to a calibration method for calibrating a displacement occurring due to an installation state of a stereo image pickup device, and the above object of the present invention is to change luminance in a reference image picked up by the stereo image pickup device A luminance information extracting step of obtaining, as a target parallax number, the number of points satisfying a certain condition among the points; a parallax calculating step of calculating a parallax using the comparison image captured by the stereo image pickup device and the reference image The disparity inspection step of obtaining the number of effective disparity among the disparity as the effective disparity number, and the shift determination to determine that the shift is occurring based on the difference between the target disparity number and the effective disparity number It is achieved by having a step.

  The above object of the present invention is that, in the luminance information extraction step, the number of pixels having a luminance difference larger than a threshold with a neighboring pixel in the reference image is set as the target parallax number, or in the parallax inspection step The ratio of the number of effective parallaxes to the number of target parallaxes is a value close to 1 by setting the number of parallaxes excluding the parallax having a large difference between the surrounding parallax and the value as the effective parallax number or in the deviation determination step In this case, it is determined that the deviation does not occur, or the screen for the reference image and the comparison image is divided into a plurality of areas, and in the luminance information extraction step, the target number of parallaxes for each area Is calculated as the target parallax number by area, the parallax calculating step calculates the parallax for each area, and the parallax inspection step uses the parallax for each area. The effective parallax number is determined as the effective parallax number by area for each area, and in the deviation determining step, the deviation occurs based on the difference between the target parallax number by area and the effective parallax number by area. Or if the ratio by region, which is the ratio of the number of effective parallaxes by region to the number of target parallaxes by region, is all close to 1, at the shift determining step. If it is determined that all of the regional ratios are substantially the same value and more than one, it is determined that a shift due to the pitch has occurred, and the regional ratio having a value close to one and a value more than one When the ratio by area having a mixture is mixed, it is determined that a shift due to a roll has occurred, or when it is determined that a shift due to the pitch has occurred in the shift determination step, While shifting the quasi-image or the comparison image at predetermined intervals in the direction of the pitch, the number of effective parallaxes by area in the central area located at the center of the screen is determined by the parallax calculation step and the parallax inspection step, By setting the amount of shift of the reference image or the comparison image when the area ratio in the central region is closest to 1 as the shift amount by the pitch, or the shift due to the roll is generated in the shift determination step If it is determined that the reference image or the comparison image is shifted at a predetermined interval in the direction of the pitch, the parallax calculation step and the parallax inspection step are performed in the central region located at the center of the screen. The reference image in the case where the region-specific ratio in the central region is closest to 1 by determining the region-specific effective parallax number Alternatively, the parallax calculation is performed while rotating the reference image or the comparison image shifted by the shift amount at a predetermined angle with the center of the screen as the origin, with the shift amount by the pitch as the shift amount of the comparison image. In the step and the parallax inspection step, the area-specific effective parallax number is determined as a rotation effective parallax number, and the reference image when all the ratios of the rotation effective parallax number to the area-specific target parallax number are close to 1 By setting the angle at which the comparison image is rotated as the amount of rotation by the roll, or in the deviation determination step, the ratio by area, which is the ratio of the effective parallax number by area to the target parallax number by area, is all 1 If the value is close to 0, it is determined that the deviation does not occur, and if all the area ratios are substantially the same value and more than 1 or have a value close to 1 When the area ratio and the area ratio having a value farther than 1 coexist, it is determined that the deviation has occurred, or it is determined that the deviation has occurred in the deviation determination step. In this case, while shifting the reference image or the comparison image at predetermined intervals in the direction of the pitch, the number of effective parallaxes according to area in the central area located at the center of the screen is determined by the parallax calculation step and the parallax inspection step. The amount by which the reference image or the comparison image is shifted when the area-specific ratio in the central region is closest to 1 is regarded as a shift amount by the pitch, and the reference image or the comparison image shifted by the shift amount The parallax calculation step and the parallax inspection step are performed according to the area while rotating the screen at a predetermined angle with the center of the screen as the origin. The number of effective parallaxes is determined as the number of effective rotation parallaxes, and the angle at which the reference image or the comparative image is rotated when all the ratios of the number of effective rotation parallaxes to the target number of parallaxes by area become close to 1 By using the amount of rotation or after determining the amount of rotation by the roll in the shift determination step, while shifting the reference image or the comparison image rotated by the amount of rotation in the direction of the pitch at a predetermined interval The reference image when the ratio of the number of effective shift parallaxes to the number of target parallaxes by region is closest to 1 is determined as the effective number of parallax disparity by area by the parallax calculation step and the parallax inspection step. Or, it is achieved more effectively by updating the shift amount due to the pitch by the shift amount of the comparison image.

  Furthermore, the above object of the present invention is achieved by a calibration program for performing the above calibration method.

  According to the calibration device of the present invention, the occurrence of the shift due to the roll or the like is determined based on the target number of parallaxes and the number of effective parallaxes obtained using the change in luminance and parallax calculated in the related art. Deviation calibration can be performed while suppressing increases in scale and calculation time. Furthermore, while moving the image, it is possible to obtain the amount of shift by the same processing as when determining the shift, it is possible to further suppress an increase in calculation time.

It is the schematic which shows the principle of measurement of the distance by a stereo camera. It is a block diagram which shows the structural example (1st Embodiment) of this invention. It is an image figure showing an example of the order which computes the difference of luminosity. It is an image figure which shows the example of the search range of the pixel in parallax calculation. It is an image figure which shows the example in the case of shifting parallax in the vertical direction and calculating parallax. It is an image figure showing an example in a case of rotating an image and calculating parallax. It is an image figure which shows the example of the surrounding pixel range in the case of performing singularity removal. It is a flowchart which shows a part of operation example (1st Embodiment) of this invention. It is a flowchart which shows a part of operation example (1st Embodiment) of this invention. It is a flowchart which shows the operation example of the shift | offset | difference amount calculation by pitch. It is a block diagram which shows the structural example (2nd Embodiment) of this invention. It is a flowchart which shows a part of operation example (2nd Embodiment) of this invention. It is a flowchart which shows a part of operation example (2nd Embodiment) of this invention. It is a block diagram which shows the structural example (3rd Embodiment) of this invention. It is a flowchart which shows the operation example of the shift | offset | difference amount recalculation by pitch. It is an image figure which shows the calculation method in sub-pixel precision.

  According to the present invention, when the stereo image pickup device is installed with high accuracy and no displacement occurs due to the installation state of the stereo image pickup device, the image picked up by the stereo image pickup device is originally acquired It is determined that a shift has occurred if the number of parallaxes to be obtained (target parallax number) and the number of actually acquired parallaxes (effective parallax number) substantially match, and if the both do not match . Inconsistencies are determined, for example, from the ratio of the two. Further, as the target parallax number, among the points where the luminance changes in one image, a certain condition, for example, the number of points satisfying the condition that the luminance difference is larger than a predetermined threshold value is adopted. In the calculation of parallax, it is difficult to correctly search for a point having a small difference in luminance, and in order to calculate the parallax correctly, the above condition is adopted because a certain degree of difference in luminance is required. In the calculation of the number of effective parallaxes, singularity removal is performed to remove inappropriate parallaxes in order to select effective parallaxes. For example, after the parallax is calculated on a pixel basis, if the parallax of a certain pixel is significantly different from the parallax of the surrounding pixels, the parallax of that pixel is not counted as the number of effective parallaxes.

  Further, in the present invention, it is possible to obtain an amount of deviation (hereinafter referred to as “calibration amount”) in order to calibrate the deviation. In order to obtain the calibration amount, the screen is divided into a plurality of (for example, 3 to 9) areas, and the target number of parallaxes (the number of target parallaxes by area) and the number of effective parallaxes (the number of effective parallaxes by area) are calculated for each area. Then, the difference between the target disparity number by region and the effective disparity number by region is calculated, for example, the ratio of the two (the ratio by region) is calculated for each region, and the occurrence of the displacement and the type of displacement occurring are judge. If it is determined that a shift has occurred, one image is moved at a predetermined interval or angle, and a calibration amount is determined based on the ratio by area to the two images after movement. Instead of the ratio, a difference amount, a difference ratio, or the like may be used. Further, when the image is moved to obtain the calibration amount, the maximum likelihood method may be applied to reduce the number of trials.

  In the present invention, in a stereo image pickup apparatus, shading correction, inter-camera sensitivity difference correction, distortion correction, and the like are performed on a pickup image. Shading is a phenomenon in which the brightness decreases as it gets closer to the edge of the screen of the captured image. The inter-camera sensitivity difference means that differences occur in how luminance is detected because there are individual differences among cameras of the same product. Since the shading and the inter-camera sensitivity difference may affect the calculation of the target parallax number and the effective parallax number, they are corrected in the stereo image pickup device. Further, the image obtained from the camera is affected by the lens of the camera, and the amount of distortion increases as the distance from the center of the image increases, so this distortion is also corrected.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the shift targeted by this embodiment is a shift due to rotation (roll) around the optical axis and a shift due to vertical movement (pitch), and is there not a shift due to horizontal movement (yaw), Or it shall be corrected by other means. In addition, the stereo image pickup device includes two cameras, and the two cameras are installed in the horizontal direction, and the vertical direction is the vertical direction, and the horizontal direction is the horizontal direction. Since the rotation that causes the pitch and roll is minute, it appears on the screen as a vertical shift and a rotational shift.

  A configuration example (first embodiment) of the calibration device according to the present invention is shown in FIG. The calibration apparatus 20 in this configuration example includes an image storage unit 210, a luminance information extraction unit 220, a parallax calculation unit 230, a parallax storage unit 240, a parallax inspection unit 250, and a shift determination unit 260. Input a reference image and a comparison image, and output a calibration amount CA.

  The stereo image capturing apparatus 10 includes two cameras, and the focal lengths of the two cameras are the same, and are adjusted in advance so that the optical axes are parallel. The stereo imaging device 10 is installed so that the two cameras are parallel and equal, and an image captured by a camera on the right side (corresponding to the position of the camera 11 in FIG. 1) toward the subject is used as a reference image. An image captured by the camera (corresponding to the position of the camera 12 in FIG. 1) is a comparison image. The reference image and the comparison image are captured at the same timing, and as described above, correction (shading correction, inter-camera sensitivity difference correction, distortion correction, etc.) and correction of deviation due to yaw are performed and output.

  The reference image SP and the comparison image CP output from the stereo image capturing device 10 are stored in the image storage unit 210 in the calibration device 20. An outline of a method of calculating the calibration amount CA by the calibration device 20 with respect to the reference image SP and the comparison image CP is as follows.

  First, the screen with respect to the reference image SP and the comparison image CP is divided into a plurality of regions aligned in the horizontal direction by the lines in the vertical direction. Although the present embodiment is divided into five areas, the number of areas may be determined arbitrarily.

In the reference image SP, the difference in luminance between adjacent pixels in the horizontal direction is calculated for each divided area, and the number of differences larger than the predetermined threshold value TD is calculated as the target parallax number by area NT _n (n = 1, 2, ... and 5).

On the other hand, the parallax is determined in pixel units using the reference image SP and the comparison image CP, and further singular point removal is performed, and the number of remaining parallaxes in each region is set as the number of effective parallaxes per region NV _n .

Then, to calculate the ratio of the regional effective parallax number NV _n for the area-based target parallax number NT _n as regional ratio RT _n. If the area ratio RT _n is a value close to 1, no deviation occurs in that area. Therefore, if all the area ratios RT _n are values close to 1, it is determined that no shift has occurred (hereinafter, this determination result is referred to as “no shift”). Conversely, if all the regional ratio RT _n and farther than 1 in similar values, it is determined that the shift due to pitch occurs (hereinafter, this determination result as "pitch generation"). When the vertical deviation occurs, the same degree of influence of the deviation is assumed in each area, so this determination is performed. In addition, when the area ratio RT _n is a value close to 1 and the area distant from 1 coexist, it is determined that a shift due to a roll having a center at a value close to 1 occurs. (Hereafter, this determination result is referred to as "roll generation"). When a rotational deviation occurs, it is assumed that the effect of the deviation is greater as the distance from the center of rotation is greater, so such a determination is made.

  When the above determination result is "pitch generation", the effective parallax number by area in the area (center area) located at the center of the screen is calculated while shifting the entire image in the vertical direction at a predetermined interval, and further the area The ratio according to area in is calculated, and a case where the ratio according to area becomes the value closest to 1 is searched, and the amount shifted up to that case is taken as the amount of shift due to the pitch.

When the determination result is "roll generation", first, the amount of deviation due to the pitch is obtained by the same procedure as the above-described case of the generation of pitch. Then, while the image is rotated at a predetermined angle with the center of the screen as the origin, the number of effective parallaxes per region (the number of effective rotation parallaxes) NV _n is calculated, and the ratio RT _{n according} to area is calculated. calculated, when all the regional ratio RT _n becomes a value close to 1 (hereinafter, this is a "match case" case) looking for, the angle rotated up to that when the rotation amount of the roll. When there is no matching case, the above-mentioned calculation of the shift amount due to the pitch and the rotation of the image are repeated using an arbitrary image of the images rotated at a predetermined angle until a matching case is found. This is because the center of rotation at the time of rotating the image fixed at the amount of shift due to the calculated pitch does not necessarily exactly coincide with the center of rotational shift, and if there is a shift between the two, a predetermined period thereafter This is done because rotation at an angle of 加 わ っ adds translational movement and a matching case may not be found. The accuracy of the center position is enhanced by determining the shift amount due to the pitch again based on the rotated image.

  The shift amount due to the pitch and the rotation amount due to the roll are output as the calibration amount CA.

When the parallax is determined using the reference image SP and the comparison image CP, there is a position where there is no corresponding pixel in the reference image SP at the right end of the screen, so the target parallax number by area NT _n and the effective parallax number by area In the calculation of NV _n , the relevant part is excluded from the target of calculation.

  The calculation of the calibration amount CA described above is performed by the luminance information extraction unit 220, the disparity calculation unit 230, the disparity storage unit 240, the disparity inspection unit 250, and the deviation determination unit 260.

The luminance information extraction unit 220 obtains the target parallax number NT _n by area. With respect to the reference image SP, as shown in FIG. 3, for each area divided by the vertical lines, the difference in luminance between two pixels adjacent in the horizontal direction is sequentially calculated, and the magnitude (absolute value Counts the number of differences greater than a predetermined threshold value TD. Then, when the above processing is completed for all the pixels, the count number in each area is set as the target parallax number by area NT _n . The target image parallax number by region NT _n may be determined using the comparison image CP instead of the reference image SP.

  The disparity calculating unit 230 calculates disparity and a calibration amount CA using the reference image SP and the comparison image CP.

In the parallax calculation, in the comparison image CP, the pixels SPE _ij (i = 1, 2,..., I, j = 1, 2,..., J in the reference image SP. I is the number of pixels in the horizontal direction, J is the vertical The pixel FCPE corresponding to the number of pixels in the direction) is searched, and the difference in position in the horizontal direction between the pixel SPE _ij and the pixel FCPE is taken as the parallax with respect to the pixel SPE _ij . Search pixel FCPE, as shown in FIG. 4, compared in the image CP, as a starting point the same position as the pixel _{SPE ij,} as the target pixel _{CPE ij} rightward, the pixel _{SPE ij} brightness and pixel _{CPE ij} The magnitude (absolute value) of the difference in luminance is calculated, and the pixel CPE _ij having the smallest magnitude of difference is taken as the pixel FCPE. And if the parallax with respect to all the pixels is calculated, it will output as parallax data PA. The parallax may be obtained by searching for a pixel corresponding to a pixel in the comparison image CP from the reference image. In this case, the search in the reference image is performed on the pixels in the left direction.

  In the calculation of the calibration amount CA, the parallax calculating unit 230 calculates the parallax by moving the image and specifies the calibration amount CA. Prior to the calculation of the calibration amount CA, the shift determination unit 260 determines the shift, and outputs the determination result Jr to the disparity calculation unit 230.

  When the determination result Jr is “no shift”, the disparity calculating unit 230 sets the calibration amount CA to 0 and outputs no calibration amount CA, assuming that no shift occurs.

When the determination result Jr is "pitch generation", the entire image is shifted by a predetermined interval SG in the vertical direction, and the parallax calculation in the central region is performed in the same procedure as described above. Specifically, the comparison image CP is shifted, and as shown in FIG. 5, the comparison image indicated by the solid line is shifted in the vertical direction by a predetermined interval SG like the comparison image indicated by the one-dot chain line. Then, the search for the pixel FCPE in the comparison image CP is performed with a position shifted by a predetermined interval SG in the vertical direction from the same position as the pixel SPE _ij as a departure point, and the parallax is calculated. The area for calculating the parallax is only the central area shown in gray in FIG. 5, and only the parallax of this area is output as the parallax data PA. After that, the entire image is further shifted in the vertical direction by a predetermined interval SG, and the parallax of the central region is calculated and output as parallax data PA as described above. Once the entire image has been shifted a predetermined number of times or within a predetermined range, calculation of parallax is ended, but the amount of shift from the original position at each time is the shift amount SA _m (m = 1, 2,..., M, M The number of times to shift is held. After the end of the parallax calculation, the information on the time when the area-specific ratio becomes the value closest to 1 is output as the determination information Jt by the deviation determination unit 260. Therefore, the deviation amount SA _m of indicated by decision information Jt gyrus because a shift amount by the pitch, and outputs it as the calibration amount CA. Incidentally, in the direction of shifting one direction, if the always shifted by a predetermined interval SG, without holding the shift amount SA _m, it may be calculated shift amount by the pitch as SG × Jt from the determination information Jt.

When the determination result Jr is "roll generation", first, the amount of deviation due to the pitch is determined in the same procedure as in the case where the determination result Jr is "pitch generation". Then, with the shift amount fixed, the image is rotated at a predetermined angle with the center of the screen as the origin, and the parallax of all the regions is calculated. For example, when the shift amount due to the pitch is SAP, as shown in FIG. 6A, the comparison image shown by the solid line is vertically shifted by SAP like the comparison image shown by the one-dot chain line. As shown in FIG. 6 (B) with the center C as the origin, as shown in FIG. 6B, the search for the pixel FCPE in the comparison image CP is performed SAP in the vertical direction from the same position as the pixel SPE _ij The parallax is calculated on the scanning line which is shifted by a predetermined angle RG and is inclined downward by a predetermined angle RG from the lateral direction, with the position rotated by the predetermined angle RG as a departure point. Once the parallaxes of all the areas have been calculated, they are output as parallax data PA. Thereafter, while the shift amount is fixed at SAP, the entire image is further rotated by a predetermined angle RG, and the parallaxes of all the regions are calculated and output as parallax data PA as described above. When the entire image is rotated a predetermined number of times or in a predetermined range, the calculation of the parallax is ended, but the amount of rotation from the original position at each time is the amount of rotation RA _k (k = 1, 2,..., K). K is held as the number of rotations). After completion of the parallax calculation, the deviation determination unit 260 outputs the information of the round which is the matching case as the determination information Jt. Thus, since the rotation amount RA _k of indicated by decision information Jt times is rotation amount by the roll, and outputs it as the calibration parameter CA with the deviation amount by the pitch. Incidentally, in the direction of the rotating direction, if you always rotates by a predetermined angle RG does not hold the amount of rotation RA _k, it may be calculated rotation amount by the roll as RG × Jt from the determination information Jt. When there is no match case, the determination information Jt from the deviation determination unit 260 notifies that. For example, when there is no match case, the shift determination unit 260 sets the determination information Jt as 0 and outputs the result to the disparity calculation unit 230. When it is detected from the determination information Jt that there is no match case, the parallax operation unit 230 selects an arbitrary rotation amount (for example, RA ₁ or RA _K etc.) from the held rotation amounts, and uses the rotation amount Based on the rotated image, the above-described calculation of the displacement due to the pitch and the search for a matching case are performed. If there is still no match case, another rotation amount is selected and the same processing is repeated.

  If the matching result is not found even if the judgment result Jr is “unable to judge” or if “roll occurrence” does not find a matching case even when searching for all rotation amounts, the calibration amount CA can not be “calibrated” to raise it as an alarm. (Eg, CA = maximum value).

  The disparity data PA output from the disparity calculating unit 230 is stored in the disparity storage unit 240.

The disparity inspection unit 250 performs singular point removal on the disparity data PA stored in the disparity storage unit 240, and outputs the number of disparity remaining in each area after removal as the number of effective disparity by area NV _n . In the singular point removal, pixels having a parallax that is largely different from the surrounding pixels are removed as a singular point. Specifically, for example, as shown in FIG. 7, the parallax magnitude of the pixel SPE _ij adds a predetermined value TP1 to the average value of the parallax magnitudes of eight surrounding pixels shown in gray. If it is larger than the average value or smaller than a value obtained by subtracting a predetermined value TP2 from the average value, the pixel SPE _ij is recognized as a singular point and excluded from the count of the effective parallax number by region NV _n . Note that the range of the surrounding pixels may be extended to the range indicated by hatching in FIG. Further, not the average value but the mode value or the like may be used, or the singular point may be recognized based on the ratio to the average value or the like.

  The deviation determination unit 260 performs determination of deviation and generation of the determination information Jt in calculation of the calibration amount.

In the determination of the displacement, first, an area ratio RT _n (= NV _n / NT _n ) is calculated using the area-specific target parallax number NT _n and the area-specific effective parallax number NV _n . Then, if all the area ratios RT _n are values close to 1 (for example, 0.8 or more), it is determined that no shift has occurred, and the determination result Jr is set to “no shift” (for example, Jr = 0). . If all the area ratios RT _n are similar values (for example, the variance is less than a predetermined value) and a value farther than 1 (for example, 0.3 or less), it is determined that the shift due to the pitch is occurring, and the determination result Let Jr be "pitch generation" (for example, Jr = 1). When a value close to 1 and a value farther than 1 coexist as the region ratio RT _n , it is determined that a shift due to a roll is occurring, and the determination result Jr is set to “roll occurrence” (for example, Jr = 2). If none of the above applies, it is determined that the determination is impossible, and the determination result Jr is set to “indeterminable” (for example, Jr = 3). When the judgment result Jr is "pitch generation" or "roll generation", the target parallax number by region NT _n is used to generate the judgment information Jt in the calculation of the calibration amount, so it is held by area. Every time the ratio RT _n is calculated, the luminance information extraction unit 220 may calculate it.

In the generation of the determination information Jt in the calculation of the calibration amount, when the determination result Jr is “pitch generation”, the effective parallax number NV _n by area of the central area sequentially output from the parallax inspection unit 250 after the determination is stored. calculating a regional ratio RT _n the central region, to hold with well regions based target parallax number NT _n. Then, the entire image When shifted by a predetermined number of times or a predetermined range, and outputs a close round of information to the value most 1 in the holding areas by the ratio RT _n as the determination information Jt. When the determination result Jr is "roll occurrence", first, the determination information Jt is output in the same procedure as in the case where the determination result Jr is "pitch occurrence". After that, the area ratio RT _n of all areas is calculated and held using the area-specific effective parallax number NV _n sequentially output from the parallax inspection unit 250 and the area-specific target parallax number NT _n. Do. Then, when the entire image is rotated a predetermined number of times or in a predetermined range, the area ratio RT _n of all the areas among the held area ratios RT _n is a value close to 1 (for example, 0.8 or more) Information on the times of matching cases is output as the determination information Jt. If there is no match case, the determination information Jt is output as 0.

  An operation example of the calibration apparatus 20 having such a configuration will be described with reference to the flowcharts of FIGS.

  The reference image and the comparison image captured by the stereo image capturing device 10 are stored in the image storage unit 210 (step S10).

The luminance information extraction unit 220 reads the reference image SP from the image storage unit 210, calculates the difference in luminance between two adjacent pixels in the horizontal direction for each area, and the difference is greater than a predetermined threshold value TD The number is counted, and the target disparity number by region NT _n is determined (step S20). The target parallax number by area NT _n is input to the deviation determination unit 260.

  The parallax calculating unit 230 reads the reference image SP and the comparison image CP from the image storage unit 210, calculates the parallax in pixel units in all the regions (step S30), and outputs the calculated parallax data PA. The disparity data PA is stored in the disparity storage unit 240 (step S40).

The parallax inspection unit 250 reads out the parallax data PA from the parallax storage unit 240, performs singular point removal for each area (step S50), sets the number of remaining parallaxes as the number of effective parallaxes per area NV _n Output.

Deviation determination unit 260 calculates the regional ratio RT _n with regional target parallax number NT _n and regional effective parallax number NV _n (step S60). When all the regional ratio RT _n value close to 1 (step S70), the determination result Jr and "Zurenashi" (step S80). Otherwise, if all the regional ratios RT _n are similar values and far from 1 (step S90), the determination result Jr is set as “pitch generation” (step S100), and a value close to 1 When distant values are mixed (step S110), the determination result Jr is set to "roll occurrence" (step S120). If neither of the cases is the case, the determination result Jr is set to "indeterminable" (step S130). The determination result Jr is input to the disparity calculating unit 230.

  The parallax calculating unit 230 having received the determination result Jr confirms the determination result Jr (step S140).

  If the determination result Jr is “no shift”, the disparity calculating unit 230 sets the calibration amount CA to 0 (step S150).

When the determination result Jr is "pitch generation", the disparity calculating unit 230, the disparity storage unit 240, the disparity inspection unit 250, and the deviation determining unit 260 determine the amount of deviation due to the pitch (step S160). The shift amount calculation due to the pitch will be described with reference to the flowchart of FIG. The parallax calculating unit 230 shifts the comparison image CP in the vertical direction by a predetermined interval SG (step S510), calculates the parallax of the central region (step S520), and stores the parallax as the parallax data PA in the parallax storage unit 240. At this time, to hold an amount of shift amount SA _m deviated from the original position (step S530). Parallax checking unit 250 performs singular point removal with respect to the parallax data PA stored in the parallax storage unit 240 (step S540), obtains a regional effective parallax number NV _n of the central region, and outputs the deviation determination unit 260 . Deviation determination unit 260 calculates a regional ratio RT _n the central area from the area-based target parallax number NT _n and regional effective parallax number NV _n (step S550), holds. If the number of shifts is not the predetermined number (step S560), the comparison image CP is further shifted in the vertical direction by the predetermined interval SG (step S510), and steps S520 to S550 are repeated. If the number of shifting becomes a predetermined number of times, deviation determination unit 260, the value in the holding areas by the ratio RT _n is elected regional ratio RT _n closest to 1 (step S570), it shifted at that time The number of times is output to the disparity calculating unit 230 as the determination information Jt (step S580). Parallax calculation unit 230, a shift amount SA _m corresponding to the determined information Jt a shift amount by the pitch (step S590), sets the calibration quantity CA.

When the determination result Jr is "roll generation", the shift amount SAP due to the pitch is obtained by the same procedure as in the case of "pitch generation" (step S170). Then, the disparity calculating unit 230 rotates the comparison image CP shifted by SAP by the predetermined angle RG (step S180), calculates the disparity of all the regions (step S190), and sets the disparity storage unit 240 as disparity data PA. Remember to At this time, to hold the rotated amount from the original position as the rotation amount RA _k (step S200). The parallax inspection unit 250 performs singular point removal on the parallax data PA stored in the parallax storage unit 240 (step S210), obtains the number of effective parallaxes per region NV _n , and outputs the number to the deviation judgment unit 260. Deviation determination unit 260 calculates a regional ratio RT _n the area-based target parallax number NT _n and regional effective parallax number NV _n (step S220), holds. If the number of rotations is not the predetermined number (step S230), the comparison image CP is further rotated by the predetermined angle RG (step S180), and steps S190 to S220 are repeated. If the number of rotation reaches a predetermined number, the deviation judgment unit 260, based on the region-specific ratio RT _n holds locate the matching case, if there is a match case (step S240), and rotation in the matching case The number of times is output as the determination information Jt to the disparity calculation unit 230 (step S250). Parallax calculation unit 230, the rotation amount RA _m corresponding to the determined information Jt as the rotation amount of the roll (step S260), sets an additional calibration parameter CA that is set a shift amount by the pitch. If there is no match case, the shift determination unit 260 sets the determination information Jt to 0 and outputs the result to the disparity calculation unit 230 (step S270). When there is a rotation amount not selected among the rotation amounts held in the parallax calculation unit 230 (step S280), the parallax calculation unit 230 selects one rotation amount (step S290), and the rotation is performed. Steps S170 to S280 are repeated based on the comparison image rotated by the amount. If all the rotation amounts have been selected, the disparity calculating unit 230 sets the calibration amount CA to “impossible to calibrate” (step S300).

  Also in the case where the determination result Jr is “indeterminable”, the disparity calculating unit 230 also sets the calibration amount CA to “impossible to calibrate” (step S300).

  When a value is set to the calibration amount CA, the disparity calculating unit 230 outputs the calibration amount CA (step S310).

The operation of calculating the target parallax number NT _n by area (step S20) and the operation from the parallax calculation to the calculation of the effective parallax number NV _n by area (steps S30 to S50) may be performed in reverse order or in parallel. .

  In the first embodiment, the procedure for obtaining the calibration amount differs between when it is determined that a shift due to pitch has occurred and when it is determined that a shift due to roll has occurred, but both shifts often occur simultaneously. The calibration amount may be determined according to the same procedure, that is, according to the procedure when it is determined that the shift due to the roll has occurred, without distinguishing between the two.

  A configuration example (second embodiment) in the case of obtaining the calibration amount in the same procedure as described above is shown in FIG. 11, and has the same configuration as the configuration example of the first embodiment shown in FIG. 230 and the shift determination unit 260 replace the disparity calculation unit 330 and the shift determination unit 360, respectively.

Similar to the shift determination unit 260, the shift determination unit 360 performs shift determination and generation of the determination information Jt in calibration amount calculation, but there is a difference in the shift determination. That is, the deviation determination unit 360, if all the regional ratio RT _n value close to 1, it is determined that the deviation in the same manner as deviation determination unit 260 does not occur, all regional ratio RT _n In the case of similar values and values farther than 1 or when values close to 1 and values farther than 1 are mixed as the area ratio RT _n , it is determined that a shift has occurred (hereinafter, this determination result As “a shift occurrence”), it does not distinguish whether it is a shift due to a pitch or a shift due to a roll. Therefore, possible values of the determination result Jrr are either "no deviation", "occurrence of deviation", or "indeterminable".

  The disparity calculating unit 330 calculates the disparity in the same procedure as the disparity calculating unit 230, but the procedure of calculating the calibration amount CA is different. That is, when the determination result Jrr is "no deviation" or "indeterminable", the same processing as the disparity calculating unit 230 is performed, but when the determination result Jrr is "displacement occurrence", the determination result in the disparity calculating unit 230 Perform the same processing as when Jr is "roll generation".

  The flowchart of the operation example of the second embodiment is as shown in FIG. 12 and FIG. 13 corresponding to the flowchart of the operation example of the first embodiment shown in FIG. 8 and FIG. The operation example of the shift amount calculation due to the pitch is the same as that of FIG.

From the start of the operation until step S60 of calculating the regional ratio RT _n perform the same operations as in the first embodiment. The deviation determination unit 360, if all the regional ratio RT _n value close to 1 (step S70), the determination result Jrr as "Zurenashi" (step S80). Otherwise, if all the regional ratios RT _n are similar values and values farther than 1 or if values close to 1 and values farther than 1 are mixed (step S 91), the judgment result J rr is It is assumed that "deviation occurs" (step S101). If neither of the cases is the case, the judgment result Jrr is set to "impossible to judge" (step S130). The determination result Jrr is input to the disparity calculating unit 330.

  The disparity calculating unit 330 which has input the determination result Jrr confirms the determination result Jrr (step S140), and the operation similar to that of the first embodiment is performed when the determination result Jrr is "no deviation" or "indeterminable". (Step S150, step S300). When the determination result Jrr is "deviation of deviation", the same operation as that of "roll occurrence" in the first embodiment is performed (steps S170 to S300). Then, when a value is set to the calibration amount CA, the disparity calculating unit 330 outputs the calibration amount CA (step S310).

Note that without performing the determination in step S91, the case where all the regional ratio RT _n not satisfied the condition that it is close to 1, i.e., if a nearby value not is regional ratio RT _n to 1 are present Alternatively, the determination result Jrr may be set as "deviation occurrence". In this case, the determination result of "indeterminable" is unnecessary.

If it is determined in the first embodiment that a shift due to the roll has occurred, or if it is determined that a shift has occurred in the second embodiment, the rotation amount due to the roll is determined to increase the accuracy of the calibration of the vertical shift. Fixing the rotation amount, and again shifting the whole image at predetermined intervals in the vertical direction, calculate the number of effective parallaxes per area (the number of shift effective parallaxes) NV _n of not only the central area but the whole area, further calculates a regional ratio RT _n, locate the case where the entire area by the ratio RT _n is close to the most 1, or a shift amount up to the case as a new shift amount by the pitch. This is similar to the case where no matching case is found, the central region has a width, and when determining the amount of rotation by the roll, the center of rotation is not necessarily at the center of the screen, but is translated by the rotation Since an operation is added, it is performed to correct that amount. In this case, the amount of rotation is correct, and as a result, the ratio of the entire screen is degraded, so the ratio of the entire region is targeted, not only the central region.

  FIG. 14 shows a configuration example (third embodiment) in which the above-described processing is added to the second embodiment, and the configuration is the same as the configuration example of the first embodiment and the second embodiment.

When the determination result Jrr is “deviation occurrence”, the parallax computation unit 430 cooperates with the deviation determination unit 460 to obtain the displacement amount due to the pitch and the rotation amount due to the roll, and then fixes this to the rotation amount to obtain the entire image. The parallaxes of all the areas are calculated by shifting in the vertical direction by a predetermined interval SG. After calculating the parallax of all the regions, and outputs the disparity data PA, to hold the shift amount SA _m. Thereafter, while the amount of rotation is fixed, the entire image is further shifted in the vertical direction by a predetermined interval SG until the same processing is shifted a predetermined number of times or a predetermined range. Then, since the times of the information becomes a value close to the entire area by the ratio RT _n most 1 at the deviation judging unit 460 is output as the determination information Jt, new deviation by the pitch deviation amount SA _m of times Amount.

When the determination result Jrr is “a shift occurrence”, the shift determining unit 460 performs the same processing as in the second embodiment, and then the effective parallax number by region NV _n sequentially output from the disparity inspection unit 250 and the holding The regional ratios RT _n of all the regions are calculated and held using the regional disparity numbers NT _n stored. Then, if the entire image is shifted a predetermined number of times or within a predetermined range, the value is the number closest to 1 among the regional ratios RT _n of all the retained regions, for example, the regional ratios RT _n of all the regions. The information of the times when the average value is closest to 1 is output as the determination information Jt.

  In the operation example of the third embodiment, in the operation example of the second embodiment shown in FIGS. 12 and 13, after the rotation amount determination by the roll in step S260, the recalculation of the shift amount due to the pitch is added. An operation example of the recalculation of the shift amount due to the pitch will be described with reference to the flowchart of FIG.

The disparity calculating unit 430 vertically shifts the comparison image CP rotated by the rotation amount obtained in step S260 by a predetermined interval SG (step S610), and calculates the disparity of all the regions (step S620). The data is stored in the disparity storage unit 240 as data PA. At this time, to hold an amount of shift amount SA _m deviated from the original position (step S630). The parallax inspection unit 250 performs singular point removal on the parallax data PA stored in the parallax storage unit 240 (step S 640), obtains the number of effective parallaxes per region NV _n , and outputs the number to the deviation judgment unit 460. The shift determining unit 460 calculates the area ratio RT _n from the area-specific target parallax number NT _n and the area-specific effective parallax number NV _n (step S 650), and holds the ratio. If the number of shifts is not the predetermined number (step S660), the comparison image CP is further shifted in the vertical direction by the predetermined interval SG (step S610), and steps S620 to S650 are repeated. If the number of shifting becomes a predetermined number of times, deviation determination unit 460, retained the regional ratio RT _n near the average value of the regional ratio RT _n of all areas in the most 1 in each region proportions RT _n The selected number of times is output as the determination information Jt to the disparity calculation unit 430 (step S680). Parallax calculation unit 430, a shift amount SA _m corresponding to the determined information Jt as a new shift amount by the pitch (step S690), replaces the set displacement amount in the calibration amount CA.

  Among the processes in the above-described embodiments (first to third embodiments), calculation of difference in luminance between adjacent pixels in the horizontal direction and singular point removal are performed in parallax image generation with a conventional stereo camera. There are also devices that conventionally have the function of shifting the image or the function of rotating it. Therefore, in a device having such processing and functions, it is possible to perform the determination and calibration of the deviation only by adding the processing based on the count of the target disparity count by region and the effective disparity count by region. The addition of new configuration can be suppressed, and the increase in cost and operation time can be suppressed. Further, the processing according to the above-described embodiment is not limited to being performed alone, but may be performed at a ratio of one frame to ten frames, for example, when images are sequentially output from the stereo imaging device 10 The present invention can be carried out as a background without inhibiting ordinary three-dimensional object detection processing and the like.

  The threshold value TD used by the luminance information extraction unit 220 is not a fixed value, but may be a value determined based on the average value of the calculated differences in luminance. In addition, when the calculated target parallax number by area is smaller than a predetermined number, it may be considered as “impossible to calibrate” because it is not possible to accurately determine the occurrence of misalignment and calculate the calibration amount.

In the search for the pixel FCPE in the parallax calculation unit, not the size of the difference between the brightness of the pixel SPE _{ij and} the brightness of the pixel CPE _ij but the square of the difference or the like may be used. Furthermore, in the calculation of the parallax, in order to increase the shift accuracy, the parallax with sub-pixel accuracy may be obtained. For example, FIG. 16 is a diagram simulating the magnitude of the difference in luminance in part of the search range shown in FIG. 4, and it is assumed that the value is the minimum value in s2. In FIG. 16, it is assumed that the size of the difference in luminance can be linearly approximated near the minimum value, and the position of one of s2 of the position at which the value is minimum and the value before and after it is larger (s3 in FIG. 16). ) Is drawn by a straight line (line s2s3 in FIG. 16), and a straight line passing the position (s1 in FIG. 16) whose inclination is opposite to that of the straight line and smaller in value is drawn. Position. The difference between the true position and the position in the lateral direction of the pixel SPE _ij is taken as the parallax. The value near the minimum value may be approximated not by a linear approximation but by a quadratic curve or the like.

  In the above embodiment, the calibration amount may be obtained by other means, etc., so that only the occurrence of the shift may be detected without obtaining the calibration amount, and the type of shift (vertical shift and rotational shift) If the specification is also unnecessary, the screen may not be divided into a plurality of areas.

  The calibration apparatus according to the present invention has been described above. However, the present invention is not limited to the above-described embodiment, and it goes without saying that various other configurations and methods can be adopted without departing from the scope of the present invention. For example, the parallax calculating unit and the deviation determining unit may be one. Further, the calibration device according to the present invention can be implemented by software (computer program) using a computer system, and ASIC (Application Specific Integrated Circuit), GPU (Graphics Processing Unit), FPGA (Field Programmable) It can of course be implemented by hardware such as Gate Array).

DESCRIPTION OF SYMBOLS 10 Stereo image pickup apparatus 11, 12 Camera 20, 30, 40 Calibration apparatus 210 Image storage part 220 Brightness information extraction part 230, 330, 430 Parallax operation part 240 Parallax storage part 250 Parallax inspection part 260, 360, 460 Shift judgment part

Claims (23)

In a calibration device for calibrating a shift that occurs due to an installation state of a stereo image pickup device,
A luminance information extraction unit which obtains, as a target parallax number, the number of points satisfying a predetermined condition among the points at which the luminance changes in the reference image imaged by the stereo image pickup device;
A disparity calculating unit that calculates disparity using the comparison image captured by the stereo image capturing apparatus and the reference image;
A parallax inspection unit which obtains the number of effective parallaxes among the parallaxes as an effective parallax number;
A calibration apparatus comprising: a shift determining unit that determines that the shift is occurring based on a difference between the target number of parallaxes and the number of effective parallaxes.   The calibration device according to claim 1, wherein the luminance information extraction unit sets, as the target number of parallaxes, the number of pixels in which the difference in luminance between the adjacent pixels in the reference image is larger than a threshold.   The calibration apparatus according to claim 1, wherein the parallax inspection unit sets the number of parallaxes excluding the parallax having a large difference between the peripheral parallax and the value as the effective parallax number.   The calibration apparatus according to any one of claims 1 to 3, wherein the deviation determining unit determines that the deviation does not occur when the ratio of the number of effective parallaxes to the target number of parallaxes is a value close to one. Dividing a screen for the reference image and the comparison image into a plurality of areas;
The luminance information extraction unit obtains the target number of parallaxes as the target number of parallaxes by area for each of the areas,
The disparity calculating unit calculates the disparity for each of the areas,
The parallax inspection unit obtains the number of effective parallaxes as an area-specific effective parallax number for each area using the parallax for each area.
The calibration apparatus according to any one of claims 1 to 3, wherein the shift determining unit determines that the shift is occurring based on a difference between the target disparity count by region and the effective disparity count by region. The deviation judgment unit
If the ratio by region, which is the ratio of the number of effective parallaxes by region to the number of target parallaxes by region, is a value close to 1 at all, it is determined that the deviation has not occurred.
If all the area ratios are substantially the same value and more than 1 value, it is determined that a shift due to the pitch has occurred,
The calibration apparatus according to claim 5, wherein it is determined that a shift due to a roll has occurred when the area ratio having a value close to 1 and the area ratio having a value further than 1 coexist. If the deviation determination unit determines that the deviation due to the pitch has occurred,
While the reference image or the comparison image is shifted at predetermined intervals in the direction of the pitch, the parallax calculating unit and the parallax inspecting unit determine the number of effective parallaxes by region in the central region located at the center of the screen.
7. The calibration apparatus according to claim 6, wherein an amount obtained by shifting the reference image or the comparison image when the regional ratio in the central region is closest to 1 is taken as a shift amount due to the pitch. When the shift determination unit determines that the shift due to the roll is occurring,
While the reference image or the comparison image is shifted at predetermined intervals in the direction of the pitch, the parallax calculating unit and the parallax inspecting unit determine the number of effective parallaxes by region in the central region located at the center of the screen.
The amount by which the reference image or the comparison image is shifted when the ratio by region in the central region is closest to 1 is defined as the amount of shift by the pitch.
While rotating the reference image or the comparison image shifted by the shift amount at a predetermined angle with the center of the screen as the origin, the parallax calculating unit and the parallax inspection unit rotate the effective parallax number by area and effective rotation Calculated as the parallax number,
The rotation amount by the roll according to claim 6, wherein an angle obtained by rotating the reference image or the comparison image when the ratio of the number of effective rotation parallaxes to the target number of parallaxes by region is all close to 1 is a rotation amount by the roll. Calibration device. The deviation judgment unit
If the ratio by region, which is the ratio of the number of effective parallaxes by region to the number of target parallaxes by region, is a value close to 1 at all, it is determined that the deviation has not occurred.
If the regional ratios all have substantially the same value and a value farther than 1 or if the regional ratios having values close to 1 and the regional ratios having values farther than 1 coexist, the deviation occurs. The calibration apparatus according to claim 5, wherein it is determined that If the deviation determining unit determines that the deviation is occurring,
While the reference image or the comparison image is shifted at a predetermined interval in the pitch direction, the parallax calculating unit and the parallax inspecting unit determine the number of effective parallaxes by region in the central region located at the center of the screen.
The amount by which the reference image or the comparison image is shifted when the ratio by region in the central region is closest to 1 is defined as the amount of shift by the pitch.
While rotating the reference image or the comparison image shifted by the shift amount at a predetermined angle with the center of the screen as the origin, the parallax calculating unit and the parallax inspection unit rotate the effective parallax number by area and effective rotation Calculated as the parallax number,
10. The calibration according to claim 9, wherein an angle obtained by rotating the reference image or the comparison image when the ratio of the number of effective rotation parallaxes to the target number of parallaxes by region is all close to 1 is the amount of rotation by the roll. apparatus. The deviation judgment unit
After the amount of rotation by the roll is determined, the parallax calculating unit and the parallax inspection unit separate each of the regions while shifting the reference image or the comparison image rotated by the amount of rotation at predetermined intervals in the direction of the pitch. Determine the number of effective parallaxes as the number of shift effective parallaxes,
The calibration according to claim 8 or 10, wherein the shift amount due to the pitch is updated by an amount obtained by shifting the reference image or the comparison image when the ratio of the number of effective shift parallaxes to the target number of parallaxes by region is closest to one. apparatus. In a calibration method for calibrating a shift generated due to an installation state of a stereo image pickup device,
A luminance information extraction step of obtaining, as a target parallax number, the number of points satisfying a predetermined condition among the points at which the luminance changes in the reference image imaged by the stereo image pickup device;
A disparity calculating step of calculating disparity using the comparison image captured by the stereo image capturing device and the reference image;
A parallax inspection step of obtaining the number of effective parallaxes among the parallaxes as an effective parallax number;
And a deviation determining step of determining that the deviation is occurring based on a difference between the target number of parallaxes and the effective number of parallaxes.   The calibration method according to claim 12, wherein, in the luminance information extraction step, the number of pixels having a luminance difference larger than a threshold with respect to an adjacent pixel in the reference image is set as the target parallax number.   The calibration method according to claim 12 or 13, wherein, in the parallax inspection step, the number of parallaxes excluding parallaxes having a large difference between the peripheral parallax and the value is set as the effective parallax number.   The calibration method according to any one of claims 12 to 14, wherein in the deviation determining step, when the ratio of the number of effective parallaxes to the target number of parallaxes is a value close to 1, the deviation is determined not to occur. Dividing a screen for the reference image and the comparison image into a plurality of areas;
In the luminance information extraction step, the target number of parallaxes is obtained as the target number of parallaxes by area for each of the areas,
In the parallax calculation step, the parallax is calculated for each of the areas,
In the parallax inspection step, the number of effective parallaxes is calculated as an area-specific effective parallax number for each area using the parallax for each area.
The calibration method according to any one of claims 12 to 14, wherein in the deviation determining step, the occurrence of the deviation is determined based on a difference between the area-specific target parallax number and the area-specific effective parallax number. In the deviation determination step,
If the ratio by region, which is the ratio of the number of effective parallaxes by region to the number of target parallaxes by region, is a value close to 1 at all, it is determined that the deviation has not occurred.
If all the area ratios are substantially the same value and more than 1 value, it is determined that a shift due to the pitch has occurred,
The calibration method according to claim 16, wherein it is determined that a shift due to a roll occurs when the area ratio having a value close to 1 and the area ratio having a value further than 1 coexist. If it is determined in the deviation determination step that a deviation due to the pitch has occurred,
While shifting the reference image or the comparison image at predetermined intervals in the direction of the pitch, the number of effective parallaxes by area in the central area located at the center of the screen is determined by the parallax calculation step and the parallax inspection step.
18. The calibration method according to claim 17, wherein an amount obtained by shifting the reference image or the comparison image when the regional ratio in the central region is closest to 1 is set as a shift amount due to the pitch. If it is determined in the shift determination step that a shift due to the roll is occurring,
While shifting the reference image or the comparison image at predetermined intervals in the direction of the pitch, the number of effective parallaxes by area in the central area located at the center of the screen is determined by the parallax calculation step and the parallax inspection step.
The amount by which the reference image or the comparison image is shifted when the ratio by region in the central region is closest to 1 is defined as the amount of shift by the pitch.
While rotating the reference image or the comparison image shifted by the shift amount at a predetermined angle with the center of the screen as the origin, the effective parallax number by region is rotated by the parallax calculation step and the parallax inspection step Calculated as the parallax number,
The rotation amount by the roll according to claim 17, wherein an angle obtained by rotating the reference image or the comparison image when the ratio of the rotation effective parallax number to the region-specific target parallax number is all close to 1 is a rotation amount by the roll. Calibration method. In the deviation determination step,
If the ratio by region, which is the ratio of the number of effective parallaxes by region to the number of target parallaxes by region, is a value close to 1 at all, it is determined that the deviation has not occurred.
If the regional ratios all have substantially the same value and a value farther than 1 or if the regional ratios having values close to 1 and the regional ratios having values farther than 1 coexist, the deviation occurs. The calibration method according to claim 16, wherein it is determined that If it is determined in the deviation determining step that the deviation is occurring,
While the reference image or the comparison image is shifted at predetermined intervals in the direction of the pitch, the effective parallax number by area in the central area located at the center of the screen is determined by the parallax calculation step and the parallax inspection step.
The amount by which the reference image or the comparison image is shifted when the ratio by region in the central region is closest to 1 is defined as the amount of shift by the pitch.
While rotating the reference image or the comparison image shifted by the shift amount at a predetermined angle with the center of the screen as the origin, the effective parallax number by region is rotated by the parallax calculation step and the parallax inspection step Calculated as the parallax number,
21. The calibration according to claim 20, wherein an angle obtained by rotating the reference image or the comparison image when the ratio of the number of effective rotation parallaxes to the target number of parallaxes by region is all close to 1 is the amount of rotation by roll. Method. After the rotation amount by the roll is obtained in the shift determination step, the parallax calculation step and the parallax inspection are performed while shifting the reference image or the comparison image rotated by the rotation amount at a predetermined interval in the direction of the pitch. According to the step, the area-specific effective parallax number is determined as a shift effective parallax number,
22. The calibration according to claim 19, wherein the shift amount due to the pitch is updated by an amount obtained by shifting the reference image or the comparison image when the ratio of the number of effective shift parallaxes to the target number of parallaxes by region is closest to one. Method. A calibration program for executing the calibration method according to any one of claims 12 to 22.