JP6446794B2 - Stereo camera and optical system - Google Patents

Description

  The present invention relates to a stereo camera and an optical system.

  In recent years, monitoring devices using a pair of cameras (stereo cameras) (for example, in-vehicle stereo cameras) have attracted attention. The monitoring apparatus uses a stereo method, which is one of three-dimensional measurement techniques. In the stereo method, the parallax of a subject image is calculated from a reference image including an image of the subject captured by one camera and a comparison image including the image of the same subject captured by the other camera. The distance information to the subject is calculated using the principle of surveying. At this time, in order to calculate the parallax of the image of the subject, it is necessary to specify the pixel block of the comparative image having a correlation with the pixel block in the reference image. Therefore, in order to increase the accuracy of the calculated distance information, it is preferable that the two cameras have the same resolution as much as possible.

  In Patent Document 1, parallax is obtained using an image corrected so that the aberration distribution of the other optical system is the same with respect to the aberration distribution of the other optical system with respect to the optical systems having different aberration distributions. A parallax detection device is disclosed that can suppress parallax detection errors by calculating, even when a camera having a low lens resolution with respect to the pixel pitch of the image sensor is used.

  However, with the conventional technology, if the image plane position fluctuates due to errors in assembling the stereo camera or changes in the position of the stereo camera over time, stereo cameras with different resolutions are configured between the left and right cameras, and parallax detection is performed. There was a problem that the error increased.

  The present invention has been made in view of the above, and a stereo camera capable of preventing a change in the resolution of the stereo camera due to, for example, an assembly error of the stereo camera or a temporal change in the image plane position of the stereo camera, and the like An object is to provide an optical system.

In order to solve the above-described problems and achieve the object, the present invention provides a change amount according to the defocus amount of MTF (Modulation Transfer Function) as an MTF according to the defocus amount of an ideal optical system having no aberration. A first optical system and a second optical system, each of which is lower than an amount of change; a first image sensor that captures a reference image including an image of a subject using the first optical system; and the second optical system. A second image sensor that uses the second image sensor to capture a comparison image including the subject image; and a calculation unit that calculates parallax of the subject image from the reference image and the comparison image.

  According to the present invention, it is possible to prevent a change in the resolution of the stereo camera due to an error in assembling the stereo camera or a change in the image plane position of the stereo camera over time.

FIG. 1 is a diagram for explaining the principle of distance measurement by the stereo method. FIG. 2 is a diagram for explaining the parallax of the image of the subject. FIG. 3 is a diagram illustrating the position of the subject image of the comparison image and the position of the subject image of the reference image. FIG. 4 is a diagram for explaining the corresponding point search processing. FIG. 5 is a diagram illustrating an example of a parallax measurement object. FIG. 6 is a diagram illustrating the relationship between the MTF difference between the first optical system and the second optical system and the parallax variation. FIG. 7 is a diagram illustrating the relationship between the MTF reduction amount and the parallax variation when there is no MTF difference between the first optical system and the second optical system (when defocusing is performed in the same manner). FIG. 8 is a diagram showing changes in the MTF characteristics according to the defocus of the ideal optical system. FIG. 9 is a diagram showing an MD curve (MTF-Defocus curve) in a region of a spatial frequency that is 1/4 of the diffraction limit of the ideal optical system. FIG. 10 is a diagram showing an MD curve obtained by normalizing the two MD curves in FIG. 9 with an MTF peak (maximum value). FIG. 11 is a diagram illustrating an example of the configuration of the stereo camera of the embodiment. FIG. 12 is a diagram illustrating an example of design values (before giving aberration) of the first optical system (second optical system) of the embodiment. FIG. 13 is a diagram showing a change in MTF (absolute value of OTF) in accordance with a change in image plane position of the first optical system (second optical system) having the design value shown in FIG. FIG. 14 is a diagram showing an example of design values of an aspheric shape (wavefront modulation surface). FIG. 15 shows the change in the image plane position after introducing the wavefront modulation surface having the design value shown in FIG. 14 between the surface 6 and the surface 7 of the first optical system (second optical system) having the design value shown in FIG. It is a figure which shows the change of MTF (absolute value of OTF) according to. FIG. 16 is a diagram illustrating an example in which the stereo camera of the embodiment is used as an in-vehicle stereo camera. FIG. 17 is a diagram showing MTF characteristics before and after image processing by the restoration unit.

  Hereinafter, embodiments of a stereo camera and an optical system will be described in detail with reference to the accompanying drawings.

First, the principle of distance measurement by parallax and the stereo method will be described. The parallax is calculated using a captured image captured by the stereo camera. FIG. 1 is a diagram for explaining the principle of distance measurement by the stereo method. In the example of FIG. 1, the first camera 1 (focal length f, optical center O ₀ , imaging surface S ₀ ) is arranged with the Z axis as the optical axis direction. A second camera 2 (focal length f, optical center O ₁ , imaging surface S ₁ ) is disposed with the Z axis as the optical axis direction. The first camera 1 and the second camera 2 are arranged in parallel to the X axis at positions separated by a distance B (baseline length).

The subject A located at a distance d from the optical center O ₀ of the first camera 1 in the optical axis direction forms an image at P ₀ , which is the intersection of the straight line A-O ₀ and the imaging surface S ₀ . On the other hand, the second camera 2, the same subject A is forms an image at a position _{P 1} on the imaging surface _{S 1.} Hereinafter, a photographed image obtained from the imaging surface S ₀ of "comparison image". The captured image acquired from the imaging surface S ₁ referred to as "reference image".

Here, an intersection of a straight line passing through the optical center O ₁ of the second camera 2 and parallel to the straight line A-O ₀ and the imaging surface S ₁ is defined as P ₀ ′. Further, let D be the distance between P ₀ ′ and P ₁ . The distance D represents a positional deviation amount (parallax) on an image obtained by photographing the same subject image with two cameras. The triangle AO ₀ -O ₁ and the triangle O ₁ -P ₀ ′ -P ₁ are similar. Therefore, d = B × f / D is established. That is, the distance d to the subject A can be obtained from the base line length B, the focal length f, and the parallax D.

  The above is the distance measurement principle by the stereo method. In order to calculate the parallax of the image of the subject, it is necessary to specify the pixel block of the comparison image having a correlation with the pixel block in the reference image. Next, a method for specifying a pixel block of a comparative image having a correlation with a pixel block in the reference image will be described with reference to FIGS.

  FIG. 2 is a diagram for explaining the parallax of the image of the subject. When the subject A is at a certain distance, the image 7 of the subject A is projected onto the first image sensor 5 (comparison image) via the first optical system 3 of the first camera 1. Similarly, the image 8 of the subject A is projected onto the second image sensor 6 (reference image) via the second optical system 4 of the second camera 2.

  FIG. 3 is a diagram showing the position of the image 7 of the subject A in the comparison image and the position of the image 8 of the subject A in the reference image. The coordinates of the image 7 of the subject A of the first image sensor 5 (comparison image) are different from the coordinates of the image 8 of the subject A of the second image sensor 6 (reference image). In order to detect parallax, it is necessary to correctly check the correspondence (correlation) between each pixel of the comparison image and each pixel of the reference image. This process is called corresponding point search.

  FIG. 4 is a diagram for explaining the corresponding point search processing. In the corresponding point search process, first, a small calculation area 9 (window) is assumed around the image 8 (reference point) of the subject A projected on the reference image. Next, a window 10 having the same size is taken in the comparison image, and the corresponding point is detected by comparing the window 10 with the window 9 of the reference image while shifting the window 10. There are various comparison methods at this time. For example, SSD (Sum of Squared Difference) is known as the most basic method. In SSD, the sum of squares of the difference in luminance value of each pixel is calculated, and the point where the sum of squares of the difference in luminance value of each pixel is minimum within the search range is detected as a corresponding point. At this time, sub-pixel level estimation is performed to increase the corresponding point detection resolution. For example, in a subpixel estimation method called equiangular straight line fitting, corresponding points are estimated by linear linear interpolation. The difference between the corresponding point detected in this way and the reference point of the reference image is calculated as the parallax D. At this time, if the resolution of the reference image is different from the resolution of the comparison image, the accuracy of the subpixel estimation method is lowered.

  Next, referring to FIG. 5 to FIG. 7, the resolution of the first camera 1 (MTF (Modulation Transfer Function) of the first optical system 3) and the resolution of the second camera 2 (MTF of the second optical system 4). And the simulation result of the parallax detection error when they are different from each other.

  FIG. 5 is a diagram illustrating an example of a parallax measurement object. FIG. 5 is an example of a parallax measurement object having a planar shape including a human image having a height of 180 cm and a width of 70 cm. The simulation result of the parallax detection error of the comparative image and the reference image when the measurement object existing 60 m ahead is projected via the first optical system 3 and the second optical system 4 will be described. The parallax detection error simulation is performed after the distortion correction of the lens of the first optical system 3 is performed on the comparative image and the distortion correction of the lens of the second optical system 4 is performed on the reference image. That is, the parallax detection error is simulated after making the comparison image and the reference image idealized (pinhole image).

  At this time, as a design value of the first optical system 3, a design value intentionally defocused from the design value of the ideal state of the first optical system 3 is used. Note that the design value of the second optical system 4 is an ideal design value.

  FIG. 6 is a diagram showing the relationship between the MTF difference between the first optical system 3 and the second optical system 4 and the parallax variation σ. The MTF difference between the first optical system 3 and the second optical system 4 is the difference between the MTF of the second optical system 4 in the ideal state and the MTF of the first optical system 3 on which the image plane is intentionally defocused. From FIG. 6, it can be confirmed that the larger the MTF difference, the larger the parallax variation σ. In other words, as described above, when the resolution of the reference image is different from the resolution of the comparison image, the parallax detection error is increased.

  Next, a change in the parallax variation σ when the resolution of the first camera 1 and the resolution of the second camera 2 simultaneously decrease by the same amount (when defocused in the same manner) will be described. FIG. 7 is a diagram illustrating the relationship between the MTF reduction amount and the parallax variation σ when there is no MTF difference between the first optical system 3 and the second optical system 4 (when defocusing is performed in the same manner). From FIG. 7, it can be confirmed that when the MTF decreases simultaneously in the first optical system 3 and the second optical system 4, the parallax variation σ hardly changes even when the amount of decrease in MTF is increased.

  From the above results, the relationship between the parallax detection error and the resolution changes greatly when the resolution of one camera decreases, but regarding the decrease in the resolution of the binocular camera, the parallax detection is performed even if the amount of resolution decrease is increased. It can be seen that there is almost no change in the error. From the above relationship, it can be seen that it is desirable for the optical system constituting the stereo camera to lower the sensitivity of the MTF change due to defocusing by dropping the peak (maximum value) of the MTF. As a result, even if the first camera 1 or the second camera 2 is defocused due to an assembly error of the stereo camera 30 or a temporal change in the image plane position of the stereo camera, the resolution and reference of the comparison image The difference from the image resolution can be reduced.

  Next, a method for designing an optical system that lowers the maximum value of the MTF and reduces the sensitivity of the change of the MTF according to the defocus amount will be described.

  First, consider the defocus characteristics of an ideal optical system (ideal lens) in which no aberration exists. The pupil coordinates of the ideal optical system can be expressed by the following equation (1).

  Here, considering the influence of aberration on the ideal optical system and introducing the aberration term W (X) and the amplitude term A (X), the pupil function including the aberration can be expressed by the following equation (2).

  Next, when the following equation (2) is substituted into equation (3) of PSF (Point spread function: h (x) calculated from Huygens' theorem, the following equation (4) is established.

ここで、次式（５）のように、座標系を２次元（ｘ，ｚ）から３次元（ｘ，ｙ，ｚ）に拡張し、Ｘ平面から、ｘ平面までの距離を考える。

Here, as in the following equation (5), the coordinate system is expanded from two dimensions (x, z) to three dimensions (x, y, z), and the distance from the X plane to the x plane is considered.

ここで、Ｘ^２＋Ｙ^２＜＜Ｚ^２とすると、次式（６）が成り立つ。

Here, when X ² + Y ² << Z ² , the following equation (6) is established.

  Substituting equation (6) into equation (4) yields the following equation (7).

  Here, when the two-dimensional Fourier transform is represented by FT, the two-dimensional Fourier transform formula can be represented by the following formula (8).

  When the phase term is normalized by k / z, the following equation (9) is obtained from equations (7) and (8).

  In the ideal optical system, when the subject is at the in-focus position, there is no aberration, so P (X, Y) = 1. Here, when the subject position is deviated by Δz from the in-focus position, it should be noted that the effect up to the in-focus position is included in the expression x. The distance from the focal point is added as a phase. Therefore, the pupil function can be expressed by the following equation (10).

  Substituting equation (10) into equation (9) yields the following equation (11).

式（１１）がフォーカスずれの影響を考慮したＰＳＦを求める理論式となる。

Expression (11) is a theoretical expression for obtaining the PSF in consideration of the influence of the focus shift.

  FIG. 8 is a diagram showing changes in the MTF characteristics according to the defocus of the ideal optical system. FIG. 8A shows the MTF characteristics when the position of the subject is in the in-focus position. 8B to 8E show the MTF characteristics when the defocus amount is increased in the order from FIG. 8B to FIG. 8E. From FIG. 8, when a certain defocus amount is given, it can be confirmed that the MTF passes through the zero point at a spatial frequency of about 1/4 from the diffraction limit of the ideal optical system (FIG. 8 (e)). . Therefore, the MTF peak (maximum value) is lowered in the region of 1/4 of the spatial frequency that is the diffraction limit in the ideal optical system, so that the sensitivity of the MTF change due to defocusing is lowered.

  FIG. 9 is a diagram showing an MD curve (MTF-Defocus curve) in a region of a spatial frequency that is 1/4 of the diffraction limit of the ideal optical system. The MD curve 41 is a defocus characteristic of an ideal optical system having no aberration. It has a peak at Δz = 0, and it can be confirmed that the MTF decreases as Δz changes.

  Here, the influence of spherical aberration is added to the ideal optical system. The phase term of the spherical aberration pupil function of the Zernike polynomial can be expressed by the following equation (12).

  Substituting equation (12) into equation (11) and combining the effect of defocus and the effect of spherical aberration yields the following equation (13).

  Here, p is the amount of spherical aberration. By giving predetermined various aberrations (spherical aberration in the present embodiment) to the ideal optical system described above, the MD characteristic indicated by the MD curve 42 in FIG. 9 is obtained. It can be confirmed that the M-D curve 42 has a lower MTF peak than the M-D curve 41 of the ideal optical system, and the sensitivity of the MTF change according to the defocus amount is lowered.

  FIG. 10 is a diagram showing an MD curve obtained by normalizing the two MD curves in FIG. 9 with an MTF peak (maximum value). It can be confirmed that the MD curve 44 of the optical system to which spherical aberration is added always shows a higher value than the MD curve 43 of the ideal optical system. In other words, the amount of change in MTF according to Δz (defocus amount) of the optical system with some aberrations is smaller than the amount of change in MTF according to Δz (defocus amount) of the ideal optical system. I can confirm that. As described above, by giving predetermined various aberrations to the ideal optical system, it is possible to reduce the peak of the MTF and reduce the sensitivity of the change of the MTF according to the defocus amount.

  Hereinafter, an example of the configuration of the stereo camera 30 including the optical system (the first optical system 3 and the second optical system 4) in which the above-described predetermined various aberrations are given to the ideal optical system will be specifically described.

  FIG. 11 is a diagram illustrating an example of the configuration of the stereo camera 30 according to the embodiment. The stereo camera 30 according to the embodiment includes a first camera 1, a second camera 2, and a parallax calculation device 20. The first camera 1 includes a first optical system 3 and a first image sensor 5. The second camera 2 includes a second optical system 4 and a second image sensor 6. The first optical system 3 and the second optical system 4 are optical systems having the same optical characteristics.

  First, an example of design values (before giving aberration) of the first optical system 3 (second optical system 4) of the embodiment will be described.

  FIG. 12 is a diagram illustrating an example of design values of the first optical system 3 (second optical system 4) according to the embodiment (before giving aberration). FIG. 13 shows a change in MTF (absolute value of an optical transfer function (OTF)) corresponding to a change in image plane position of the first optical system 3 (second optical system 4) of the design value of FIG. FIG. In FIG. 13, the spatial frequency of the image plane is 60 lp / mm. From FIG. 13, it can be confirmed that if the image plane fluctuates in the 20 μm plus direction from the in-focus position, the MTF is reduced by about 20%.

  Here, in order to drop the peak of MTF, a wavefront modulation surface is introduced between the surface 6 and the surface 7 in FIG. By introducing a wavefront modulation surface around the stop, aberration can be added to the entire light beam. The aspherical shape of the wavefront modulation surface is expressed by the following equation (14).

Here, _{a n} is the n-th order aspherical coefficient, h is the distance from the center, H is the normalized reference circle radius (Normalized Radius). FIG. 14 is a diagram showing an example of design values of an aspheric shape (wavefront modulation surface). 15 shows the position of the image plane after the wavefront modulation surface having the design value shown in FIG. 14 is introduced between the surface 6 and the surface 7 of the first optical system 3 (second optical system 4) having the design value shown in FIG. It is a figure which shows the change of MTF (absolute value of OTF) according to change. From FIG. 15, it can be confirmed that when the wavefront modulation surface is introduced, even if the image plane moves in the plus direction of 20 μm from the in-focus position, unlike the case of the example of FIG. . From FIG. 15, it can be confirmed that when the wavefront modulation surface is introduced, the peak of MTF is lower than that in FIG. That is, the wavefront modulation surface having the design value shown in FIG. 14 is introduced between the surfaces 6 and 7 of the first optical system 3 (second optical system 4) having the design value shown in FIG. As a result, the peak of the MTF is dropped, and the sensitivity of the change of the MTF according to the defocus amount can be lowered.

  Returning to FIG. 11, the first optical system 3 (second optical system 4) of the embodiment is between the surface 6 and the surface 7 of the first optical system 3 (second optical system 4) having the design value of FIG. 12. 14 is implemented as an optical system in which the peak of the MTF is reduced and the change amount of the MTF according to the defocus amount is reduced by introducing the wavefront modulation surface having the design value of FIG.

  The first image sensor 5 acquires a comparative image including the image 7 of the subject A projected via the first optical system 3 (see FIGS. 2 and 3). The first image sensor 5 inputs the comparison image to the parallax calculation device 20. Similarly, the second image sensor 6 acquires a reference image including an image 8 of the subject A projected through the second optical system 4 (see FIGS. 2 and 3). The second image sensor 6 inputs the reference image to the parallax calculation device 20.

  The parallax calculation device 20 includes a reception unit 21, a first correction unit 22, a second correction unit 23, a storage unit 24, a calculation unit 25, and a restoration unit 26.

  The receiving unit 21 receives an input of a comparison image. The receiving unit 21 outputs the comparison image to the first correction unit 22. The receiving unit 21 receives an input of a reference image. The receiving unit 21 outputs the reference image to the second correction unit 23.

  The first correction unit 22 receives the comparison image from the reception unit 21. The first correction unit 22 corrects the comparison image using a predetermined correction parameter. The correction parameter is a parameter for performing correction according to the usage pattern of the stereo camera 30, for example.

  FIG. 16 is a diagram illustrating an example in which the stereo camera 30 according to the embodiment is used as an in-vehicle stereo camera. In the example of FIG. 16, the stereo camera 30 captures an object through the windshield 15. At this time, since the light beam emitted from the subject is refracted on the front and back surfaces of the windshield 15, the position of the subject image projected on the first image sensor 5 is the position of the subject image that arrives without the windshield 15. Deviation from position.

  Returning to FIG. 11, the first correction unit 22 corrects the comparative image using a correction parameter for correcting the positional deviation of the image of the subject due to the influence of the windshield 15, for example. The first correction unit 22 outputs the corrected comparison image to the calculation unit 25 and the restoration unit 26.

  Similarly, the second correction unit 23 receives the reference image from the reception unit 21. The second correction unit 23 corrects the reference image using the correction parameters described above. The second correction unit 23 outputs the corrected reference image to the calculation unit 25 and the restoration unit 26.

  The storage unit 24 stores the above correction parameters.

  The calculation unit 25 receives the corrected comparison image from the first correction unit 22 and receives the corrected reference image from the second correction unit 23. The calculation unit 25 calculates the parallax from the subject image included in the corrected comparison image and the subject image included in the corrected reference image. The calculation unit 25 calculates a parallax for each pixel, and generates a parallax image in which the parallax is expressed by a density value.

  The restoration unit 26 receives the corrected comparison image from the first correction unit 22 and receives the corrected reference image from the second correction unit 23. The restoration unit 26 restores the MTF of the comparison image and the MTF of the reference image.

  Specifically, the restoration unit 26 restores the MTF of the comparison image and the MTF of the reference image by inverse conversion processing using a general inverse filter or Wiener filter, or image processing using a maximum entropy method or the like. The restoration unit 26 of the present embodiment performs image processing using a Wiener filter.

  The restoration unit 26 performs image processing using an image processing filter R defined by the following equation (15).

Here, H represents the OTF of the optical system, S ² represents the power spectrum of the subject, and W ² represents the power spectrum of noise unique to the sensor. FIG. 17 is a diagram illustrating MTF characteristics before and after image processing by the restoration unit 26. FIG. 17A shows the MTF characteristics before image processing by the restoration unit 26. FIG. 17B shows MTF characteristics after image processing by the restoration unit 26. From FIG. 17, it can be confirmed that the high-frequency MTF characteristic is restored by the image processing of the restoration unit 26. With this restoration processing, even if the first optical system 3 and the second optical system 4 of the embodiment are used, it is possible to maintain or improve the recognition accuracy of a fine pattern such as a sign or a distant subject.

  Returning to FIG. 11, the restoration unit 26 restores the MTF characteristics of the comparison image by the above-described image processing, thereby generating a luminance image of the first camera 1 with improved resolution. Similarly, the restoration unit 26 restores the MTF of the reference image to generate a luminance image of the second camera 2 with improved resolution.

  As described above, the stereo camera 30 according to the embodiment reduces the sensitivity of the change in the MTF according to the defocus amount by dropping the peak (maximum value) of the MTF. Thereby, even when the position of the first image sensor 5 or the second image sensor 6 fluctuates from the design value due to the influence of the assembly error or the change with time, the change in the resolution of the comparison image or the reference image can be suppressed. Therefore, according to the stereo camera 30 of the embodiment, the parallax detection error can be reduced.

  In the description of the embodiment, the wavefront modulation surface having the design value shown in FIG. 14 is introduced between the surface 6 and the surface 7 of the first optical system 3 (second optical system 4) having the design value shown in FIG. However, the surface around the stop (for example, the surface 6) may be aspherical, so that the maximum value of MTF may be reduced and the amount of change in MTF corresponding to the defocus amount may be reduced.

DESCRIPTION OF SYMBOLS 1 1st camera 2 2nd camera 3 1st optical system 4 2nd optical system 5 1st image sensor 6 2nd image sensor 15 Windshield 20 Parallax calculation device 21 Reception part 22 1st correction | amendment part 23 2nd correction | amendment part 24 Memory | storage Unit 25 Calculation unit 26 Restoration unit 30 Stereo camera

Japanese Patent No. 4782899

Claims (7)

A first optical system and a second optical system in which the amount of change according to the defocus amount of MTF (Modulation Transfer Function) is lower than the amount of change of MTF according to the defocus amount of an ideal optical system without aberration; ,
A first image sensor that captures a reference image including an image of a subject using the first optical system;
A second image sensor that captures a comparative image including an image of the subject using the second optical system;
A calculation unit for calculating parallax of the image of the subject from the reference image and the comparison image;
Stereo camera with The stereo camera according to claim 1, wherein the amount of change in MTF according to the defocus amount is targeted for a region having a spatial frequency that is ¼ of the diffraction limit of the ideal optical system. The first and second optical systems have at least one aspherical shape, thereby reducing the maximum value of the MTF and reducing the amount of change in MTF according to the defocus amount. 2. The stereo camera according to 2. The stereo camera according to claim 3, wherein the aspheric shape is provided on a diaphragm of the first and second optical systems, or on a lens surface before and after the diaphragm. The stereo camera according to claim 3, wherein the aspheric shape is a shape for giving a spherical aberration amount to an ideal lens having no aberration. The stereo camera according to claim 1, further comprising: a restoration unit that restores the MTF of the reference image and the MTF of the comparison image by image processing.   The optical system of the stereo camera according to any one of claims 1 to 6.

Images