WO2025126535A1 - Stereo image processing device and stereo image calibration method - Google Patents

Abstract

The objective of the present invention is to provide a stereo image processing device and a stereo image calibration method capable of calibrating a wide-angle camera mounted in a vehicle with a high degree of accuracy without increasing the size of a chart. A calibration chart includes patterns, wherein: the lengths of the patterns in the baseline direction of a plurality of cameras differ at least between a first region and a second region; and a calibration processing unit obtains a calibration parameter on the basis of a result of performing stereo matching in each of the first region and the second region of the calibration chart, assuming that the different patterns are the same point.

Description

STEREO IMAGE PROCESSING APPARATUS AND STEREO IMAGE CALIBRATION METHOD

The present invention relates to a stereo image processing device mounted on, for example, a vehicle (vehicle-mounted) and a stereo image calibration method.

Stereo cameras are known as devices for recognizing objects three-dimensionally. Stereo cameras use the differences in how images are captured by multiple cameras placed in different positions to detect parallax between the multiple cameras based on trigonometry, and use this parallax to detect the depth and position of an object, making it possible to accurately detect the position of the measurement target.

Such stereo cameras are mounted on automobiles and other vehicles and are used in technology (in-vehicle sensing technology) to detect the position of obstacles, etc. In order to accommodate many use cases, in-vehicle sensing technology is required to detect obstacles, etc. with a wide angle of view, as well as to detect obstacles at greater distances (wider angle of view and longer range).

　In-vehicle stereo cameras are generally installed inside the vehicle to avoid the effects of dirt and other factors, but as the angle of view becomes wider, the effects of the vehicle's windshield cannot be ignored. Conventionally, a calibration process for the stereo camera, known as aiming, has been carried out during vehicle manufacturing and inspection to correct for misalignment of the stereo camera, but further measures are required to widen the angle of view.

Patent Document 1 is known as a method for calibrating a stereo camera that takes into account the influence of a windshield. Patent Document 1 describes the problem as "to provide a stereo camera calibration device and calibration method that can properly calibrate a stereo camera using a target board even in a narrow space," and describes the solution as "a stereo camera calibration device that measures the distance to an object based on parallax, comprising a stereo camera 3 having a pair of cameras 4, 6 spaced apart by a predetermined base line length B, a target board 8 that is arranged parallel to the arrangement direction of the pair of cameras 4, 6 and includes at least two targets 8a, 8b spaced apart by the same distance as the base line length B, and a calculation device that assumes that the target board 8 is located at an infinite distance relative to the stereo camera 3, treats the two targets 8a, 8b included in the images of the target board 8 photographed by each of the pair of cameras 4, 6 as the same target, and calibrates the parallax shift of the pair of cameras 4, 6." Patent Document 2 also describes a similar calibration technology. Patent Document 2 states that the problem is to "variably set the search range when performing stereo matching according to the position on the image," and as a solution, it states that "the system has a comparison image line memory 7, an address generation circuit 10, and a stereo matching circuit 8. The line memory 7 stores image data within a reference pixel area in one captured image and image data on a horizontal line corresponding to the vertical position of the reference pixel area in the other captured image. The address generation circuit 10 sets the search range when performing stereo matching, and instructs the line memory 7 to read image data within the set search range and image data within the reference image area. In addition, the stereo matching circuit 8 identifies the correlation destination of the reference pixel area by stereo matching based on the image data within the search range read from the line memory 7 and the image data of the reference pixel area. Here, the address generation circuit 10 calibrates the position of the search range for the reference pixel area based on the degree of deviation of the infinitely distant corresponding point based on the horizontal position of the reference pixel area."

JP 2017-62150 A JP 2001-92968 A

Patent Documents 1 and 2 claim that by placing a chart near a stereo camera, matching the distance between the two cameras (baseline length) and the period of the chart pattern with the baseline length, and virtually treating the chart as being at infinity, it is possible to calibrate the effects of shifts in the inclination of light rays due to refraction by the windshield. However, these technologies do not take into account the effects of shifts in the position of light rays due to refraction by the windshield, and there is an issue that complete calibration cannot be achieved if a chart is actually placed near a stereo camera.

The present invention has been made in consideration of the above problems, and provides a stereo image processing device and stereo image calibration method that can calibrate a wide-angle stereo camera with high accuracy by placing a chart near the stereo camera, without increasing the size of the chart.

One aspect of the stereo image processing device according to the present invention includes a stereo matching unit that performs stereo matching of multiple images captured by multiple cameras that capture an image of a subject through a light-transmitting member to determine parallax, and a calibration processing unit that determines calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member based on the results of stereo matching of multiple images obtained by the stereo matching unit capturing an image of a calibration chart with the multiple cameras, the calibration chart including a pattern, the pattern being a repeating pattern that is larger than the baseline length of the multiple cameras, and the calibration processing unit determines the calibration parameters based on the results of stereo matching assuming that different patterns are the same point.

The stereo image processing device and stereo image calibration method of the present invention can provide a stereo image processing device and stereo image calibration method that can calibrate a wide-angle camera with high accuracy without increasing the size of the chart.

　Problems, configurations and advantages other than those mentioned above will become clear from the description of the embodiments below.

FIG. 1 is a block diagram illustrating a configuration of a stereo camera image processing device according to a first embodiment. FIG. 1 is a diagram for explaining a problem of a conventional method according to a first embodiment; FIG. 11 is another diagram for explaining the problem of the conventional method according to the first embodiment. 3 shows the shape of a chart area according to the first embodiment. 4 shows patterns in each area of the chart according to the first embodiment. 2 shows an optical path according to the first embodiment. 1 shows a parallax displacement in a conventional method according to the first embodiment. 2 shows the distance measurement in the conventional method according to the first embodiment. 4 shows parallax displacement according to the first embodiment. 3 shows the distance measurement according to the first embodiment. 4 shows the distance D according to the first embodiment. 4 shows other shapes of the chart area according to the first embodiment. 4 shows other shapes of the chart area according to the first embodiment. 6 shows other patterns for each area of the chart according to the first embodiment. FIG. 11 is a block diagram illustrating a configuration for creating calibration data in a stereo camera image processing device according to a second embodiment.

Below, the present embodiment will be described with reference to the attached drawings. In the attached drawings, functionally identical elements may be indicated by the same numbers. Note that the attached drawings show embodiments in accordance with the principles of the present disclosure, but these are for the purpose of understanding the present disclosure and are in no way to be used to interpret the present disclosure in a restrictive manner. The descriptions in this specification are merely typical examples and do not limit the scope or application examples of the present disclosure in any way.

In this embodiment, the disclosure has been described in sufficient detail for those skilled in the art to implement the disclosure, but it should be understood that other forms are possible, and that changes to the configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the disclosure. Therefore, the following description should not be interpreted as being limited to this.

[Example 1]
A configuration of a stereo camera image processing device 10 (hereinafter, referred to as "image processing device 10" or "stereo image processing device 10") according to a first embodiment will be described with reference to Fig. 1. The image processing device 10 is mounted on a vehicle such as an automobile, and is used to detect the distance from the vehicle to a three-dimensional object (other automobiles, buildings, pedestrians, etc.) around the vehicle. In the following, a case where the image processing device 10 is mounted on a vehicle will be described as an example, but the present invention is not limited to this.

FIG. 1 is a block diagram showing an example of the configuration of an image processing device 10 according to a first embodiment. This image processing device 10 is configured to detect surrounding three-dimensional objects based on images obtained by a right camera 50 and a left camera 60, and to issue an alarm if necessary. The right camera 50 and the left camera 60 form a stereo camera. Note that the number of cameras that form the stereo camera is not limited to two, a left and a right camera.

As an example, the image processing device 10 is configured to include an image processing unit 100, a stereo disparity image generating unit 200, a stereo disparity image calibration unit 99, a road surface cross-sectional shape estimating unit 400, a stereoscopic three-dimensional object detection unit 500, and an alarm control unit 700.

In this image processing device 10, in an area where images can be captured by both the right camera 50 and the left camera 60 (hereinafter referred to as the "stereo vision area"), a stereo parallax image is generated in the stereo parallax image generation unit 200 by utilizing the parallax between the right camera 50 and the left camera 60. Then, the stereo vision three-dimensional object detection unit 500 measures the distance from the vehicle to the three-dimensional object according to the parallax.

The right camera 50 and the left camera 60 each have a lens and an image sensor, not shown. The right camera 50 and the left camera 60 each capture (take) an image of an object using an image sensor via a lens. In this embodiment, the right camera 50 and the left camera 60 are mounted inside the vehicle and capture images of subjects around the vehicle through a light-transmitting member such as the windshield. The image processing device 10 captures an image P1 (first image) from the right camera 50 and an image P2 (second image) from the left camera 60.

The image processing unit 100 is configured to include, as an example, affine processing means 20a, 20b, luminance correction means 21a, 21b, pixel interpolation means 22a, 22b, and luminance information generation means 23a, 23b. The image processing unit 100 applies a predetermined image processing to the images P1 and P2 obtained by the right camera 50 and the left camera 60, and supplies them to the stereo parallax image generation unit 200.

The affine processing means 20a applies affine processing to image P1 from the right camera 50. The affine processing is, for example, a linear coordinate transformation process, but may also include non-linear calculations. As a result of performing this affine processing, the affine processing means 20a obtains image P3 (third image). Similarly, the affine processing means 20b applies affine processing to image P2 from the left camera 60 to obtain image P4 (fourth image).

The affine processing means 20a and 20b may also perform distortion transformation processing other than affine processing. In this embodiment, fsinθ, which is the projection method of the fisheye lens, is projected into a coordinate system of (ftanθx, ftanθy). Here, f is the focal length of the fisheye lens, θ is the angle of view incident on the fisheye lens, and θx and θy are the horizontal and vertical components of the angle of view incident on the fisheye lens. Furthermore, in this embodiment, the vertical pixel displacement caused by the influence of the windshield is calibrated by the affine processing means 20a and 20b.

The luminance correction means 21a corrects the luminance of each pixel of the image P3. For example, the luminance of each pixel of the image P3 is corrected based on the gain of the right camera 50, the difference in gain of each pixel in the image P3, etc. Similarly, the luminance correction means 21b corrects the luminance of each pixel of the image P4.

The pixel interpolation means 22a performs demosaicing processing on image P3. For example, a raw image is converted into a color image. Similarly, the pixel interpolation means 22b performs demosaicing processing on image P4.

The luminance information generating means 23a generates luminance information for image P3. For example, it converts information representing a color image into luminance information for generating a parallax image. Similarly, the luminance information generating means 23b generates luminance information for image P4.

The stereo disparity image generating unit 200 uses the image of the aforementioned stereo viewing area (common viewing area) from the obtained images P3 and P4 to generate a stereo disparity image of the stereo viewing area.

The stereo parallax image generating unit 200 includes an exposure adjustment unit 210 and a sensitivity correction unit 220, and can execute feedback control to the right camera 50 and the left camera 60 regarding the exposure amount, sensitivity, etc. of the right camera 50 and the left camera 60. The stereo parallax image generating unit 200 further includes a geometric correction unit 230 that performs geometric correction of the left and right images, and a matching unit 240 that performs matching processing of the left and right images. The matching unit 240 performs stereo matching of the left and right images to obtain parallax (stereo parallax image).

The stereo parallax image calibration unit 99 calibrates the parallax shift caused by light-transmitting materials such as the windshield based on the data previously acquired from the stereo parallax calibration data recording unit 98.

The road surface cross-sectional shape estimation unit 400 estimates the cross-sectional shape of the road surface of the road along which the vehicle equipped with the image processing device 10 is scheduled to travel.

The stereoscopic three-dimensional object detection unit 500 detects three-dimensional objects in the stereoscopic field according to the stereoscopic disparity images generated by the stereoscopic disparity image generation unit 200. In addition, stereo matching is applied to the detected three-dimensional objects to detect disparity and identify the type of the three-dimensional object (pedestrian, bicycle, vehicle, building, etc.). By detecting and identifying the type of three-dimensional object, the type to be used for preventive safety is further specified. When a vehicle is detected, the detection result can be used for following control of the preceding vehicle and emergency braking control. When the detected three-dimensional object is a pedestrian or bicycle, emergency braking control and warning control can be executed. Compared to stationary objects, warnings and control are executed for objects that jump out toward the vehicle within a wide field of view. By measuring the distance to these detected objects and estimating the moving speed of the objects tracked over time, the warning control unit 700 can execute more appropriate warnings and control.

In FIG. 1, this embodiment is characterized by creating data in the stereo parallax calibration data recording unit 98.

Next, the effect of this embodiment will be described. First, the problems of the conventional technology will be described in detail. As described above, in Patent Document 1 (hereinafter referred to as "Conventional Technology 1") and Patent Document 2 (hereinafter referred to as "Conventional Technology 2"), a chart is placed near a stereo camera, the interval between the two cameras of the stereo camera (baseline length) and the period of the chart pattern are made to match the baseline length, and the chart is virtually at infinity, so that the influence of the tilt shift of the light beam due to the refraction of the windshield can be calibrated. On the other hand, there are mainly two influences due to the refraction of the windshield. The first is image distortion due to the tilt shift of the light beam. This image distortion occurs in the same way for both a chart placed nearby and a chart placed far away. The second is the position shift of the light beam. The position shift of the light beam has a characteristic that the influence is large for a chart or a measurement object placed nearby, but can be ignored for a chart or a measurement object placed far away.

Of these two factors, Prior Art 1 and Prior Art 2 do not take into account the effect of the misalignment of light rays due to refraction by the windshield. As described above, this misalignment of light rays is detected when measuring the distance to a nearby object, but can be ignored when detecting a distant object. Normally, parallax shift is calibrated to optimize distance measurement. On the other hand, if parallax shift is calibrated using a nearby chart, parallax shift occurs when a distant object is detected due to the misalignment of light rays due to refraction by the windshield. For this reason, Prior Art 1 and Prior Art 2 have the problem that parallax shift cannot be completely calibrated when a chart is placed nearby.

Figure 2 is a diagram explaining the problem. This figure shows a cross section connecting the lens pupil positions of the right camera 50 and the left camera 60. Here, to simplify the explanation, the amount of light ray displacement by the windshield 1 is shown larger than it actually is. In this figure, the light rays at a horizontal angle of view of 0 degrees in the right camera 50 and the left camera 60 are respectively light ray R51 and light ray R61. The distance between the light ray R61 and light ray R51 that are incident on the windshield 1 (light ray R61 and light ray R51 on the opposite side of the stereo camera image processing device 10 with respect to the windshield 1) is set to distance D1. The light rays on the horizontal wide angle side in the right camera 50 and the left camera 60 are set to light ray R52 and light ray R62. Similarly, the distance between the light ray R62 and light ray R52 that are incident on the windshield 1 is set to distance D2. Also, the line extending the light ray R51 between the windshield 1 and the right camera 50 is defined as axis K51, and the line extending the light ray R61 between the windshield 1 and the left camera 60 is defined as axis K61. Similarly, the line extending the light ray R52 between the windshield 1 and the right camera 50 is defined as axis K52, and the line extending the light ray R62 between the windshield 1 and the left camera 60 is defined as axis K62. Here, the distance between the axis K51 and the axis K61 and the distance between the axis K52 and the axis K62 are equal to the baseline length B.

First, consider a horizontal angle of view of 0 degrees. Although ray R51 and axis K51 are almost the same, they do not coincide completely. The same is true for ray R61 and axis K61. For this reason, baseline length B and interval D1 do not coincide. Meanwhile, interval D2 at a horizontal wide angle is also different from baseline length B. What is characteristic here is that interval D1 and interval D2 differ due to the influence of windshield 1. Prior art 1 and prior art 2 estimate the effect of distortion caused by the windshield by utilizing the fact that interval D1, interval D2, and baseline length B coincide. However, in reality, interval D1, interval D2, and baseline length B do not coincide.

FIG. 3 is a diagram for explaining the influence of differences in baseline length B and interval D (interval D1, interval D2). Like FIG. 2, this diagram shows a cross section connecting the lens pupil positions of the right camera 50 and the left camera 60. Here, for ease of explanation, the amount of light displacement caused by the windshield 1 is shown larger than it actually is. The charts described in prior art 1 and prior art 2 use a pattern with the same period as the baseline length. In this diagram, a chart G10 is arranged with a similar pattern period that is the same as the baseline length. In this diagram, the axes of a predetermined angle of view from the pupil positions of the right camera 50 and the left camera 60 are axis K53 and axis K63. The points of contact between the chart G10, axis K53, and axis K63 are positions Q1 and Q2, respectively. Since the inclinations of axis K53 and axis K63 are the same, the interval between position Q1 and position Q2 is the baseline length B. This configuration is the same as that of prior art 1 and prior art 2. The light rays that detect the images at positions Q1 and Q2 by the right camera 50 and the left camera 60 are defined as light rays R53 and R63. The lines extending light rays R53 and R63 between the pupil positions of the right camera 50 and the left camera 60 and the windshield 1 are defined as lines S53 and S63, respectively, and the intersection of lines S53 and S63 is defined as position TP1.

In Prior Art 1 and Prior Art 2, it is said that by making the chart G10 into a pattern with the same period as the baseline length, it is possible to virtually detect infinity. To do this, two axes such as axis K53 and axis K63 must be parallel lines. However, the angle of light ray R53 that passes through the windshield 1 and enters the right camera 50 is different from the angle of light ray R63 that enters the left camera 60. For this reason, a stereo camera adjusted with this chart will generate large errors. In this diagram, for example, what should be infinity will be detected as if the chart was at position TP1, and the fact that it is calibrated at distance L poses a major problem.

This embodiment takes this into consideration. Figure 4 shows the shapes of the areas of chart G20 in this embodiment. Each area has a similar pattern (a square shape in the illustrated example). In this embodiment, each area (pattern) of chart G20 is characterized by having a different width in the horizontal direction (baseline direction). For example, this shows that the horizontal width da1 of area C33 (first area) of chart G20 is different from the horizontal width da2 of area C35 (second area). Width da2 is larger than width da1. In this figure, when the plane including the lens optical axes of the two cameras of the stereo camera and the perpendicular bisectors of the two cameras are on the area C33 on the chart G20 (in other words, when the area including the plane including the lens optical axes of the two cameras of the stereo camera and the position on the chart G20 where the perpendicular bisectors of the two cameras intersect is area C33, or when the plane including the lens optical axes of the two cameras of the stereo camera and the area where the perpendicular bisectors of the two cameras intersect is area 33, and the area whose horizontal position (baseline direction) is different from area C33 is area 35), the widths da1 and da2 are both larger than the base line length B, and the width da1 is closer to the base line length B than the width da2 (i.e., the width da2 is larger than the width da1). Note that the smaller the radius of curvature of the windshield, the more the widths da1 and da2 are separated from the base line length B (see FIG. 11). On the other hand, as the radius of curvature of the windshield increases, widths da1 and da2 approach baseline length B (see Figure 11). Because the radius of curvature of the windshield varies depending on the vehicle, the chart spacing must be adjusted accordingly. Also, if the distance between the stereo camera and the chart is small, widths da1 and da2 move away from baseline length B. On the other hand, as the distance between the stereo camera and the chart increases, widths da1 and da2 approach baseline length B.

FIG. 5 shows the patterns in each area of chart G20 in FIG. 4. For example, the pattern in area C33 on chart G20 is shown. Here too, the horizontal width of the areas is different. For example, the horizontal width db1 of (0,0) is different from the horizontal width db2 of (2,0). Width db2 is larger than width db1. The detected light amount of each area is set to be different, so that incorrect matching does not occur. With this configuration, for example, the amount of deviation between corresponding points in an image in which area C34 of chart G20 is detected by the right camera 50 and an image in which area C33 is detected by the left camera 60 is detected is detected. The amount of deviation can be calculated using a detection method such as block matching or feature point extraction used in stereo cameras.

Figure 6 shows the light path when the positional deviation of the light rays due to the windshield is taken into consideration. First, the light rays of the left camera 60 are the same as those of Figure 3. Meanwhile, the angle of the light ray R54 is changed so that the angle of the light ray R63 incident on the left camera 60 matches the angle of the light ray R54 incident on the right camera 50 (lines S54 and S63 are parallel, and the distance between them is the base line length B). At this time, the horizontal periodic pattern on the chart G20 (in other words, the distance between positions Q3 and Q2 of the images on the chart G20 detected by the right camera 50 and left camera 60 with the light rays R54 and R63) is positioned a distance D apart.

In this embodiment, the parallax shift caused by the inclination shift of the light beam due to the refraction of the windshield can be obtained by using the same process as in FIG. 1. That is, in the stereo parallax image calibration unit 99, the matching unit 240 obtains a calibration parameter for calibrating the parallax shift caused by (the refraction of) the windshield, which is at least a light-transmitting member, based on the result of stereo matching of the left and right images (images P1 and P2) obtained by capturing images of the chart G20 with the left and right cameras (right camera 50 and left camera 60). At this time, the stereo parallax image calibration unit 99 obtains the above-mentioned calibration parameter based on the result of stereo matching assuming that different patterns are the same point. At this time, the result of (the calibration data of) the stereo parallax image calibration unit 99 becomes this parallax shift. The data is processed to correct this parallax shift, recorded in the stereo parallax calibration data recording unit 98, and calibration is performed by the stereo parallax image calibration unit 99. When such a calibration is performed, high accuracy can be achieved for a distant measurement object, but for a nearby measurement object, a parallax shift due to the refraction of the windshield occurs as a cause of positional deviation. However, the tolerance for parallax shift is greater in close proximity than in distant locations, so the impact is small.

7 shows the simulation results for Conventional Technique 1 and Conventional Technique 2. Here, the calculation was performed with the following parameters.
<Windshield>
・Curvature radius -Horizontal: 5.0m
-Vertical: 5.0m
・Windshield tilt angle: 45 degrees ・Windshield thickness: 5.0 mm
Front glass refractive index: 1.52
・Distance between stereo camera and windshield: 50mm
<Stereo camera>
・Baseline length: 200mm
・Focal length: 4.0mm
・Sensor pixel pitch: 0.00375 mm
- Angle of view: -72 degrees to +72 degrees (horizontal), 0 degrees (vertical)
<Chart>
・Camera-chart distance: 0.5m
<Measurement object>
・Distance between stereo camera and measurement object: 50m
・(When estimating the effect of the windshield) Distance between stereo camera and measurement object: 1000m

Figure 7 shows the horizontal angle of view dependency of parallax shift. The vertical axis shows parallax shift, and the horizontal axis shows horizontal angle of view. This figure shows the results for two conditions: with and without a windshield. When a windshield is present, the difference between the parallax shift detected using the proximity chart and the parallax shift when the distance between the stereo camera and the measurement object is set to 1000 m was calculated. In other words, this shows the state in which the effect of the inclination shift of the light beam due to refraction by the windshield has been removed. Therefore, the parallax shift on the vertical axis is the effect of the position shift of the light beam due to the refraction by the windshield described above. In actual calibration, the inclination shift of the light beam due to the windshield and the parallax shift, which is the cause of the position shift, are detected simultaneously, so when the calibration described in Prior Art 1 and Prior Art 2 is performed, this parallax shift remains.

Figure 8 shows the dependency of the measured distance on the horizontal angle of view. The vertical axis shows the measured distance of the stereo camera, and the horizontal axis shows the horizontal angle of view. Here, the measured distance of the stereo camera is calculated using the parallax shift shown in Figure 7. When there is no windshield, accurate distance measurement (50 m) is achieved. On the other hand, the presence of a windshield results in larger distance measurement errors. Although distance measurement errors occur even in areas with small horizontal angles of view (for example, horizontal angle of view 0 degrees), the distance measurement errors are larger on the wide-angle side.

Figure 9 shows the horizontal angle of view dependency of parallax shift when this embodiment is applied. The vertical axis shows parallax shift, and the horizontal axis shows horizontal angle of view. As with Figure 7, this is a state where the effects of distortion due to the windshield have been removed. This shows that there is almost no parallax shift when this embodiment is applied. Therefore, it can be seen that this embodiment can calculate the parallax shift caused by distortion due to the windshield with high accuracy.

Figure 10 shows the dependency of the distance measurement on the horizontal angle of view when this embodiment is applied. The vertical axis shows the distance measurement of the stereo camera, and the horizontal axis shows the horizontal angle of view. As shown above, it can be seen that highly accurate distance measurement (50 m) can be achieved by applying this embodiment.

Figure 11 shows the horizontal angle of view dependency of the optimal chart spacing D (Figure 6) (spacing of periodic or repeating patterns). The vertical axis shows spacing D, and the horizontal axis shows horizontal angle of view. In Figures 9 and 10, the parallax shift and the stereo camera distance measurement distance are calculated at this spacing D. As shown in Figure 11, spacing D is set longer than the baseline length (200 mm) in all horizontal angle ranges. This allows for highly accurate results as shown in Figures 9 and 10.

As described above, this embodiment, like prior art 1 and prior art 2, compares different areas (performs stereo matching by assuming that different patterns are the same point) and detects the amount of deviation. The difference between this embodiment and prior art 1 and prior art 2 is that, as shown in FIG. 6, the distance D of the horizontal periodic pattern on chart G20 is different from the baseline length B (more specifically, is greater than the baseline length B). And, this embodiment can improve distance measurement accuracy by making the horizontal lengths of the two cameras different at least in the first and second areas.

Here, in this embodiment, as in FIG. 4, there is no difference in the vertical angle of view, but as in FIG. 12, for example, the amount of change in the horizontal (baseline) width may be changed with respect to the vertical angle of view. For example, the horizontal (baseline) width may be changed between area C33 (first area) of chart G20 and areas C13, C23, C43, and C53 (third areas) that are perpendicular to the baseline direction of the sensor surface of the camera of area C33 (first area). This is because the windshield is tilted, so the optimal interval D in the horizontal (baseline) direction changes with respect to the vertical angle of view. In this way, the distance measurement accuracy of the entire image can be improved. Each area may be connected in stages (step-like) as in FIG. 12, or each area may be connected smoothly as in FIG. 13. It goes without saying that the effect can be obtained by using two intervals D, for example, the horizontal narrow angle part and the horizontal wide angle part.

Furthermore, in this embodiment, a rectangular chart pattern as shown in FIG. 5 is used, but a circular chart pattern as shown in FIG. 14 may also be used.

In addition, in FIG. 1, the influence of the windshield is calibrated using the stereo parallax image calibration unit 99 and the stereo parallax calibration data recording unit 98, but this is not limited to this. For example, calibration can also be performed by sending the calibration data of the stereo parallax calibration data recording unit 98 to the affine processing means 20a or affine processing means 20b.

[Example 2]
A calibration method for the image processing device 10 according to the second embodiment will be described with reference to Fig. 15. The calibration method in the first embodiment is a calculation method similar to the calibration methods in the conventional techniques 1 and 2, but is not limited to this. In the second embodiment, a calibration method will be described in which the influence of the shift in position of light caused by refraction of the windshield when a chart is placed nearby is taken into consideration.

FIG. 15 shows an example of the configuration of the image processing device 10 including the flow of estimating the influence of the windshield. As in the first embodiment, this embodiment is characterized by using the charts shown in FIG. 4, FIG. 12, and FIG. 13. As shown in FIG. 15, this embodiment is characterized by obtaining the vertical deviation (Y deviation) image of two images in the Y deviation image generating unit 201. Then, the ΔY deviation image generating unit 80 calculates the difference (ΔY deviation image) between the Y deviation image and the Y deviation image under reference conditions obtained by calculation or from an actual device. Then, the Δ parallax deviation calculation processing unit 85 generates a Δ parallax deviation image by multiplying the ΔY deviation image by a predetermined coefficient. Finally, the parallax deviation calculation processing unit 90 adds the Δ parallax deviation image and the parallax deviation image under reference conditions to obtain the parallax deviation caused by the refraction of the windshield. Here, the Δ parallax deviation and the ΔY deviation indicate the parallax deviation and Y deviation amount under reference conditions such as the design value or the center value of the product. Here, we use Δparallax shift and ΔY shift to accommodate variations in the radius of curvature of the windshield, the mounting position of the camera relative to the windshield, and thickness variations in the windshield.

Next, the reason why the parallax shift can be calibrated in this embodiment will be explained. Since the windshield has a radius of curvature in the horizontal and vertical directions and a tilt angle of the windshield, deviations due to deviation factors such as variations in the radius of curvature of the windshield, the mounting position of the camera relative to the windshield, and deviations in the thickness of the windshield, not only a change in the horizontal direction (Δ parallax shift) but also a change in the vertical direction (ΔY shift) occurs. Here, there is a correlation between Δ parallax shift and ΔY shift. Therefore, by detecting the ΔY shift, it is possible to obtain the Δ parallax shift. Then, the parallax shift can be obtained from the Δ parallax shift and the parallax shift under the reference conditions. In this embodiment, the parallax shift data obtained in this manner is recorded in the stereo parallax calibration data recording unit 98 shown in FIG. 1, and calibration is performed in the stereo parallax image calibration unit 99.

As described above, by taking into account the positional shift of light rays due to refraction by the windshield when a chart is placed nearby, it is possible to estimate with high accuracy the effect of the tilt shift of light rays due to refraction by the windshield. Here, in this embodiment, the shift in the Y direction is used, but this is not limited to this, and it goes without saying that a similar effect can be obtained if a chart is used to calibrate the effect of the windshield. Note that when the effect of variation factors such as the windshield and the mounting position of the camera is small, the calibration method of embodiment 1 can be used. On the other hand, when the effect of these variation factors is large, it is desirable to use a calibration method like that of this embodiment.

[summary]
As described above, the stereo image processing device 10 of this embodiment includes a stereo matching unit (matching unit 240) that performs stereo matching of a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that capture images of a subject through a light-transmitting member (windshield 1) to determine a parallax, and a calibration processing unit (stereo parallax image calibration unit 99) that determines calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member (windshield 1) based on a result of stereo matching performed by the stereo matching unit (matching unit 240) on a plurality of images obtained by capturing images of a calibration chart with the plurality of cameras, the calibration chart including a pattern, and the pattern is a repeated pattern that is larger than the baseline lengths of the plurality of cameras ( FIG. 11 ), and the calibration processing unit (stereo parallax image calibration unit 99) determines the calibration parameters based on a result of stereo matching performed assuming that different patterns are the same point.

The stereo image processing device 10 of this embodiment also includes a stereo matching unit (matching unit 240) that performs stereo matching of multiple images captured by multiple cameras (right camera 50, left camera 60) that capture an image of a subject through a light-transmitting member (windshield 1) to determine parallax, and a calibration processing unit (stepper 240) that performs stereo matching of multiple images obtained by capturing an image of a calibration chart with the multiple cameras to determine calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member (windshield 1) based on the results of the stereo matching performed by the stereo matching unit (matching unit 240) on the multiple images obtained by capturing an image of a calibration chart with the multiple cameras. and a stereo parallax image calibration unit 99), the calibration chart includes a pattern, and the length of the baseline direction of the multiple cameras is different in at least a first region and a second region of the pattern (in other words, the pattern has a first region and a second region in which the lengths of the baseline direction of the multiple cameras are different) (da2>da1 in FIG. 4, FIG. 11, etc.), and the calibration processing unit (stereo parallax image calibration unit 99) finds the calibration parameters based on the results of stereo matching in the first region and the second region of the calibration chart, assuming that different patterns are the same point.

In addition, in the stereo image processing device 10 of this embodiment, when the first region is an area on a plane including the lens optical axes of the multiple cameras and including the position on the calibration chart where the perpendicular bisectors of the multiple cameras intersect (and the second region is an area where the baseline positions of the multiple cameras differ from the first region), the second region has a longer length in the baseline direction of the multiple cameras than the first region (in other words, the further the baseline positions of the multiple cameras are from the first region, the longer the length in the baseline direction of the multiple cameras than the first region) (da2>da1 in Figure 4, Figure 11, etc.).

In addition, the stereo image processing device 10 of this embodiment has a third area of the calibration chart in a direction perpendicular to the baseline direction of the sensor surface of the camera in the first area of the calibration chart, and the lengths of the baseline direction of the multiple cameras are different between the first area and the third area (Figures 12 and 13).

The stereo image calibration method of this embodiment includes a stereo matching process (matching unit 240) that performs stereo matching of multiple images captured by multiple cameras (right camera 50, left camera 60) that capture an image of a subject through a light-transmitting member (windshield 1) to determine parallax, and a calibration process (stereo parallax image calibration unit 99) that determines calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member (windshield 1) based on the results of stereo matching of multiple images obtained by capturing images of a calibration chart with the multiple cameras in the stereo matching process (matching unit 240). The calibration chart includes a pattern, and the pattern is a repeating pattern that is larger than the baseline length of the multiple cameras (Figure 11). In the calibration process (stereo parallax image calibration unit 99), the calibration parameters are determined based on the results of stereo matching performed assuming that different patterns are the same point.

The stereo image calibration method of this embodiment includes a stereo matching process (matching unit 240) for determining parallax by performing stereo matching on a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that capture an image of a subject through a light-transmitting member (windshield 1), and a calibration process (stereo matching unit 240) for determining calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member (windshield 1) based on the results of the stereo matching of a plurality of images obtained by capturing an image of a calibration chart by the plurality of cameras in the stereo matching process (matching unit 240). and a stereo parallax image calibration unit 99), the calibration chart includes a pattern, and the length of the baseline direction of the multiple cameras is different in at least a first region and a second region of the pattern (in other words, the pattern has a first region and a second region in which the lengths of the baseline direction of the multiple cameras are different) (da2>da1 in FIG. 4, FIG. 11, etc.), and in the calibration process (stereo parallax image calibration unit 99), the calibration parameters are calculated based on the results of stereo matching in the first region and the second region of the calibration chart, where different patterns are considered to be the same point.

In other words, the stereo image processing device 10 and stereo image calibration method of this embodiment change the pattern size of the calibration chart (calibration chart) G20 from the center to the outside, thereby suppressing positional deviations caused by refraction of light-transmitting members such as a windshield.

The stereo image processing device 10 and stereo image calibration method according to this embodiment can provide a stereo image processing device 10 and stereo image calibration method that can calibrate a wide-angle camera with high accuracy without increasing the size of the chart.

　Although the embodiments of the present invention have been described in detail above using the drawings, the specific configuration is not limited to these embodiments, and any design changes that do not deviate from the gist of the present invention are also included in the present invention.

The present invention is not limited to the above-mentioned embodiment, but includes various other variations. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described.

Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, in part or in whole, for example by designing them as integrated circuits. Further, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information on the programs, tables, files, etc. that realize each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

Furthermore, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

1... Windshield 10... Stereo camera image processing device (stereo image processing device)
50: right camera 60: left camera 20a: affine processing means 20b: affine processing means 98: stereo parallax calibration data recording unit 99: stereo parallax image generating unit (calibration processing unit)
200: stereo disparity image generating unit 201: Y-shift image generating unit 400: road surface cross-sectional shape estimating unit 500: stereo vision three-dimensional object detecting unit

Claims (8)

a stereo matching unit that performs stereo matching of a plurality of images captured by a plurality of cameras that capture an image of a subject through a light transmitting member, and obtains a parallax;
a calibration processing unit that determines a calibration parameter for calibrating at least a parallax shift caused by the light transmitting member based on a result of stereo matching performed by the stereo matching unit on a plurality of images obtained by capturing images of a calibration chart with the plurality of cameras,
the calibration chart includes a pattern;
the pattern is a repeating pattern larger than a baseline length of the plurality of cameras;
The stereo image processing device according to claim 1, wherein the calibration processing unit determines the calibration parameters based on a result of stereo matching performed by assuming that different patterns are the same point. a stereo matching unit that performs stereo matching of a plurality of images captured by a plurality of cameras that capture an image of a subject through a light transmitting member, and obtains a parallax;
a calibration processing unit that determines a calibration parameter for calibrating at least a parallax shift caused by the light transmitting member based on a result of stereo matching performed by the stereo matching unit on a plurality of images obtained by capturing images of a calibration chart with the plurality of cameras,
the calibration chart includes a pattern;
The pattern includes a first region and a second region, and the lengths of the plurality of cameras in a baseline direction are different from each other at least in the first region and the second region;
the calibration processing unit determines the calibration parameters based on a result of stereo matching in which different patterns are assumed to be the same point in each of the first area and the second area of the calibration chart. 3. The stereo image processing device according to claim 2,
When the first area is defined as an area including a position on the calibration chart where a plane including the lens optical axes of the plurality of cameras and perpendicular bisectors of the plurality of cameras intersect,
13. A stereo image processing device, wherein the second area has a longer length in a baseline direction of the plurality of cameras than the first area. 3. The stereo image processing device according to claim 2,
a third area of the calibration chart is disposed in a direction perpendicular to a baseline direction of a sensor surface of the camera in the first area of the calibration chart;
13. A stereo image processing device, comprising: a plurality of cameras each having a different length in a baseline direction between the first area and the third area. A stereo matching process for obtaining a parallax by performing stereo matching on a plurality of images captured by a plurality of cameras that capture an image of a subject through a light-transmitting member;
a calibration process for determining a calibration parameter for calibrating at least a parallax shift caused by the light transmitting member, based on a result of performing stereo matching of a plurality of images obtained by capturing images of a calibration chart with the plurality of cameras in the stereo matching process;
the calibration chart includes a pattern;
the pattern is a repeating pattern larger than a baseline length of the plurality of cameras;
A stereo image calibration method, comprising the steps of: determining the calibration parameters based on a result of stereo matching performed by assuming that different patterns are the same point in the calibration process. A stereo matching process for obtaining a parallax by performing stereo matching on a plurality of images captured by a plurality of cameras that capture an image of a subject through a light-transmitting member;
a calibration process for determining a calibration parameter for calibrating at least a parallax shift caused by the light transmitting member, based on a result of performing stereo matching of a plurality of images obtained by capturing images of a calibration chart with the plurality of cameras in the stereo matching process;
the calibration chart includes a pattern;
The pattern includes a first area and a second area, and the lengths of the plurality of cameras in a baseline direction are different from each other at least in the first area and the second area;
a stereo image calibration method comprising: determining, in the calibration process, the calibration parameters based on a result of stereo matching performed in each of the first area and the second area of the calibration chart, assuming that different patterns are the same point. 7. A stereo image calibration method according to claim 6, comprising:
When the first area is defined as an area including a position on the calibration chart where a plane including the lens optical axes of the plurality of cameras and perpendicular bisectors of the plurality of cameras intersect,
A stereo image calibration method, characterized in that the second area has a longer length in the baseline direction of the multiple cameras than the first area. 7. A stereo image calibration method according to claim 6, comprising:
a third area of the calibration chart is disposed in a direction perpendicular to a baseline direction of a sensor surface of the camera in the first area of the calibration chart;
A stereo image calibration method, characterized in that the lengths of the baseline directions of the multiple cameras are different between the first area and the third area.

Images