DE112021005870T5 - STEREO IMAGE PROCESSING DEVICE AND IMAGE CORRECTION DEVICE - Google Patents

Abstract

A stereo camera with a wide field angle is accurately calibrated without increasing a panel size. This stereo image processing apparatus includes: a stereo matching unit that performs stereo matching on multiple images imaged by an imaging unit that images an object using multiple cameras and detects parallax; and a pixel shift correction processing unit that corrects a pixel shift caused when the object is imaged. A field angle of the imaging unit includes a first field angle range and a second field angle range that is wider than the first field angle range in a horizontal direction. The pixel shift correction processing unit performs pixel shift correction processing on the second field angle range based on information of the first field angle range.

Description

technical field

The present invention relates to a stereo image processing device and an image correction device.

background area

A stereo camera is known as a device for three-dimensionally recognizing an object. The stereo camera detects a parallax between plural cameras based on trigonometry using a difference showing images of the plural cameras arranged at different positions, and using this parallax detects a depth and a position of an object and can determine the position of an observation object.

Such a stereo camera is mounted on an automobile and applied to a technique (a vehicle onboard detection technique) for detecting a position of an obstacle or the like. In order to support many applications, the vehicle onboard detection technique is required to detect an obstacle or the like with a wide field angle and to detect a more distant obstacle (field angle widening and long-distance detection).

For example, PTL 1 has been proposed as a technique for meeting a requirement of field angle expansion and long-distance detection. Although an in-vehicle stereo camera is generally mounted in a cabin of an automobile to avoid an influence of dirt or the like, an influence of a windshield mounted on the vehicle cannot be ignored due to field angle expansion. Although conventionally, a calibration work of a stereo camera called alignment is performed at the time of vehicle manufacture or at the time of inspection to correct an assembly deviation of the stereo camera, further measures are required for widening the field angle.

PTL 2 is known as a calibration method of a stereo camera considering an influence of a windshield. PTL 2 discloses a calibration method of a stereo camera that includes a first camera and a second camera and photographs an object through a transparent body. In order to calculate a highly accurate calibration parameter, this calibration method calculates a correction parameter for calibrating an absolute positional deviation, which is the deviation of coordinates of an image of the object caused by the transparent body in an image photographed by the first camera and/or or an image photographed by the second camera. Further, the stereo camera photographs a calibration tool and captures a first photographed image obtained by the first camera and a second photographed image obtained by the second camera. Further, based on the first photographed image and the second photographed image, a correction parameter for calibrating a relative positional deviation indicating a deviation of parallax between the image of the object in the first photographed image and the image of the object in the second photographed image is calculated. The stereo camera is caused to store a correction parameter based on a correction parameter for calibrating the absolute positional deviation and the correction parameter for calibrating the relative positional deviation. According to the above procedure, the stereo camera is calibrated.

PTL 2 describes that it is possible to calculate a highly accurate correction parameter. On the other hand, in the case of a stereo camera with a wide angle of view, as described in PTL 1, an evaluation board (hereinafter referred to as a board) is required for the stereo camera, and therefore it is inevitable to increase a board size, and a vehicle manufacturer needs to consolidate a vehicle production line renovate on a large scale. if e.g. For example, when a panel is placed at a position 3 m away, in a stereo camera having a field angle of 40 degrees in a horizontal direction, the panel is required to be approximately 2 m. On the other hand, when the panel is placed at a position 3 m away from the stereo camera having a horizontal field angle of 120 degrees, the panel is required to be approximately 10 m. Further, when the panel is placed at a position 3 m away from the stereo camera having a horizontal field angle of 150 degrees, the panel is required to be approximately 22 m.

As for a panel size, although it is conceivable to reduce the necessary panel size by making a distance between the stereo camera and the panel narrower, an image on a sensor is blurred, and therefore it is not possible to perform accurate calibration . As described above, the panel size largely depends on the horizontal field angle of the stereo camera, which is a problem for the field angle expansion of the stereo camera.

citation list

patent literature

• PTL 1: JP 2019-32409 A
• PTL 2: JP 2019-132855 A

Summary of the Invention

Technical problem

The present invention has been made in view of the above problem, and provides a stereo image processing apparatus and an image correction device which can accurately calibrate a stereo camera having a wide angle of view without increasing a panel size.

the solution of the problem

This stereo imaging device according to the present invention includes: an imaging unit that images an object using a plurality of cameras; a stereo matching unit that performs stereo matching on a plurality of images imaged by the imaging unit and detects parallax; and a pixel shift correction processing unit that corrects a pixel shift caused when the object is imaged. A field angle of the imaging unit includes a first field angle range and a second field angle range that is wider than the first field angle range in a horizontal direction. The pixel shift correction processing unit performs pixel shift correction processing on the second field angle range based on the pixel shift amount obtained from the first field angle range.

Advantageous Effects of the Invention

According to a stereo image processing apparatus according to the present invention, it is possible to provide a stereo image processing apparatus and an image corrector which can accurately calibrate a stereo camera having a wide angle of view without increasing a panel size.

character list

• 1A 14 is a block diagram for describing a configuration of a stereo camera image processing device 10 according to a first embodiment.
• 1B 14 is an outline diagram for describing the operation of the stereo camera image processing device 10 according to the first embodiment.
• 2 12 is a schematic diagram illustrating a relation between a windshield 1 and a light beam.
• 3 12 illustrates a change of an image on an image sensor of a right camera 50 when the windshield 1 is mounted on a vehicle.
• 4 FIG. 11 illustrates an example of a calculation result of an amount of pixel shift in a vertical direction caused by the windshield 1. FIG.
• 5 1 illustrates a typical influence of a fluctuation of the windshield 1.
• 6 FIG. 13 shows an example of a graph showing a relation between pixel shift amounts in the vertical direction at a field angle A and a field angle B. FIG.
• 7 12 is a schematic diagram illustrating a situation that various manufacturing variations of a windshield occur and a situation that relative positions of a stereo camera and the windshield deviate compared with a norm (a target state).
• 8th Fig. 12 is a flowchart showing a procedure for obtaining a correction parameter of the stereo camera.
• 9 Fig. 12 is a graph for describing the derivation of a correction function.
• 10 14 is a graph illustrating an effect of correction according to the present embodiment.
• 11 illustrates a method for deriving a correction function C100.
• 12 illustrates an example of a calibration panel.
• 13 14 is a flowchart for describing a stereo image processing apparatus according to a second embodiment.
• 14 FIG. 12 is an example of a graph showing a relation between an interval change amount BD and a pixel shift amount in a vertical direction at field angle A. FIG.
• 15 14 is a flowchart for describing a stereo image processing apparatus according to a third embodiment.
• 16 14 is a flowchart for describing a stereo image processing apparatus according to a fourth embodiment.
• 17 Fig. 12 illustrates a calculation result of a pixel shift in the horizontal direction caused by the windshield.
• 18 12 illustrates a pixel shift amount in the horizontal direction at a position of 0 degrees of vertical field angle after transformation into (fsin θx, fsin θy) by an affinity processor 20a and subsequent processing by a left/right monocular image generation unit 300.
• 19 illustrates a method for deriving a correction function C400.
• 20 14 is a graph illustrating an effect of correction according to the fourth embodiment

Description of the embodiments

In the following, the present embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, in some cases functionally the same elements are denoted by the same numbers. It should be noted that the accompanying drawings illustrate embodiments consistent with the principle of the present disclosure and are useful for understanding the present disclosure, but are not used to interpret the present disclosure in a limiting manner. A description in this specification is by way of example only and is not intended to limit the claims or the applications of the present disclosure in any way.

Although the present embodiment is described in sufficient detail to enable those skilled in the art to implement the present disclosure, it is necessary to understand that the present embodiment can be achieved in other embodiments and it is possible to change configurations and structures and diversify Replace elements without departing from the scope and spirit of the technical idea of the present disclosure. Accordingly, the following description should not be construed as limited thereto.

[First embodiment]

A configuration of a stereo camera image processing device 10 (hereinafter referred to as an image processing device 10) according to the first embodiment is described with reference to FIG 1A described. This image processing device 10 is mounted on a vehicle such as an automobile and is used to detect a distance from the vehicle to a three-dimensional object (such as another automobile, a building, a pedestrian or the like) around the vehicle. Although a case where the image processing device 10 is mounted on a vehicle is described below as an example, the present invention is not limited thereto.

1A 14 is a block diagram illustrating a configuration example of the image processing device 10 according to the first embodiment. This image processing device 10 is configured to be able to detect a three-dimensional object in the vicinity based on images obtained by left and right cameras 50 and 60 and issue an alarm if necessary. The left and right cameras 50 and 60 form a stereo camera.

This image processing device.10 contains e.g. B. an image processing unit 100, a stereo parallax image generating unit 200, a left/right monocular image generating unit 300, a road surface cross-sectional shape estimating unit 400, a stereo imaging unit 500 for detecting a three-dimensional object, a hybrid unit 600 for detecting a three-dimensional object, and an alarm control unit 700

As in 1B , in this image processing apparatus 10, the stereo parallax image generation unit 200 generates using a parallax between the left and right cameras 50 and 60 in an area (hereinafter referred to as a “stereo imaging area”) defined by the left and right cameras 50 and 60 can be imaged together, a stereo parallax image. Further, the stereo imaging unit 500 for detecting a three-dimensional object measures a distance from the vehicle to the three-dimensional object according to parallax.

On the other hand, there is an area (hereinafter referred to as a “monocular imaging area”) from which an image can be obtained only by one of the left and right cameras (50 or 60) on the left and right sides of the stereo imaging area. Although the distance measurement accuracy in the monocular imaging field is poor compared to the stereo imaging field, it is through the adjustment Of such a monocular imaging range, it is possible to configure a stereo camera that can detect a three-dimensional object with a wider angle. As will be described later, the distance from the vehicle to the three-dimensional object in the monocular imaging area is measured according to a distance measurement result based on the parallax in the stereo imaging area, and the surroundings of the vehicle are recognized.

Although not illustrated, the right camera 50 and the left camera 60 each include a lens and an image sensor. The left and right cameras 50 and 60 each capture (image) an image of an object through a lens using the image sensor. The image processing device 10 captures an image P1 (first image) from the right camera 50 and captures an image P2 (second image) from the left camera 60.

The image processing unit 100 includes e.g. affinity processors 20a and 20b, luminance correctors 21a and 21b, pixel interpolators 22a and 22b, and luminance information generators 23a and 23b. The image processing unit 100 applies predetermined image processing to the images P1 and P2 obtained by the left and right cameras 50 and 60, and supplies the images P1 and P2 to the stereo parallax image generation unit 200 and the left/right image generation unit 300. right monocular image.

The affinity processor 20a applies affinity processing to the image P<b>1 from the right camera 50 . The affinity processing is z. B. a linear coordinate transformation processing, but may include a non-linear operation. As a result of performing this affinity processing, the affinity processor 20a captures an image P3 (third image). Likewise, the affinity processor 20b applies affinity processing to the image P2 from the left camera 60 and captures an image P4 (fourth image).

The affinity processors 20a and 20b can also perform distortion transformation processing other than the affinity processing. In the present embodiment, fsin θ of a projection method of a fisheye lens is projectively transformed into a coordinate system of (ftan θx, ftan θy). Here, f represents a focal length of the fisheye lens, θ represents a field angle incident on the fisheye lens, and θx and θy represent horizontal and vertical components of the field angle incident on the fisheye lens. Further, in the present embodiment, the affinity processors 20a and 20b correct a pixel shift in a vertical direction caused by an influence of the windshield. That is, the affinity processors 20a and 20b function as a pixel shift correction processing unit that corrects the pixel shift.

The luminance corrector 21a corrects the luminance of each pixel of the image P3. For example, the luminance corrector 21a corrects the luminance of each pixel of the image P3 based on a gain of the right camera 50, a difference in gain of each pixel in the image P3, or the like. Also, the luminance corrector 21b corrects the luminance of each pixel of the image P4.

The pixel interpolator 22a performs interpolation processing on the image P3. The pixel interpolator 22a sets e.g. B. Convert a RAW image to a color image. Also, the pixel interpolator 22b performs interpolation processing on the image P4.

The luminance information generator 23a generates luminance information of the picture P3. The luminance information generator 23a sets e.g. B. converts information indicating a color image into luminance information for generating a parallax image. Also, the luminance information generator 23b generates luminance information of the picture P4.

The stereo parallax image generating unit 200 generates a stereo parallax image of the stereo imaging area (the common field area) described above from the obtained images P3 and P4 using an image of the stereo imaging area described above. The stereo parallax image generation unit 200 includes an exposure adjustment unit 210 and a sensitivity adjustment unit 220 and may be configured to be able to control feedback of exposure amounts, sensitivity, and the like of the left and right cameras 50 and 60 to the cameras 50 and 60 . In addition, the stereo parallax image generation unit 200 further includes a geometry correction unit 230 that performs geometric correction on the left and right images, a matching unit 240 that performs matching processing on the left and right images, a pixel shift amount calculation unit 260 that calculates a pixel shift amount is calculated, and a correction function deriving unit 270 which derives a correction function which will be described later. The pixel shift amount calculation unit 260 and the correction function derivation unit 270 constitute pixel shift correction processing together with the affinity processors 20a and 20b unit. The road surface cross-sectional shape estimating unit 400 estimates a cross-sectional shape of a road surface of a road on which the vehicle on which the image processing apparatus 10 is mounted is scheduled to travel. Details of the road surface cross-sectional shape estimating unit 400 may be the same as those of an apparatus described in FIG WO 2019/116958 A is revealed.

The left/right monocular image generation unit 300 generates an image of an area other than the above-described stereo imaging area from the obtained images P3 and P4 as a monocular imaging image. Here, the left/right monocular imaging image generating unit 300 performs a projective transformation such that lengths of the unit length along the same horizontal line of the left and right monocular imaging images indicate mutually equal distances.

The stereo imaging unit 500 for detecting a three-dimensional object detects a three-dimensional object in the stereo imaging area according to the stereo parallax image generated by the stereo parallax image generating unit 200 . Furthermore, stereo matching is applied to the detected three-dimensional object to detect the parallax and identify a type of the three-dimensional object (a pedestrian, a bicycle, a vehicle, a building, or the like). The three-dimensional object is detected and the type of the three-dimensional object is identified to further specify the type to be used for preventive safety. When a vehicle is detected, a result of this detection can be used for following control of a preceding vehicle, braking control at the time of an emergency, and the like. When the detected three-dimensional object is a pedestrian or a bicycle, it is possible to perform emergency brake control or alarm control. For an object suddenly appearing toward a vehicle, an alarm is issued or control is performed for an object within a wide-angle field of view compared to a stationary object. By measuring the distance to the detected object and estimating a moving speed of the object being tracked chronologically, the alarm control unit 700 can issue an alarm and perform control more appropriately.

The hybrid three-dimensional object detection unit 600 synthesizes the stereo parallax image and the monocular imaging image to generate a hybrid image, and detects the three-dimensional object based on this hybrid image. The hybrid image is synthesized by placing the monocular imaging image on the left and right sides of the stereo parallax image. The hybrid three-dimensional object detection unit 600 detects, in the hybrid image, a three-dimensional object present in the monocular imaging range according to information about the parallax in the stereo parallax image or the like, and detects a distance to that three-dimensional object. Further, the three-dimensional object detection hybrid unit 600 detects a three-dimensional object becoming an obstacle in the monocular imaging image, and further performs the identification of a pedestrian, a bicycle, a vehicle, or the like as specifying a kind by pattern matching in a manner similar to the stereo imaging area through.

As described above, the stereo parallax image generation unit 200 and the left/right monocular image generation unit 300 perform image conversion such that the lengths of the unit length along the same horizontal line on the images of the stereo imaging area and the monocular imaging area are mutually equal distances . Therefore, e.g. B. when the three-dimensional object (measurement object) detected in the monocular imaging area is a person, a foot position of the person in an up/down direction of the monocular imaging image is detected in the monocular imaging area, and the distance to the person is calculated using the position information, obtained by the road surface cross-sectional shape estimating unit 400 and obtained through stereo imaging are estimated. In the case of the vehicle, it is possible to estimate a distance from the vehicle to the person by detecting a ground contact area between the vehicle and a ground surface.

The stereo imaging unit 500 for detecting a three-dimensional object and the hybrid unit 600 for detecting a three-dimensional object detect the three-dimensional object, identify whether the three-dimensional object is a pedestrian, a bicycle or a vehicle from the three-dimensional objects, and thereby further specify the type that should be used for preventive safety. A vehicle detection result is used for following a preceding vehicle, braking control at the time of an emergency, and the like. In the case of a pedestrian or a bicycle, a measure is basically taken to apply emergency braking and an alarm to the pedestrian or the bicycle that suddenly appears and used in particular to control a vehicle. An alarm is given for a subject appearing suddenly as compared to a stationary subject, or control is performed for a subject in a field wide-angle area. By measuring a distance to the detected object and estimating a moving speed of the object being tracked chronologically, the alarm control unit 700 issues an alarm or performs control more appropriately.

As described above, the distance to the object in the monocular imaging range of the stereo camera according to the present embodiment is estimated using the position of the ground contact surface with a person's foot or the ground of the vehicle. Therefore, if a detection position is caused to deviate in the up/down direction of the image due to an influence of a windshield, a distance measurement error occurs.

2 12 is a schematic diagram illustrating a relation between a windshield 1 and a light beam. 2(a) 12 illustrates a horizontal cross section of an optical axis of the image processing apparatus 10 and 2 B) illustrates a vertical cross section.

When a ray of light is incident on the windshield 1, the ray of light is refracted according to Snell's law of refraction. Here, when the inclinations of two surfaces of the windshield 1 through which a light beam propagates are the same, the inclinations of the light beams of the incident light and the emission light are the same. As in 2(a) 1, therefore, an angle of a light ray R00 incident from the spatial vicinity of an end face of the windshield 1 does not change much after the ray i is incident and after the light ray is emitted. On the other hand, the direction of a light beam such as a light beam R01 of light obliquely incident from a left-right direction of the vehicle with respect to the windshield 1 changes greatly before and after the light beam is incident. Therefore, the object appears at a position deviated from an actual position.

As in 2 B) 1, the windshield 1 is mounted inclined with respect to the vertical direction in many cases, and therefore many rays of light are incident obliquely with respect to the surface of the windshield 1. The directions of both a light ray R109 passing near a camera optical axis and a light ray R11 having a lower angle of field change significantly before and after the light rays are incident compared to the light ray R00 and the light ray R01. Thus, when the windshield 1 comes between the object and the camera, the object will appear on a lower side than the object actually is.

3 12 illustrates a change of an image on the image sensor of the right camera 50 when the windshield 1 is mounted on the vehicle. Illustrated here 3 a case where objects (here indicated by squares) at three positions of a position near a center 052 of the optical axis of the lens on the image sensor, a position of a wide field angle along an axis 0521 in the horizontal direction and a position of a wide field angle along an axis 0522 in the vertical direction.

It should be mentioned that 3 illustrates the positions of the images of the objects as pairs of dotted lines and solid lines. The dotted lines indicate the positions of the images of the objects when the windshield 1 is not present, and the solid lines indicate the positions of the images of the objects when the windshield 1 is present. Furthermore, the arrows in 3 indicates a direction of change in the positions of the images of the objects caused when the windshield 1 is used.

As referring to 2 is described, the windshield 1 is generally fixed to a vehicle body while being inclined with respect to the vertical direction, and therefore a deviation of light rays in the vertical direction is large. The light beam is incident on the image sensor through the lens, and a deviation of this light beam causes a change in a position where the light beam is incident on the image sensor (hereinafter referred to as “pixel shift”). This image shift appears as a distortion on the image. A distance to the object is estimated in the monocular imaging range according to the present embodiment using a position of a ground contact area between a foot of a person or the vehicle and the ground. Therefore, such a pixel shift causes a distance measurement error and hinders regular operation of a vehicle onboard sensing device.

• - Objektivparameter
  • • Brennweite: 4 mm
  • • Projektion: fsin θ
• - Glasparameter
  • • Krümmungsradius (vertikal, horizontal): 2 m
  • • Brechungsindex: 1,543
• - Sensorparameter
  • • Pixelschrittweite: 4 µm
• - Positions-/Stellungsparameter
  • • Grundlinienlänge: 160 mm
  • • Abstand zwischen Objektiv und Glas: 40 mm
  • • Neigung des Glases: 25 Grad

A graphic in 4 illustrates an example of a calculation result of an amount a pixel shift in the vertical direction caused by the windshield 1. Here, an axis in the horizontal direction of the graph indicates the angle of field in the horizontal direction of the image, and an axis in the vertical direction indicates an angle of field in the vertical direction of the image. Also, shading in the graph indicates the amount of pixel shift at the corresponding field angles (the horizontal and vertical field angles). A right direction in the horizontal direction of the graph is a plus direction of field angle in the horizontal direction, and an upward direction in the vertical direction is a plus direction of field angle. Further, the pixel shift amount is expressed as the shading of the graphic such that a shift down is a plus shift. 4 FIG. 12 illustrates a result based on the left and right monocular images generated by the left/right monocular image generating unit 300. FIG. The following are calculation parameters used for calculation in 4 be used.

• - Lens parameters
  • • Focal Length: 4mm
  • • Projection: fsin θ
• - Glass parameters
  • • Radius of curvature (vertical, horizontal): 2 m
  • • Refractive Index: 1.543
• - Sensor parameters
  • • Pixel pitch: 4 µm
• - Position/pose parameters
  • • Baseline length: 160 mm
  • • Distance between lens and glass: 40mm
  • • Glass tilt: 25 degrees

Here, although an actual shape of the windshield 1 is a curved surface that has not only a radius of curvature but also a higher-order coefficient, a larger curvature (a second-order coefficient) that changes significantly mainly affects an inclination of a light beam and therefore only curvature is used in the present embodiment. This is because an incident range from the windshield 1 of the stereo camera is local.

Conventionally z. For example, a pixel shift correction method such as the technique of PTL 2 is provided to correct this amount of image shift. However, as described above, it is inevitable to increase a panel size, and a vehicle manufacturer needs to renovate a vehicle production line on a large scale. On the other hand, although it is also conceivable to correct a pixel shift caused by the windshield with a fixed value, an error occurs due to an influence of a manufacturing variation or a variation of the windshield attachment or the like.

5 illustrates a typical influence of a fluctuation. 5(a) illustrates an amount of change (absolute value) of a pixel shift in the vertical direction when the windshield is rotated +1 degree from a state (target state) shown in FIG 4 illustrated is inclined. Further illustrated 5(b) a change amount (absolute value) of the pixel shift in the vertical direction when the center of the windshield is lowered by 3 mm from the target state (when the center of the windshield is changed in such a direction that the curvature increases). Here the size of the glass is 0.8 m in the vertical direction and is 1.2 m in the horizontal direction.

5(a) and (b) show that e.g. B. The higher a horizontal field angle becomes at the same vertical field angle, the larger the pixel shift amount becomes. Also show 5(a) and 5(b) also that the pixel shift amount becomes larger the higher the vertical field angle becomes at the same horizontal field angle. 5(a) and 5(b) show that a pixel shift in the vertical direction occurs at any position in the horizontal or the vertical direction. The pixel shift is affected by a fact that the windshield has a curved surface.

Here z. B. A field of view A, which has a horizontal field of view of 60 degrees and a vertical field of view of 0 degrees, and a field of view B, which has a horizontal field of view of 0 degrees and a vertical field of view of -20 degrees ( 5 ), examined. 6 FIG. 13 shows an example of a graph showing a relation between pixel shift amounts in the vertical direction at the field angle A and the field angle B. FIG. In the 6 Illustrated graphic is previously acquired per vehicle kind by actual measurement, simulation, or the like.

1. (a) einen Fall, bei dem die Windschutzscheibe von einer Normposition nach links oder nach rechts abweicht (±2 mm)
2. (b) einen Fall, bei dem die Windschutzscheibe in der Oben/Unten-Richtung von der Normposition abweicht (±2 mm)
3. (c) einen Fall, bei dem ein Neigungswinkel der Windschutzscheibe von einem Normwinkel abweicht (±1 Grad)
4. (d) einen Fall, bei dem die Glasdicke der Windschutzscheibe von einem Normwert abweicht (±0,1 mm)
5. (e) einen Fall, bei dem eine Mittelposition der Windschutzscheibe von einer Normposition abweicht (Abweichung des Krümmungsradius: ±3 mm)
6. (f) einen Fall, bei dem die Windschutzscheibe eine rotatorische Abweichung in der Links/Rechts-Richtung von einer Normposition bewirkt (±1 Grad)
7. (g) einen Fall, bei dem ein Intervall zwischen der Kamera und der Windschutzscheibe von einem Normwert abweicht (±2 mm)

The vertical axis of the graph in 6 indicates the pixel shift amount in the vertical direction at the field angle A, and the horizontal axis indicates the pixel shift amount in the vertical direction at the field angle B. Several points in the graph 6 indicate a relation between pixel shift amounts SA and SB in the vertical direction at the field angle A and the field angle B in various situations. Various situations refers to a situation that various manufacturing variations of a windshield occur, and a situation that relative positions of a stereo camera and the windshield deviate compared with a norm (a target state). In particular, the various situations, as in 7(a) to (g) is illustrated:

1. (a) a case where the windshield deviates from a standard position to the left or to the right (±2 mm)
2. (b) a case where the windshield deviates from the standard position in the up/down direction (±2mm)
3. (c) a case where a tilt angle of the windshield deviates from a standard angle (±1 degree)
4. (d) a case where the windscreen glass thickness deviates from a standard value (±0.1 mm)
5. (e) a case where a center position of the windshield deviates from a standard position (deviation in radius of curvature: ±3 mm)
6. (f) a case where the windshield causes a rotational deviation in the left-right direction from a standard position (±1 degree)
7. (g) a case where an interval between the camera and the windshield deviates from a standard value (±2mm)

How out 6 As can be seen, the pixel shift amounts SA and SB in the vertical direction at the angle of view A and the angle of view B have a correlation regardless of a manufacturing variation or an attachment error of the windshield or the like. This is because the glass has a curved surface in the horizontal direction and in the vertical direction, and therefore even if fluctuation occurs, pixel shift occurs in the vertical direction vertical to the horizontal direction. Although e.g. For example, when the amount of change with respect to the fluctuation varies depending on the radius of curvature and the inclination of the glass, the correlation is maintained. Therefore, in the present embodiment, e.g. B. the stereo camera using a function (a straight line in 6 ) of the pixel shift amounts SA and SB derived from such a correlation is calibrated as the correction parameter by the stereo parallax image generation unit 200 .

8th Fig. 12 is a flowchart showing a procedure of obtaining a correction parameter of the stereo camera. Further, for the sake of simplicity of description, the correction of the pixel shift amount at 0 degrees of the vertical field angle will be described as an example.

First, the stereo camera is mounted on the vehicle where the windshield is not already mounted (F101). A calibration panel is then imaged by the stereo camera (without the windshield) (F102).

Further, the windshield is mounted on the vehicle (F103), and the stereo camera images the calibration panel through the windshield (F104). Further, the image (F102) of the calibration panel imaged without the windshield and the image (F104) of the calibration panel photographed through the windshield are compared, and the pixel shift amount in the vertical direction at a predetermined field angle such as im Range from -20 degrees to +20 degrees of horizontal angle of field is measured (F105). The calibration board according to the present embodiment is a calibration board of such a size that the stereo camera covers an area of e.g. B. from -20 degrees to +20 degrees of horizontal field angle and from -20 degrees to +20 degrees of vertical field angle instead of the total field angle (e.g. -60 degrees to +60 degrees), which is an imaging target .

Then, the pixel shift amount at field angle A (horizontal field angle -60 degrees and vertical field angle 0 degrees) is calculated from the pixel shift amount SB at field angle B (horizontal field angle 0 degrees and vertical field angle -20 degrees) and the correlation function in 6 estimated (F106). As in 9 1, the correction function deriving unit 270 further derives a correction function C100 using the measurement result (F105) of the pixel shift amount in the range of -20 degrees to +20 degrees of the horizontal field angle and the pixel shift amount in the vertical direction at the field angle A (F107). Further, the affinity processor 20a and the affinity processor 20b perform correction based on parameters of affinity processing when there is no windshield and the correction function C100.

10 14 is a graph illustrating an effect of correction according to the present embodiment. 10 12 illustrates a state before correction and a state after correction under the condition of a rotational deviation. 10 shows that it is possible is to substantially suppress the pixel shift amount using this correction.

As described above, in the present embodiment, the image (F102) of the configuration panel imaged without the windshield and the image (F104) of the calibration panel imaged through the windshield are compared, and the pixel shift amount in the vertical Direction at the given field angle is detected. Further, using a correlation function ( 6 ) obtained in advance is given a correction function and the pixel shift is also corrected in the monocular imaging area. By doing so, it is possible to correct a wide-angle portion (a monocular imaging range) based on a measurement result of the calibration board having the predetermined size for the stereo camera having the wide field angle. By performing correction according to the present embodiment, it is possible to correct the pixel shift amount even in the case of the stereo camera with the wide field angle using a calibration board with a small size that does not contain the entire imaging area.

In the present embodiment, by correcting the two cameras 50 and 60, it is possible to suppress an influence of a distance measurement error caused when light is refracted at the windshield. According to the correction method according to the present embodiment, the pixel shift amount is suppressed to approximately 0.5 pixels. As described above, according to the present embodiment, the distance to the object in the monocular imaging range of the stereo camera is estimated using information about a position of the person's foot around the vehicle and the position of the ground contact area between the vehicle and the ground. The monocular imaging area is an area with a wide angle of view, and in the case of the wide angle of view, the accuracy of a distance measurement on a side closer to a face is not required when measuring a relatively close object, such that an error of approx 0.5 pixels is sufficiently acceptable. Consequently, it is possible to correct the influence of the windshield of the wide-angle camera without using a large calibration board.

It should be noted that although the present embodiment has described the example in which the pixel shift amount is corrected at 0 degree of the vertical field angle for the sake of convenience of description, it goes without saying that it is possible to correct also at a vertical field of view angles other than 0 degrees. 11 12 illustrates a relation between the pixel shift amount SB at the field angle B and the pixel shift amount SA at the field angle A when only the vertical field angle is subject to the condition in 6 is changed (0 degrees, -10 degrees and +10 degrees). How out 11 As can be seen, even if the vertical field angle changes, there is a correlation between the pixel shift amount in the vertical direction at each vertical field angle and the pixel shift amount in the vertical direction at field angle B. It is possible to obtain a correction function of the angle of view other than 0 degrees in the vertical direction and correct the influence of the windshield using these pixel shift amounts. It should be noted that the correlation shown in the graph in 11 illustrated is obtained in advance per vehicle type.

An example of the calibration table is given with reference to 12(a) and (b): Using these patterns, it is possible to accurately detect the pixel shift amount.

Further, although the amount of pixel shift in the vertical direction caused by the influence of the windshield is corrected by affinity processing in the present embodiment, a method that performs correction at a time when the hybrid unit 600 is to detection of a three-dimensional object performs a distance measurement on the monocular imaging area.

The correction procedure outlined in the flow chart in 8th illustrated can be performed at a time of manufacturing a vehicle on which the stereo camera is mounted, at a time of replacing the windshield, at a time of checking the vehicle, and the like. Although a positional change of the stereo camera caused when the windshield is attached is not considered in the present embodiment, the positional change of the vehicle can be detected using a calibration board. For example, it may be possible to change the colors of the calibration charts in 12(a) and (b) to randomize and detect the position of the stereo camera. Furthermore, by incorporating a predetermined pattern into the calibration charts 12(a) and (b) and detecting the position of the stereo camera, the change in position of the stereo camera is detected.

Furthermore, although in the present embodiment, the field angle A has a horizontal field angle of 0 degrees and a vertical image field angle of -60 degrees and the field angle B has a horizontal field angle of -20 degrees and a vertical field angle of 0 degrees, this is just an example and a similar effect can be obtained even if the values of the horizontal field angles and the vertical Field angles of the field angles A and B are values other than the above values. Further, although in the present embodiment only the pixel shift amount of only one point of the field wide-angle area (the monocular imaging area) is obtained, pixel shift amounts of plural points can be obtained using the similar method. In this case, although the amount of calculation increases, it is possible to further improve the correction accuracy.

Further, although in the present embodiment the straight line is used as the function expressing the correlation in 6 indicates, it is possible to obtain a similar effect using a curved line. Further, although in the present embodiment there is a rotational deviation of the axis in the vertical direction/horizontal direction of the panel as an error factor, e.g. For example, a measuring instrument such as an autocollimator measures the rotational error to rotate and adjust an actual panel. Furthermore, the amount of rotation of the actual panel can be corrected by geometric calculation. Further, although in the present embodiment fsin θ of the lens projection method is projectively transformed into a coordinate system of (ftan θx, ftan θy), a format of the image after the projective transformation may be a coordinate system other than (ftan θx, ftan θy) as long as a horizontal one line is an epipolar line. By doing this, by comparing two images on a horizontal line, it is possible to provide an advantage that the amount of calculation is relatively small and parallax calculation can be performed.

[Second embodiment]

A stereo image processing apparatus according to the second embodiment is described with reference to FIG 13 described. A schematic configuration of the stereo image processing apparatus according to this second embodiment can be similar to that of the first embodiment ( 1 ) be. In this regard, in this second embodiment, a procedure of deriving a correction parameter is different from that of the first embodiment.

A schedule in 13 12 illustrates the procedure of obtaining the correction parameter of the stereo image processing apparatus according to the second embodiment. The difference from the first embodiment is that a windshield and a stereo camera are already attached to a vehicle in an unillustrated previous process. As in the first embodiment, in this second embodiment, an image (F102 in 8th ) of a calibration panel imaged without the windshield and an image (F104) of a calibration panel imaged through the windshield are compared, and no pixel shift amount in a vertical direction is detected at a given field angle. Instead, in the second embodiment, an actual size of a calibration pattern detected in advance and a size of an imaged calibration pattern are compared, and the pixel shift amount in the vertical direction at the given field angle is detected.

As described above, a pixel shift increases in proportion to a glass incidence angle. Therefore, the more the windshield is tilted, the larger the pixel shift in the vertical direction becomes. On the other hand, the windshield does not tilt in a horizontal direction, and therefore the pixel shift amount is small, and hence is suitable for use as a reference. In an area near an optical axis of the lens, pixel shift is very small and an image size hardly changes.

In practice, although the stereo camera has an optical axis deviated from a center of curvature of the windshield, the pixel shift in the horizontal direction is very small compared to the vertical direction. For example, an error caused by the windshield viewed from an interval ranging from -10 degrees to +10 degrees in the horizontal direction is approximately 0.01% and is very small compared to the vertical direction (pixel shift in the horizontal direction at 10 degrees in the horizontal direction effected by the windshield: 0.016 pixels, and the number of pixels in the range from 0 to 10 degrees in the horizontal direction: 176 pixels). Therefore, in the present embodiment, an amount of change in the interval on the image between predetermined positions in the vertical direction is obtained, and the amount of image shift is detected according to this amount of change.

In a procedure 13 cameras 50 and 60 image first calibration panels through the windshield (F201). Here z. B. the calibration panel in 12(a) pictured. Regarding the calibration table in 12(a) the positions of the white and black areas are known in advance, and information on a size of this black-and-white pattern is acquired in advance.

When the calibration board imaging is completed in F201, a distance Z1 between the stereo camera and the calibration board is calculated using the pixel sizes of the white-and-black pattern of an image selected from the image captured by the left camera 60. has been subjected to affinity transformation and the information on the size of the white-and-black pattern measured in advance is estimated (F202). Especially when the sample size of the calibration board is in 12(a) , which is imaged at a predetermined distance Z0 is D0, and when the pattern size of the image of the calibration pattern imaged by the right camera 50 is D1, the distance Z1 between the stereo camera and the calibration panel can be obtained as follows (equation 1). $$\text{<figure-callout id="1" label="Z" filenames="DE112021005870T5\_0005.png,DE112021005870T5\_0008.png" state="{{state}}">Z</figure-callout>}1=\text{<figure-callout id="1" label="D" filenames="DE112021005870T5\_0005.png,DE112021005870T5\_0008.png" state="{{state}}">D</figure-callout>}1/\text{<figure-callout id="0" label="D" filenames="DE112021005870T5\_0011.png,DE112021005870T5\_0012.png" state="{{state}}">D</figure-callout>}0\times \text{<figure-callout id="0" label="Z" filenames="DE112021005870T5\_0011.png,DE112021005870T5\_0012.png" state="{{state}}">Z</figure-callout>}0$$

Then, an interval Y1 is recorded on the captured image between a position at a field angle B1 of 0 degrees of horizontal field angle/-20 degrees of vertical field angle and a position at a field angle B2 of 0 degrees of horizontal field angle/+20 degrees of vertical field angle in the image subjected to affinity transformation, and an interval change amount BD caused by an influence of insertion of the windshield is calculated from an interval Y0 between the position of the field angle B1 and the position of the field angle B2 assuming the predetermined distance Z0 calculated (F203). $$\text{BD}=\left|\text{<figure-callout id="1" label="Y" filenames="DE112021005870T5\_0005.png,DE112021005870T5\_0008.png" state="{{state}}">Y</figure-callout>}1\times \mathrm{<figure-callout\; id="0"\; label="Z"\; filenames="DE112021005870T5\_0011.png,DE112021005870T5\_0012.png"\; state="\{\{state\}\}">Z</figure-callout>}0/\text{<figure-callout id="1" label="Z" filenames="DE112021005870T5\_0005.png,DE112021005870T5\_0008.png" state="{{state}}">Z</figure-callout>}1-\text{<figure-callout id="0" label="Y" filenames="DE112021005870T5\_0011.png,DE112021005870T5\_0012.png" state="{{state}}">Y</figure-callout>}0\right|$$

Further, the pixel shift amount in the vertical direction at a field angle A (horizontal field angle -60 degrees and vertical field angle 0 degrees) is estimated using a correction parameter as shown in FIG 14 is illustrated (F204). A graphic in 14 . is determined in advance by actual measurement or simulation in a manner similar to the graph in 6 recorded. In this regard, the horizontal axis differs between the graph in 6 and the graphic in 14 . While the horizontal axis in the graph is in 6 indicates a pixel shift amount SB at a field angle B, the horizontal axis of the graph indicates in 14 the interval change amount BD.

Then, in a manner similar to the first embodiment, a correction function C200 is derived using a measurement result of an image shift amount at a horizontal field angle in the range of -20 degrees to +20 degrees and a shift amount in the vertical direction at field angle A (F205). Further, an affinity processor 20a and an affinity processor 20b perform correction based on parameters of affinity processing when there is no windshield and this correction function C200.

Accordingly, in the second embodiment, similarly to the first embodiment, it is also possible to correct a wide-angle portion (a monocular imaging range) based on a measurement result of a panel having a predetermined size for the stereo camera having the wide field angle. By performing correction according to the present embodiment, even the stereo camera with the wide field angle can correct a pixel shift using a small-sized calibration board that does not contain the entire imaging area. It should be noted that although the field angle B is 10 degrees in the horizontal direction and -20 degrees in the vertical direction and the field angle B2 is 0 degrees in the horizontal direction and +20 degrees in the vertical direction in the present embodiment, the present Embodiment is not limited thereto and even if the conditions of the horizontal and vertical field angles are changed, a similar effect can be obtained.

[Third Embodiment]

A stereo image processing apparatus according to the third embodiment is described with reference to FIG 15 described. A schematic configuration of the stereo image processing apparatus according to the third embodiment can be similar to that of the first embodiment ( 1 ) be. In this regard, this third embodiment differs from the first embodiment in a procedure for deriving a correction parameter. 15 12 illustrates a flow chart for obtaining the correction parameter of a windshield of the stereo imaging device according to the third embodiment. A difference from the first embodiment is that a calibration chart is imaged through the windshield and a shift amount is estimated based on this captured image.

First, a calibration panel is imaged by a camera 60 through the windshield (F301). In this regard, relative positions/postures between the camera 60 and the calibration board are changed to perform imaging a number of times.

Then, according to NPL 1 (Learning OpenCV3: Computer Vision in C++ with the OpenCV Library, Adrian Kaehler, Gary Bradski), a difference between several recorded images of the calibration panel shown and an actual size of the calibration panel at the specified field angle is detected (F302). For example, in the present embodiment, the plural images of the calibration board imaged in the range of -20 degrees to +20 degrees in horizontal field angle and from -20 degrees to +20 degrees in vertical field angle and the size of the calibration board in them Field angles are compared and a difference therebetween is detected. It is possible to measure the pixel shift amount in a range of a given field angle based on this difference.

Then, a shift amount at a field angle A (horizontal field angle -60 degrees and vertical field angle 0 degrees) from a shift amount at a field angle B (horizontal field angle 0 degrees and vertical field angle -20 degrees) and a correlation function in 6 estimated (F303). Further, in a similar manner to the first embodiment, a correction function C100 is derived using a measurement result ranging from -20 degrees to +20 degrees of the horizontal field angle and the shift amount in a vertical direction at the field angle A (F304). Further, an affinity processor 20a and an affinity processor 20b perform correction based on parameters of affinity processing when there is no windshield and the correction function C100. By performing correction according to the present embodiment in this way, even a stereo camera with a wide field angle can correct the windshield using a small-sized panel. Note that, according to a method of NPL 1, the wider a field angle becomes, the larger the number of repetitions of measurement becomes, and a calculation load also becomes larger. However, it is possible to reduce the field angle to be detected by performing correction according to the present embodiment, so that it is possible to achieve correction with a small number of repetitions of measurement. The method according to the present embodiment is z. B. suitable when a large industrial facility such as a dealer maintenance factory cannot be introduced.

[Fourth embodiment]

A stereo image processing apparatus according to the fourth embodiment will be described with reference to FIG 16 described. A schematic configuration of the stereo image processing apparatus according to this fourth embodiment can be similar to that of the first embodiment ( 1 ) be. In this regard, this fourth embodiment differs from the first embodiment in a procedure for deriving a correction parameter.

16 12 illustrates a flow chart for obtaining the correction parameter of a windshield of the stereo imaging device according to the fourth embodiment. While a pixel shift amount in a vertical direction is corrected in the above-described embodiments, a pixel shift in a horizontal direction is corrected in the fourth embodiment. Although the deviation of an image in the horizontal direction in the monocular imaging range does not affect the distance measurement, a horizontal position of the image in which an object is detected is caused to deviate, affecting a decision of a collision at an intersection or the like. In the fourth embodiment, a pixel shift in the horizontal direction of the monocular imaging area is corrected.

17 Fig. 12 illustrates a calculation result of a pixel shift in the horizontal direction caused by the windshield. A calculation condition is the same as that of the first embodiment, and the vertical axis indicates a pixel shift amount in the horizontal direction at a position with a vertical field angle of 0 degrees. Further illustrated 17 the pixel shift amount in the horizontal direction at the position with a vertical field angle of 0 degrees after projective transformation into (ftan θx, ftan θy) by an affinity processor 20a and subsequent processing by a left/right monocular image generation unit 300.

17 shows that the pixel shift amount in the horizontal direction changes non-linearly in the horizontal method with respect to the field angle. The pixel shift amount is very large particularly in an area of a wide field angle. Therefore, even if z. For example, when a pixel shift in the horizontal direction is detected in the range of -20 degrees to +20 degrees of horizontal field angle, it is difficult to estimate a wide-angle side (horizontal field angle -60 to -20 degrees and +20 to +60 degrees). It it is considered that a correction value changes greatly especially when a measurement error or the like causes an error.

18 12 illustrates the pixel shift amount in the horizontal direction at a position with a vertical field angle of 0 degrees after transformation into (fsin θx, fsin θy) by the affinity processor 20a and subsequent processing by the left/right monocular image generation unit 300. In contrast to 17 illustrated 18 also a substantially linear pixel shift amount even at a wide field angle. The shape of the windshield has a curved surface and therefore it is better to perform a transformation into a coordinate system based on a sine function rather than a diverging tangent function. This is because Snell's law of refraction is expressed by a sine component, and therefore it is considered that the influence of the sine component depends substantially linearly on the sine function. In the present embodiment, correction is performed using these characteristics.

16 12 illustrates the flowchart for obtaining the correction parameter of the windshield of the stereo imaging device according to the fourth embodiment. A correction of the pixel shift amount at a vertical field angle of 0 degrees will be described as an example for the sake of convenience of description. In this second embodiment, a correction function is obtained using two affinity processors.

First, an image processing device 10 is mounted on the vehicle (excluding the windshield) (F401). Then, the image processing apparatus 10 images a calibration table (F402), and performs a first projective transformation onto the coordinate system of (fsin θx, fsin θy) by affinity processing (F403). The calibration board according to the present embodiment is a calibration board with such a size that the stereo camera z. B. measure a range of -20 degrees to +20 degrees of horizontal field angle and -20 degrees to +20 degrees of vertical field angle instead of the total field angle (e.g. -60 degrees to +60 degrees), which is an imaging target can.

Further, the windshield is mounted on the vehicle (F404) and the calibration panel is imaged through the windshield (F405). Subsequently, the image of this mapped configuration table is subjected to a first projective transformation into the coordinate system of (fsin θx, fsin θy) by affinity processing (F406). Then, the calibration panel image processed by the left/right monocular image generation unit 300, subsequently imaged without the windshield and subjected to the first projective transformation, and the calibration panel image through the windshield has been photographed and subjected to the first projective transformation is compared and a pixel shift amount in the vertical direction at a predetermined field angle such as in the range of -20 degrees to +20 of the horizontal field angle is measured (F407). As in 19 1, the pixel shift amount in the range of -20 degrees to +20 degrees of the horizontal field angle is linearly approximated to obtain a correction function C400 including a wide field angle (F408). Here the horizontal lines on the image are not equidistant and therefore this image is subjected to a second projective transformation in the present embodiment. Then, the first projective transformation into the coordinate system of (ftan θx, ftan θy) is performed by affinity processing using this correction function C400 to obtain a correction function C401 (F409). Further, the affinity processor 20a and an affinity processor 20b perform correction based on parameters of affinity processing when there is no windshield and this correction function C401.

As described above, in the fourth embodiment, two projective transformations are performed to correct a pixel shift in the horizontal direction caused by the windshield. According to the first projective transformation, the detected image is transformed into the coordinate system of (fsin θx, fsin θy) and the pixel shift is detected. Further, the wide field angle correction function C400 is estimated. Then, the pixel shift is projectively transformed according to the second projective transformation to obtain the correction function C401, for which the affinity processing is performed.

20 14 is a graph illustrating an effect of correction according to the fourth embodiment. 20 12 illustrates a state before correction and a state after correction under a condition of no fluctuation. 20 shows that it is possible to substantially suppress the pixel shift amount in the horizontal direction using the two projective projections.

It should be noted that although in the present embodiment a transformation into the coordinate system of (fsin θx, fsin θy) is performed by the first projective transformation, the present invention is not limited thereto. Even in the case of e.g. B. (fθx, fθy), (2fsin θx/2, 2fsin θy/2) and (2ftan θx/2; 2ftan θy/2), it is possible to obtain similar effects even if the magnitude of the effects are different . Depending on the curvature or the slope of the windshield, linearity is lost when e.g. B. the vertical field angle becomes larger (-20 degrees or -30 degrees), and therefore the effect of a correction according to the present embodiment decreases. In this regard, it is possible to obtain the effect of correction according to the present embodiment even in such a state. Furthermore, the correction function C400 of the first projective transform can be non-linear in order to support this effect. Further, although the present embodiment has not described the jitter as in the first embodiment, even if there is jitter, a similar effect can be obtained by correcting the jitter also with the correction function C401.

Further, although according to the present embodiment, a pixel shift in the horizontal direction is obtained by the same method as that in the first embodiment, a pixel shift in the horizontal direction can be obtained by the same method as those in the second embodiment and the third embodiment. The first to fourth embodiments can be combined to correct pixel shift in the horizontal and vertical directions at the same time.

It should be noted that the fourth embodiment described above has described the example in which a stereo imaging area and a monocular imaging area are set, and a model based on a hybrid image which has been integrated with the two areas, in a similar manner to that in FIG PTL 1 described device is used. An aspect of estimating a pixel shift in the horizontal direction at the wide field angle using two projective transformations is also applicable to a regular stereo camera that has only one common field of view area. Furthermore, a similar correction is also applicable to a monocular camera. if e.g. For example, when a monocular camera is used, it is possible by transforming the image of the monocular camera into the coordinate system of (fsin θx, fsin θy) by the first projective transformation and then transforming a measurement result of a pixel shift amount into the coordinate system of (ftan θx, ftan θy) by the second projective transformation it is possible to obtain an image of the coordinate system of (ftan θx, ftan θy) in which the pixel shift caused by the windshield is suppressed. This processing only needs to be performed at a time of manufacturing a vehicle, a time of replacing a glass, and a time of inspecting the vehicle, such that it does not matter whether the processing takes time. Further, a similar effect can be obtained even when processing of performing correction after the first projective transformation and performing the second projective transformation on the result of the correction is employed, although an amount of calculation increases.

<Modification>

The present invention is not limited to the above embodiments, and includes various modifications other than the above-described embodiments. For example, the above embodiments have been described in detail to describe the present invention for easy understanding, and are not necessarily limited to those including all components described. Furthermore, a portion of a configuration of a specific embodiment can be replaced with a configuration of the other embodiment, and the configuration of the other embodiment can also be added to the configuration of the specific embodiment. Furthermore, the other components can also be added, deleted or replaced as part of the components of each embodiment. The correction of a pixel shift caused by the windshield, in particular, is not limited to the affinity processing unit and can e.g. B. be performed by a hybrid unit 600 for detection of a three-dimensional object or by soft processing. When the soft processing is performed, a pixel shift amount can be obtained from a horizontal position of an object using a correction function, and can be processed as an offset when distance measurement of a monocular imaging area is performed.

Reference List

1

windshield

10

image processing device

50

right camera

60

left camera

20a

affinity processor

20b

affinity processor

20

stereo parallax image generation unit

300

Left/right monocular image forming unit

400

road surface cross-sectional shape estimating unit

500

Stereo imaging unit for detecting a three-dimensional object

600

Hybrid unit for detecting a three-dimensional object

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• JP 2019032409 A [0007]
• JP2019132855A [0007]
• WO 2019/116958 A [0025]

Claims (7)

Stereo imaging device, comprising: an imaging unit that images an object using multiple cameras; a stereo matching unit that performs stereo matching on a plurality of images imaged by the imaging unit and detects parallax; and a pixel shift correction processing unit that corrects a pixel shift caused when the object is imaged, wherein a field angle of the imaging unit includes a first field angle range and a second field angle range that is wider than the first field angle range in a horizontal direction, and the pixel shift correction processing unit performs processing of correcting a pixel shift in the second field angle range based on a pixel shift amount obtained in the first field angle range. Stereo image processing device claim 1 , wherein the pixel shift correction processing unit contains in advance correlation information of a pixel shift amount in a vertical direction in the first field angle range and a pixel shift amount in the second field angle range, measures the pixel shift amount in the vertical direction in the first field angle range and based on the measured pixel shift range in the vertical direction in the first field angle range and the correlation information estimates a pixel shift amount in the vertical direction in the second field angle range. Stereo image processing device claim 2 wherein the pixel shift correction processing unit calculates a correction function of the pixel shift amount based on an estimated value of the pixel shift amount in the vertical direction in the second field angle range and the pixel shift amount in the first field angle range. Stereo image processing device claim 2 wherein the pixel shift correction processing unit measures the pixel shift amount in the first field angle range by comparing a plurality of images obtained by the imaging unit by imaging a calibration table. Stereo image processing device claim 4 wherein the pixel shift correction processing unit measures the pixel shift amount in the first field angle range by comparing a first image obtained by imaging the calibration panel without a windshield and a second image obtained by imaging the calibration panel through the windshield. Stereo image processing device claim 1 , wherein the pixel shift correction processing unit is configured to estimate a distance between the camera of the imaging unit and a calibration board using an image obtained by imaging the calibration board and information about a size of a pattern of the calibration board. Stereo image processing device claim 3 , wherein the pixel shift correction processing unit applies a first projective transformation to images detected in the first field angle range, compares the images after the first projective transformation, measures a pixel shift amount in the first field angle range, a second projective transformation that is different from the first projective transformation , applies to a first correction function obtained based on the measured pixel shift amount, and obtains a second correction function.

Images