CN113450420A - Method and system for jointly calibrating vehicle windshield and camera lens and vehicle - Google Patents
Method and system for jointly calibrating vehicle windshield and camera lens and vehicle Download PDFInfo
- Publication number
- CN113450420A CN113450420A CN202110785302.4A CN202110785302A CN113450420A CN 113450420 A CN113450420 A CN 113450420A CN 202110785302 A CN202110785302 A CN 202110785302A CN 113450420 A CN113450420 A CN 113450420A
- Authority
- CN
- China
- Prior art keywords
- camera lens
- axis
- plane
- windshield
- displacement deviation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000006073 displacement reaction Methods 0.000 claims abstract description 66
- 230000003287 optical effect Effects 0.000 claims abstract description 62
- 238000012937 correction Methods 0.000 claims abstract description 36
- 238000003384 imaging method Methods 0.000 claims abstract description 17
- 239000011521 glass Substances 0.000 claims description 48
- 238000004364 calculation method Methods 0.000 claims description 21
- 238000004422 calculation algorithm Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 16
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000009795 derivation Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 102100034112 Alkyldihydroxyacetonephosphate synthase, peroxisomal Human genes 0.000 description 1
- 101000799143 Homo sapiens Alkyldihydroxyacetonephosphate synthase, peroxisomal Proteins 0.000 description 1
- 238000000848 angular dependent Auger electron spectroscopy Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000003702 image correction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention provides a method and a system for jointly calibrating a vehicle windshield and a camera lens and a vehicle. The method comprises the following steps: s1: jointly modeling an optical system composed of the windshield (G) and the camera lens (C), S2: calculating a displacement deviation of an imaging pixel point of the object on the camera lens (C) in any direction of the real physical world, with and without the windshield (G), wherein the any direction is defined perpendicular to an optical axis of the camera lens (C), S3: and realizing the joint correction of the optical system by using the original internal reference distortion correction of the camera lens (C) and the obtained displacement deviation.
Description
Technical Field
The invention relates to the field of vehicle windshield and camera lens combined calibration, which is used for mapping a three-dimensional real world derived from a two-dimensional image, in particular to a method and a system for vehicle windshield and camera lens combined calibration and a vehicle.
Background
In advanced driver assistance as well as autonomous driving, the camera is the main sensor, completing the projection from the real physical world to the CMOS image sensor.
In the prior art, a distortion model of a camera is mainly described by internal parameters, camera module manufacturers generally calibrate the internal parameters of the modules through specific equipment in a production process, and calibration results are transmitted to software and an algorithm in the form of distortion coefficients to correct distortion. However, the existing distortion correction equipment only performs distortion correction on a camera independently and does not consider the factor of glass.
The reasons for this include that the modeling of the whole optical system integrating the camera and the glass is complex, and a certain tilt angle exists when the whole vehicle is integrated and installed, and previous researches often make spherical assumption on the lens, which can be described by polar coordinates, so that when the tilt angle exists, the characteristic cannot be described by polar coordinates.
There is therefore a need to provide new methods for modeling and calibrating a combined lens and glass optical system.
Patent document No. CN 103481824B provides an apparatus and method for removing reflected light from an image of an imaging apparatus. The device comprises: a filter controller configured to output a control signal for controlling an opening or closing operation of at least one filter to a windshield or an imaging device lens at predetermined intervals, wherein the filter may be disposed on the windshield and a camera lens of the vehicle to pass reflected light; an image obtaining unit configured to obtain an image captured by a camera, the camera being mountable in a direction toward a windshield of a vehicle; a differential image generating unit configured to generate a differential image by subtracting a reflected image photographed when the filter is turned on from a normal image photographed when the filter is turned off; and a difference image correction unit configured to obtain a final image by correcting a region corresponding to the reflection image in the difference image based on the blurred image of the difference image.
Disclosure of Invention
The invention aims to realize modeling and calibration of a combined optical system combining a vehicle-mounted camera lens and a windshield.
Furthermore, the present invention is also directed to solve or alleviate other technical problems of the prior art.
The present invention solves the above problems by providing a method, a system and a vehicle for joint calibration of a vehicle windshield and a camera lens, and in particular, according to an aspect of the present invention, there are provided:
a method for joint calibration of a windscreen and a camera lens of a vehicle, said camera lens being arranged behind said windscreen, wherein said method comprises the steps of:
s1: jointly modeling an optical system consisting of the windshield and the camera lens, wherein the joint modeling comprises
S11: establishing a three-dimensional rectangular coordinate system OXYZ, wherein an origin O is an optical center of the camera lens, a Z axis is an optical axis of the camera lens and points towards the direction of the vehicle head, an X axis and a Y axis are orthogonal to each other and a formed XOY plane is perpendicular to the Z axis,
s12: the windshield is assumed to be a flat plate having a uniform thickness H, a refractive index of constant n, smooth and parallel surfaces, and having an inclination angle with respect to an imaging plane of the camera lens,
s13: assuming that an object to be imaged has a distance L in the real physical world in the direction of the Z-axis with respect to the optical center of the camera lens,
s2: calculating displacement deviation of imaging pixel points of the object on the camera lens in any direction of the real physical world under the conditions of existence and absence of the windshield, wherein any direction is defined to be perpendicular to an optical axis of the camera lens and comprises
S21: the displacement deviation is calculated taking into account the refraction of the light by the windscreen,
s3: and realizing the joint correction of the optical system by using the original internal reference distortion correction of the camera lens and the obtained displacement deviation.
Optionally, according to an embodiment of the present invention, the displacement deviation includes a projection component displacement deviation Δ X in a direction of an X axis of an object plane described by a coordinate system X, Y, Z ═ L and/or a projection component displacement deviation Δ Y in a direction of a Y axis of the object plane, where X and Y are projection component coordinate values of the object on the X axis and the Y axis of the object plane in the real physical world without the windshield, respectively, and the calculation formulas are:
in the XOZ plane, under the condition that the windshield is not arranged, the object forms a conjugate image relative to the optical center of the camera lens, and the object inclination angle of the connecting line of the conjugate image and the object relative to the Z axis is enabled to be thetaxzThe inclined angle between the glass plane of the windshield and the X axis is alphaxzWherein the glass plane is defined as a plane formed by the length and width of the windscreen (G),
in the YOZ plane, under the condition that the windshield is not arranged, the object forms a conjugate image relative to the optical center of the camera lens, and the object inclination angle of the connecting line of the conjugate image and the object relative to the Z axis is enabled to be thetayzThe inclined angle between the glass plane of the windshield and the Y axis is alphayzWherein the glass plane is defined as a plane formed by the length and width of the windshield.
Optionally, in accordance with an embodiment of the invention, the camera lens has a focal length f,
the displacement deviation comprises a projection component displacement deviation Δ X in the direction of the X-axis of the image plane and/or a projection component displacement deviation Δ Y in the direction of the Y-axis of the image plane, which are described by a coordinate system X, Y, Z ═ f, wherein X and Y are projection component coordinate values of the object on the X-axis and Y-axis of the image plane without the windshield, respectively, and the calculation formulas are:
in the XOZ plane, under the condition that the windshield is not arranged, the object forms a conjugate image relative to the optical center of the camera lens, and the object inclination angle of the connecting line of the conjugate image and the object relative to the Z axis is enabled to be thetaxzThe inclined angle between the glass plane of the windshield and the X axis is alphaxzWherein the glass plane is defined as a plane formed by the length and width of the windshield,
in the YOZ plane, under the condition that the windshield is not arranged, the object forms a conjugate image relative to the optical center of the camera lens, and the object inclination angle of the connecting line of the conjugate image and the object relative to the Z axis is enabled to be thetayzThe inclined angle between the glass plane of the windshield and the Y axis is alphayzWherein the glass plane is defined as a plane formed by the length and width of the windscreen (G).
Alternatively, according to an embodiment of the present invention, the distance r 'of the object on the object plane with respect to the optical axis of the camera lens in consideration of the displacement deviation Δ X and the displacement deviation Δ Y is obtained by the following formula'o：
Wherein r isoIs the distance of the object in the real physical world in the object plane relative to the optical axis of the camera lens without the windscreen.
Alternatively, according to an embodiment of the present invention, the distance r 'of the object on the image plane with respect to the optical axis of the camera lens in consideration of the displacement deviation Δ x and the displacement deviation Δ y is obtained by the following formula'i：
Wherein r isiIs the distance of the image of the object in the image plane relative to the optical axis of the camera lens without the windscreen.
Optionally, in step S3, the result of the joint correction is obtained by multiplying the coordinate value of the imaging pixel after the original internal reference distortion correction of the camera lens by the ratio of the distance L to the focal length f of the camera lens, and adding the displacement deviation in step S2.
Optionally, according to an embodiment of the present invention, the method further includes step S4: performing iterative calibration on the joint modeling, wherein the iterative calibration comprises the following steps:
s41: taking a calibration reference image, the refractive index, the thickness and the inclination angle of the calibration glass and the distance L of the calibration reference image relative to the optical center of the camera lens C in the Z-axis direction as input;
s42: performing joint correction of the optical system through steps S2 and S3;
s43: calculating the internal reference of the camera by an internal reference calibration algorithm;
s44: calculating a reprojection error after internal reference calibration;
s45: determining whether the reprojection error is less than a threshold,
if so, S46: outputting the internal parameters of the camera and the refractive index, thickness and inclination angle of the calibration glass;
if not, S47: and adjusting the refractive index, the thickness and the inclination angle of the calibration glass, and re-executing the step S42.
According to another aspect of the present invention, the present invention provides a combined calibration system for a windshield and a camera lens of a vehicle, wherein the combined calibration system is configured to perform any one of the above methods, wherein the combined calibration system has an acquisition module, a calculation module and an execution module, which are in communication connection with each other, the acquisition module acquires the thickness H, the refractive index n, the inclination angle, the distance L, the internal parameters of the camera lens and outputs these parameters to the calculation module, the calculation module receives the output of the acquisition module and performs the combined correction of the optical system according to the method and outputs the result of the combined correction to the execution module, and the execution module acquires the result of the combined correction of the calculation module and outputs the result.
Optionally, according to an embodiment of the invention, the joint calibration system is configured in an ECU of the vehicle.
According to yet another aspect of the present invention, a vehicle is provided, wherein the vehicle has any of the above-described joint calibration systems.
The provided method, system and vehicle for jointly calibrating the windshield and the camera lens of the vehicle have the advantages that: through modeling analysis of a combined optical system of an in-vehicle camera and a windshield which are widely used in ADAS and automatic driving, a method for distortion correction according to camera internal parameters is optimized, accuracy of restoring a real world to 2D plane projection from an image coordinate system under a small-hole imaging model is improved, and a better foundation is laid for subsequent related processing such as computer stereo vision and multi-view geometry based on a projection geometry principle.
Drawings
The above and other features of the present invention will become apparent with reference to the accompanying drawings, in which,
FIG. 1 shows a schematic flow diagram of a method according to the invention;
FIG. 2 shows a modeling diagram of a method according to the invention;
FIG. 3 shows a schematic diagram of X-axis displacement deflection according to one method of the present invention;
FIG. 4 shows a schematic diagram of Y-axis displacement bias according to one method of the present invention;
FIG. 5 shows a schematic diagram of X-axis displacement deflection according to another method of the present invention;
FIG. 6 is a schematic diagram illustrating the principle of distance calculation of an object from an origin for X-axis and Y-axis displacement offset according to one method of the present invention;
FIG. 7 shows a flow diagram of iterative calibration according to one method of the present invention; and
FIG. 8 shows a block schematic of a joint calibration system according to the present invention.
Detailed Description
It is easily understood that according to the technical solution of the present invention, a person skilled in the art can propose various alternative structures and implementation ways without changing the spirit of the present invention. Therefore, the following detailed description and the accompanying drawings are merely illustrative of the technical aspects of the present invention, and should not be construed as all of the present invention or as limitations or limitations on the technical aspects of the present invention.
The terms of orientation of up, down, left, right, front, back, top, bottom, and the like referred to or may be referred to in this specification are defined relative to the configuration shown in the drawings, and are relative terms, and thus may be changed correspondingly according to the position and the use state of the device. Therefore, these and other directional terms should not be construed as limiting terms. Furthermore, the terms "first," "second," "third," and the like are used for descriptive and descriptive purposes only and not for purposes of indication or implication as to the relative importance of the respective components.
Reference is made to fig. 1 and 2, which show a flow diagram of a method according to the invention and a modeling diagram of a method according to the invention, respectively.
The method is used for the joint calibration of a windshield G and a camera lens C of a vehicle, wherein the camera lens C is arranged behind the windshield G, and the method comprises the following steps:
s1: jointly modeling an optical system consisting of the windshield G and the camera lens C, wherein the optical system comprises
S11: establishing a three-dimensional rectangular coordinate system OXYZ, wherein an origin O is an optical center of the camera lens C, a Z axis is an optical axis of the camera lens C and points towards the direction of the vehicle head, an X axis and a Y axis are orthogonal to each other and a formed XOY plane is perpendicular to the Z axis,
s12: the windshield G is assumed to be a flat plate having a uniform thickness H, a constant refractive index n, smooth and parallel surfaces, and has an inclination angle with respect to the imaging plane of the camera lens C,
s13: assuming that an object to be imaged has a distance L in the real physical world in the direction of the Z-axis with respect to the optical center of the camera lens C,
s2: calculating the displacement deviation of the imaging pixel points of the object on the camera lens C in any direction of the real physical world under the conditions of the existence of the windshield G and the absence of the windshield G, wherein any direction is defined to be perpendicular to the optical axis of the camera lens C and comprises
S21: the displacement deviation is calculated taking into account the refraction of the light by the windshield G,
s3: and realizing the joint correction of the optical system by utilizing the original internal reference distortion correction of the camera lens C and the obtained displacement deviation.
It should be understood that the camera lens C is arranged behind the windshield G, viewed in the direction of the vehicle head, so that light rays emitted by an object to be observed in the real physical world need to be refracted by the windshield G before being collected by the camera lens C (e.g., its CMOS), and thus distortion correction performed only for the distortion of the camera itself cannot solve the problem of additional distortion caused by refraction of the windshield G. The camera may be an in-vehicle or in-vehicle camera.
The person skilled in the art knows the rules for establishing a rectangular coordinate system in three dimensions, an example of which can be seen in fig. 2. For example, the X-axis and the Y-axis may be the height direction and the width direction of the vehicle, respectively, and the Y-axis is directed outward in fig. 2. Where the point R is a coordinate point of the object in the real physical world, and has coordinate values X, Y, Z (Z ═ L), θxzAnd thetayzAs will be described later. In the present method, since the distance L has been assumed, only the displacement deviation on the plane perpendicular to the L direction needs to be considered. Further, the air refractive index may be set to approximately 1. And the imaging plane of the camera lens C is a plane vertical to the Z axis. In addition, "the coordinate point of the object in the real physical world" does not necessarily mean that the object necessarily needs to be in the objective real physical world to apply the method. But rather, only coordinate data reflecting the position of the object in the real physical world may be obtained. That is, the method can also be used in conjunction with other sensors (e.g., radars) that detect coordinate data of an object from the front-end radar and use it as an input to the method.
In addition, it should be noted that the numbering or naming of the steps herein does not represent the order of the steps, but merely for convenience of description. The order of execution of the steps may be varied, or even performed simultaneously, without significant conflict between steps.
Through the technical scheme, not only the intrinsic lens distortion of the camera due to the intrinsic properties (such as light converging of the convex lens and light diverging of the concave lens) is considered, but also the distortion influence caused by the refraction of the light by the windshield of the vehicle is considered, and the combined distortion correction of the whole optical system is realized. The resulting correction can be output to appropriate software or algorithms for subsequent processing, such as computer stereo vision and multi-view geometry.
The displacement deviation in the plane perpendicular to the L direction is exemplified herein by the displacement deviation in the X-axis and Y-axis directions exemplarily. Reference is made to fig. 3 and 4, which show a schematic representation of X-axis displacement deviation and a schematic representation of Y-axis displacement deviation, respectively, according to one method of the invention. In particular, the displacement deviation comprises a projection component displacement deviation Δ X in the direction of the X-axis of the object plane OP described in the coordinate system X, Y, Z ═ L and/or a projection component displacement deviation Δ Y in the direction of the Y-axis of the object plane OP. X and Y are projection component coordinate values of the object on the X axis and the Y axis of the real physical world on the object plane OP without the windshield G, respectively.
The reflection of the light reflected by the object through the windshield G can be seen. In the XOZ plane, under the condition that the windshield G is not arranged, the object forms a conjugate image x relative to the optical center of the camera lens C, and the object inclination angle of a connecting line Xx of the conjugate image and the object relative to the Z axis is thetaxzThe inclined angle between the glass plane of the windshield G and the X axis is alphaxzWherein the glass plane is defined as a plane formed by the length and width of the windshield G, f is the focal length of the camera lens, βxzIs an angle of incidence of line Xx at the glass plane, from point A into the glass plane, through'xzExits from point B on the opposite side after refraction as an angle of refraction at the glass plane. X' is a virtual image of an imaging pixel point of the object on the camera lens C in the real physical world after displacement deviation, dxzIs the distance between the BX' line and the xX line.
Fig. 4 can also be similarly interpreted. Wherein, in a YOZ plane, the object is relative to the camera lens without the windshield GC forms a conjugate image y with the optical center, and the object inclination angle of the connecting line Yy of the conjugate image and the object relative to the Z axis is thetayzThe inclined angle between the glass plane of the windshield G and the Y axis is alphayzWherein the glass plane is defined as a plane formed by the length and width of the windshield G, f is the focal length of the camera lens, βyzIs an angle of incidence of line Yy at the glass plane, incident into the glass plane from point A, via β'yzExits from point B on the opposite side after refraction as an angle of refraction at the glass plane. Y' is a virtual image of an imaging pixel point of the object on the camera lens C in the real physical world after displacement deviation, dyzIs the distance between the BY' line and the yY line.
The following expressions can be made from fig. 3:
dxz＝AB*Sin(βxz-β′xz) Wherein, in the step (A),ABi.e. the distance of the points a and B,
Sinβxz＝nSinβ′xz，
βxz＝αxz-θxz，
thus, the displacement deviation Δ X is calculated by the formula:
similarly, the following expression can be made from FIG. 4:
dyz＝AB*Sin(βyz-β′yz)，
Sinβyz＝nSinβ′yz，
βyz＝αyz-θyz，
thus, the displacement deviation Δ Y is calculated by the formula:
after knowing the displacement deviation, the distortion compensation caused by the refraction of light by the windshield G can be realized by mapping the corresponding point position on the object plane by the point position on the image plane for application to the real physical world.
It is noted that the displacement deviation on the object plane OP is taken as an example in the present description, however, similarly, the present invention may also be implemented by displacement deviation on the image plane IP, for which reference is made to fig. 5, which shows a schematic diagram of the X-axis displacement deviation according to another method of the present invention.
In the method described above, the derivation is performed by taking an image plane point corresponding to two object plane points as an example, and in the method, the derivation is performed by taking an object plane point corresponding to two image plane points as an example. The principle is similar and therefore will not be described in detail.
In this method, the displacement deviation comprises a projection component displacement deviation Δ X in the direction of the X-axis of the image plane IP and/or a projection component displacement deviation Δ Y in the direction of the Y-axis of the image plane IP, which are described in a coordinate system X, Y, Z ═ -f, where X and Y are projection component coordinate values of the object on the X-axis and Y-axis of the image plane IP without the windshield G, respectively,
the formula is also used here:
Δx＝-(f/L)*ΔX，
therefore, the calculation formula is:
wherein, in the XOZ plane, under the condition of not having the windshield G, the object forms a conjugate image relative to the optical center of the camera lens C, and the object inclination angle of the connecting line of the conjugate image and the object relative to the Z axis is thetaxzThe inclined angle between the glass plane of the windshield G and the X axis is alphaxzWherein the glass plane is defined as a plane formed by the length and width of the windshield G,
and/or
Similarly, the calculation formula of the projection component displacement deviation Δ y is as follows:
wherein, in the YOZ plane, under the condition of not having the windshield G, the object forms a conjugate image relative to the optical center of the camera lens C, and the object inclination angle of the connecting line of the conjugate image and the object relative to the Z axis is thetayzThe inclined angle between the glass plane of the windshield G and the Y axis is alphayzWherein the glass plane is defined by the barrierThe length and width of the windshield G.
Referring to FIG. 6, a schematic diagram illustrating the principle of distance calculation from the origin for imaging an object under X-axis and Y-axis displacement bias according to one method of the present invention is shown. Wherein r isoIs the distance R' of the object from the origin X-0, Y-0, Z-L to the point R (i.e. where the object is located) of the object plane OP without the windshield G.oWhen the displacement deviation Δ X and the displacement deviation Δ Y are considered, the virtual image of the object is R ', X and Y are coordinate values of the object in the real physical world on the X axis and the Y axis, respectively, R is a distance between the object in the real physical world and the original point X of the camera lens object plane, Y is 0, Z is L, phi is an angle between a connecting line of the point R and the original point and the X axis, and phi ' is an angle between a connecting line of the point R ' and the original point and the X axis. This gives:
the following formula is finally obtained:
similarly, the distance r 'of the object from the optical axis of the camera lens C on the image plane IP in consideration of the displacement deviation Δ x and the displacement deviation Δ y'i：
Wherein r'iThe object is under the condition of considering the displacement deviation delta x and the displacement deviation delta yA distance, r, on the image plane IP relative to an optical axis of the camera lens CiIs the distance of the image of the object in the image plane relative to the optical axis of the camera lens C without the windscreen G.
The specific implementation manner of step S3 may be: the coordinate value of the imaging pixel point after the original internal reference distortion correction of the camera lens C is multiplied by the ratio of the distance L to the focal length f of the camera lens C, and the displacement deviation in the step S2 is added to obtain the result of the joint correction. In this example, the respective coordinate values after the distortion correction of the camera lens are x 'and y'. This makes it possible to obtain:
wherein x' ═ x; y ═ y. Thereby completing the distortion correction of the optical system combining the camera lens and the windshield.
In order to further improve the modeling accuracy of the method and the accuracy of the result of the distortion correction, the method may further include step S4: and carrying out iterative calibration on the combined modeling. Referring specifically to fig. 7, a flow diagram of iterative calibration according to one method of the present invention is shown.
S41: taking a calibration reference image, the refractive index, the thickness and the inclination angle of the calibration glass, and the distance L of the calibration reference image relative to the optical center of the camera lens C in the Z-axis direction as input;
s42: performing joint correction of the optical system through steps S2 and S3;
s43: calculating the internal reference of the camera by an internal reference calibration algorithm;
s44: calculating a reprojection error after internal reference calibration;
s45: determining whether the reprojection error is less than a threshold,
if so, S46: outputting the internal parameters of the camera and the refractive index, thickness and inclination angle of the calibration glass;
if not, S47: and adjusting the refractive index, the thickness and the inclination angle of the calibration glass, and re-executing the step S42.
It should be understood that a calibration reference image, i.e. a standardized reference image dedicated to camera lens calibration, having known coordinate values, may typically be used with a calibration plate. For example, a calibration board having a plurality of constituent matrices is used to represent respective positions of objects on the real physical world, and the calibration room formed by the calibration board is three-dimensional, and particularly, a sufficient number of and dense calibration boards are arranged on the front side, the left side, and the right side of the camera lens for comprehensively calibrating the camera lens. The calibration glass is the reference glass used for checking the physical parameters of the glass in the method.
Step S42 is used to convert the calibration reference image into an image without the influence of the windshield G. And calibrating the internal reference of the camera according to the existing calibration algorithm. The algorithm may be, for example, a gnomon scaling method. The reprojection error in S44 is the difference between two projections, where the first projection is the projection of a three-dimensional space point onto an image plane when the camera takes a picture or records a video. Then, the images are used for carrying out triangulation positioning on the characteristic points, a triangle is constructed by using geometric information to determine the position of the three-dimensional space point, and finally, the calculated three-dimensional point coordinates and the calculated camera pose are used for carrying out secondary projection (namely, re-projection). At this time, the difference between the projection of the real three-dimensional space point on the image plane (i.e., the pixel point on the image) and the re-projection (the virtual pixel point obtained by the previous calculation). The smaller this difference, the better. And when the reprojection error is smaller than a threshold value, outputting the internal parameters of the camera, the refractive index, the thickness and the inclination angle of the calibration glass as the input of the method, and realizing a more excellent comprehensive distortion correction effect.
Referring to FIG. 8, a block schematic diagram of a joint calibration system 100 according to the present invention is shown. Since the specific shape and connection of the various components are not the subject of the present invention, all of the components are schematically shown in the form of structural modules for the sake of clarity and conciseness, and those skilled in the art can select the appropriate module shape and connection mode at will based on the teaching of the structural diagram. In addition, the structural diagram is given as an embodiment of the invention, and those skilled in the art can make various modifications without departing from the spirit of the invention after referring to the diagram, and the modifications are also within the scope of the invention.
The joint calibration system 100 is used for a windshield G and a camera lens C of a vehicle, wherein the joint calibration system 100 is configured to execute any one of the above methods, wherein the joint calibration system 100 has an acquisition module 1, a calculation module 2 and an execution module 3 which are in communication connection with each other, the acquisition module 1 acquires the thickness H, the refractive index n, the inclination angle, the distance L, and the internal parameters of the camera lens C and outputs these parameters to the calculation module 2, the calculation module 2 receives the output of the acquisition module 1 and executes joint correction of the optical system according to the method and outputs the result of the joint correction to the execution module 3, and the execution module 3 acquires the result of the joint correction of the calculation module 2 and outputs the result. For a review of the description of the method above, reference is made to the embodiments and features of the system.
It should additionally be mentioned that the joint calibration system 100 may be built into the ECU (also referred to as electronic control unit, running computer) of the vehicle. This makes it possible to seamlessly integrate the method and system without affecting the existing structure of the vehicle.
In addition, the combined calibration system can be arranged on various vehicles, including gasoline vehicles, diesel vehicles, cars, trucks, passenger cars, hybrid vehicles, pure electric vehicles and the like. Accordingly, the subject matter of the present invention is also intended to protect various vehicles equipped with the joint calibration system 100 of the present invention.
In summary, the present invention provides a model for joint modeling of an optical system composed of an optical lens of a camera and glass, where the model describes, through analysis of refractive characteristics of the glass, a relationship of an object in the real world imaged on the lens (such as a CMOS) with and without a windshield, or conversely, a relationship of any point on the lens corresponding to incident light rays at different angles and positions in the real world with and without a windshield, and based on such a corresponding relationship, internal parameters of the original camera can be corrected after the windshield is added.
It should be understood that all of the above preferred embodiments are exemplary and not restrictive, and that various modifications and changes in the specific embodiments described above, which would occur to persons skilled in the art upon consideration of the above teachings, are intended to be within the scope of the invention.
Claims (10)
1. A method for jointly calibrating a windscreen (G) and a camera lens (C) of a vehicle, said camera lens (C) being arranged behind said windscreen (G), characterized in that it comprises the steps of:
s1: jointly modeling an optical system consisting of the windshield (G) and the camera lens (C), including
S11: establishing a three-dimensional rectangular coordinate system OXYZ, wherein an origin O is an optical center of the camera lens (C), a Z axis is an optical axis of the camera lens (C) and points towards a vehicle head direction, an X axis and a Y axis are orthogonal to each other and a formed XOY plane is perpendicular to the Z axis,
s12: assuming the windshield (G) as a flat plate having a uniform thickness H, a constant refractive index n, smooth and parallel surfaces, and having an inclination angle with respect to an imaging plane of the camera lens (C),
s13: assuming that an object to be imaged has a distance L in the real physical world in the direction of the Z-axis with respect to the optical center of the camera lens (C),
s2: calculating the displacement deviation of the imaging pixel point of the object on the camera lens (C) in any direction of the real physical world under the condition of the existence of the windshield (G) and the absence of the windshield (G), wherein any direction is specified to be perpendicular to the optical axis of the camera lens (C), and the displacement deviation comprises
S21: calculating the displacement deviation taking into account the refraction of the windscreen (G) on the light,
s3: and realizing the joint correction of the optical system by using the original internal reference distortion correction of the camera lens (C) and the obtained displacement deviation.
2. The method of claim 1,
the displacement deviation comprises a projection component displacement deviation Δ X in the direction of the X-axis of an Object Plane (OP) described in a coordinate system X, Y, Z ═ L and/or a projection component displacement deviation Δ Y in the direction of the Y-axis of the Object Plane (OP), wherein X and Y are projection component coordinate values of the object in the real physical world on the X-axis and Y-axis of the Object Plane (OP) without the windshield (G), respectively, and the calculation formula is:
wherein, in the XOZ plane, under the condition of not having the windshield (G), the object forms a conjugate image relative to the optical center of the camera lens (C), and the object inclination angle of the connecting line of the conjugate image and the object relative to the Z axis is enabled to be thetaxzThe angle of inclination of the glass plane of the windshield (G) to the X-axis is alphaxzWherein the glass plane is defined as a plane formed by the length and width of the windscreen (G),
in the YOZ plane, the object is relative to the image capture without the windshield (G)The optical center of the head lens (C) forms a conjugate image, and the object inclination angle of the connection line of the conjugate image and the object relative to the Z axis is thetayzThe angle of inclination of the glass plane of the windshield (G) to the Y axis is alphayzWherein the glass plane is defined as a plane formed by the length and width of the windscreen (G).
3. The method of claim 1,
the camera lens (C) has a focal length f,
the displacement deviation comprises a projection component displacement deviation Δ X in the direction of the X-axis of an Image Plane (IP) and/or a projection component displacement deviation Δ Y in the direction of the Y-axis of the Image Plane (IP) described in a coordinate system X, Y, Z ═ f, wherein X and Y are projection component coordinate values of the object in the absence of the windshield (G) on the X-axis and Y-axis of the Image Plane (IP), respectively, and the calculation formula is:
wherein, in the XOZ plane, under the condition of not having the windshield (G), the object forms a conjugate image relative to the optical center of the camera lens (C), and the object inclination angle of the connecting line of the conjugate image and the object relative to the Z axis is enabled to be thetaxzThe angle of inclination of the glass plane of the windshield (G) to the X-axis is alphaxzWherein the glass plane is defined as a plane formed by the length and width of the windscreen (G),
in the YOZ plane, the object forms a conjugate image relative to the optical center of the camera lens (C) under the condition of not having the windshield (G), and the connecting line of the conjugate image and the object is relative to the object of the Z axisBody inclination angle of thetayzThe angle of inclination of the glass plane of the windshield (G) to the Y axis is alphayzWherein the glass plane is defined as a plane formed by the length and width of the windscreen (G).
4. Method according to claim 2, characterized in that the distance r 'of the object on the Object Plane (OP) relative to the optical axis of the camera lens (C) is obtained by the following formula taking into account the displacement deviation Δ X and the displacement deviation Δ Y'o：
Wherein r isoIs the distance of the object in the real physical world in the object plane relative to the optical axis of the camera lens (C) without the windscreen (G).
5. Method according to claim 3, characterized in that the distance r 'of the object on the Image Plane (IP) relative to the optical axis of the camera lens (C) is obtained by the following formula taking into account the displacement deviation Δ x and the displacement deviation Δ y'i：
Wherein r isiIs the distance of the image of the object in the image plane relative to the optical axis of the camera lens (C) without the windscreen (G).
6. The method of claim 1, wherein in step S3, the result of the joint correction is obtained by multiplying the coordinate value of the imaging pixel after the correction of the intrinsic internal distortion of the camera lens (C) by the ratio of the distance L to the focal length f of the camera lens (C), and adding the displacement deviation in step S2.
7. The method according to claim 1, further comprising step S4: performing iterative calibration on the joint modeling, wherein the iterative calibration comprises the following steps:
s41: taking a calibration reference image, the refractive index, the thickness and the inclination angle of the calibration glass and the distance L of the calibration reference image relative to the optical center of the camera lens C in the Z-axis direction as input;
s42: performing joint correction of the optical system through steps S2 and S3;
s43: calculating the internal reference of the camera by an internal reference calibration algorithm;
s44: calculating a reprojection error after internal reference calibration;
s45: determining whether the reprojection error is less than a threshold,
if so, S46: outputting the internal parameters of the camera and the refractive index, thickness and inclination angle of the calibration glass;
if not, S47: and adjusting the refractive index, the thickness and the inclination angle of the calibration glass, and re-executing the step S42.
8. A joint calibration system (100) of a windscreen (G) and a camera lens (C) of a vehicle, characterized in that said joint calibration system (100) is configured for carrying out the method according to any one of claims 1 to 7, wherein said joint calibration system (100) has an acquisition module (1), a calculation module (2) and an execution module (3) communicatively connected to each other, said acquisition module (1) acquiring internal parameters of said thickness H, said refractive index n, said inclination angle, said distance L, said camera lens (C) and outputting these parameters to said calculation module (2), said calculation module (2) receiving the output of said acquisition module (1) and carrying out a joint correction of said optical system according to said method and outputting the result of said joint correction to said execution module (3), the execution module (3) acquires the result of the joint correction of the calculation module (2) and outputs the result.
9. The joint calibration system (100) of claim 8, wherein the joint calibration system (100) is configured within an ECU of the vehicle.
10. A vehicle, characterized in that it has a joint calibration system (100) according to claim 8 or 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110785302.4A CN113450420A (en) | 2021-07-12 | 2021-07-12 | Method and system for jointly calibrating vehicle windshield and camera lens and vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110785302.4A CN113450420A (en) | 2021-07-12 | 2021-07-12 | Method and system for jointly calibrating vehicle windshield and camera lens and vehicle |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113450420A true CN113450420A (en) | 2021-09-28 |
Family
ID=77815906
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110785302.4A Pending CN113450420A (en) | 2021-07-12 | 2021-07-12 | Method and system for jointly calibrating vehicle windshield and camera lens and vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113450420A (en) |
-
2021
- 2021-07-12 CN CN202110785302.4A patent/CN113450420A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5455124B2 (en) | Camera posture parameter estimation device | |
| JP5273356B2 (en) | Compound eye image input device and distance measuring device using the same | |
| WO2019100933A1 (en) | Method, device and system for three-dimensional measurement | |
| JP7038345B2 (en) | Camera parameter set calculation method, camera parameter set calculation program and camera parameter set calculation device | |
| CN112529966B (en) | On-line calibration method of vehicle-mounted looking-around system and vehicle-mounted looking-around system thereof | |
| JP4414661B2 (en) | Stereo adapter and range image input device using the same | |
| CN111559314B (en) | Depth and image information fused 3D enhanced panoramic looking-around system and implementation method | |
| JP7002007B2 (en) | Camera parameter set calculation device, camera parameter set calculation method and program | |
| JP2004132870A (en) | Regulator for stereoscopic camera, and method of regulating stereoscopic camera | |
| JP4857143B2 (en) | Camera posture calculation target device, camera posture calculation method using the same, and image display method | |
| US10810762B2 (en) | Image processing apparatus | |
| JP2001091245A (en) | Stereo-image processing device | |
| JP6602982B2 (en) | In-vehicle camera, in-vehicle camera adjustment method, in-vehicle camera system | |
| JP2023505891A (en) | Methods for measuring environmental topography | |
| US20200213574A1 (en) | Stereo Image Processing Device | |
| US20200349723A1 (en) | Method and system for aligning image data from a vehicle camera | |
| CN115239922A (en) | AR-HUD three-dimensional coordinate reconstruction method based on binocular camera | |
| JPWO2020153317A1 (en) | In-vehicle camera | |
| JP2012019552A (en) | Driving support device | |
| JP6543935B2 (en) | PARALLEL VALUE DERIVING DEVICE, DEVICE CONTROL SYSTEM, MOBILE OBJECT, ROBOT, PARALLEL VALUE DERIVING METHOD, AND PROGRAM | |
| JP2020520588A (en) | Automotive 3D vision system and method for controlling a 3D vision system | |
| JP2013024712A (en) | Method and system for calibrating multiple camera | |
| KR101995344B1 (en) | A dual depth camera module without blind spot | |
| TWM568376U (en) | Multiple camera module adapted to autonomous driving and multiple camera module adapted to an electronic device | |
| CN113450420A (en) | Method and system for jointly calibrating vehicle windshield and camera lens and vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |