CN106952219B - Image generation method for correcting fisheye camera based on external parameters - Google Patents
Image generation method for correcting fisheye camera based on external parameters Download PDFInfo
- Publication number
- CN106952219B CN106952219B CN201710148774.2A CN201710148774A CN106952219B CN 106952219 B CN106952219 B CN 106952219B CN 201710148774 A CN201710148774 A CN 201710148774A CN 106952219 B CN106952219 B CN 106952219B
- Authority
- CN
- China
- Prior art keywords
- fisheye camera
- view
- model
- field
- fisheye
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000013507 mapping Methods 0.000 claims abstract description 25
- 238000003384 imaging method Methods 0.000 claims abstract description 6
- 238000012937 correction Methods 0.000 abstract description 6
- 238000013519 translation Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 230000036544 posture Effects 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 230000002457 bidirectional effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 241000251468 Actinopterygii Species 0.000 description 1
- 101100348047 Mus musculus Ncam2 gene Proteins 0.000 description 1
- 241001591024 Samea Species 0.000 description 1
- NIXOWILDQLNWCW-UHFFFAOYSA-N acrylic acid group Chemical group C(C=C)(=O)O NIXOWILDQLNWCW-UHFFFAOYSA-N 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 238000003466 welding Methods 0.000 description 1
Images
Classifications
-
- G06T3/047—
Abstract
The invention relates to an image generation method based on a fisheye camera, an original plane model for imaging and a field-of-view spherical model establishing step, wherein a corresponding field-of-view spherical model is respectively established for each fisheye camera according to initial internal parameters; an internal parameter correction step, namely respectively determining a reference mapping relation between a field-of-view spherical model and a reference spherical model of each fisheye camera according to initial external parameters; and an external parameter correcting step, namely determining the same second characteristic points in the reference view field spherical model as homonymous groups, calculating deviation distances between the second characteristic points in each homonymous group, correcting the initial external parameters of each fisheye camera until the sum of the deviation distances of all homonymous groups is minimum to obtain corrected external parameters, and a panoramic imaging step, namely projecting the images acquired by each fisheye camera to the reference spherical model according to the corrected internal parameters and the corrected external parameters to obtain panoramic images.
Description
Technical Field
The invention relates to an image processing method, in particular to an image generation method for correcting a fisheye camera based on external parameters.
Background
The panoramic video is a low-cost virtual reality technology realized based on an image splicing technology, and is a research hotspot in the technical fields of virtual reality and computer vision. The panoramic stitching technology has been widely applied to a plurality of fields as a cheap and intuitive implementation mode, such as VR/AR, video shooting, video monitoring, real estate, decorative design, tourist attractions, virtual campuses, street view services and the like, and has wide market demands. In order to shoot a larger and more complete high-definition panoramic image, a panoramic camera is generally composed of more than two cameras to cover an omnidirectional viewing angle, and because the lens of the fisheye camera has the characteristic of an ultra-large viewing angle, the panoramic camera is mostly composed of the fisheye camera.
Generating a panoramic image by splicing the acquired original images through a plurality of fisheye cameras, wherein if the splicing is to be realized, accurate internal parameters and external parameters of each fisheye camera must be obtained; and the relative pose relationship between the fisheye cameras, but the relative pose relationship between the cameras cannot meet the requirement of accurate alignment of theoretical values of the pose relationship between the cameras due to the limitation of industrial manufacturing technology, and the video images acquired by different cameras are spliced to generate obvious dislocation and ghost images by directly utilizing the theoretically designed pose values.
Disclosure of Invention
The invention aims to provide an image generation method for correcting a fisheye camera based on external parameters, which has a simple calibration mode and is closer to the actual scene condition, improves the precision and solves the technical problems; the technical problem solved by the invention can be realized by adopting the following technical scheme:
an image generation method for correcting a fisheye camera based on extrinsic parameters, which provides two fisheye cameras with coincident fields of view, each fisheye camera having initial intrinsic and extrinsic parameters, and an original plane model for imaging,
comprises that
Step S1, establishing a corresponding view field spherical model for each fisheye camera according to initial internal parameters;
step S2, providing a first calibration plate, and correcting the initial internal parameters of each fisheye camera through the first calibration plate to obtain corrected internal parameters;
step S3, establishing a reference spherical model, and respectively determining the reference mapping relation between the field of view spherical model of each fisheye camera and the reference spherical model according to the initial external parameters;
step S4, providing a plurality of second calibration plates in the coincidence field, wherein each second calibration plate comprises a plurality of second feature points, each fisheye camera respectively collects the second feature points and projects the feature points to a reference field spherical model according to a corresponding reference mapping relation, the same second feature points are determined to be homonymy groups in the reference field spherical model, the deviation distance between the second feature points in each homonymy group is calculated, and the initial external parameters of each fisheye camera are corrected until the sum of the deviation distances of all homonymy groups is minimum, so that the corrected external parameters are obtained;
and step S5, projecting the image collected by each fisheye lens to the reference spherical model according to the corrected internal parameters and the corrected external parameters to obtain a panoramic image.
Further, the center point of the reference spherical model coincides with the center point of one of the field-of-view spherical models.
Further, the central point of the reference spherical model is a midpoint between the central points of all the field-of-view spherical models.
Further, the number of the fisheye cameras is two, the overlapped fields of view of the two fisheye cameras are annular fields of view, and the second calibration plates are uniformly arranged along the annular fields of view.
Furthermore, each second calibration plate is provided with an identification pattern, and the fisheye camera collects images of the identification patterns to obtain identification codes for distinguishing the second calibration plates.
Further, the step S4 includes a step S41 of determining that the same second feature point is in a same name group;
the step S41 includes establishing a common view field model, where the common view field model and the view field spherical model of each fisheye camera have a mapping relationship; for any first fisheye camera and any second fisheye camera with coincident fields of view, mapping the second feature points acquired by the first fisheye camera and the second fisheye camera to the common field of view model respectively, and if a second feature point corresponding to the first fisheye camera and a second feature point corresponding to the second fisheye camera are the closest second feature points on the common field of view model, determining the two second feature points in the same group.
Further, the shape of the common view field model is an annular belt-shaped curved surface.
Furthermore, the second calibration plate comprises a plurality of black-white square patterns, and the coincident corner points of every two adjacent square patterns are the second characteristic points.
Has the advantages that: by adopting the technical scheme, the internal parameters and the external parameters of the fisheye camera are corrected again based on the initial internal parameters and the initial external parameters, so that the corrected internal parameters and the corrected external parameters are closer to the actual scene condition, the panoramic splicing precision of the fisheye camera is higher, the calibration method is simpler and more convenient, and the calibration method has more practical significance compared with other calibration modes.
Drawings
FIG. 1 is a schematic illustration of a fish-eye image reprojection;
FIG. 2 is a schematic diagram of a fish-eye camera adjustment method internal parameter correction model;
FIG. 3 is a side view of a spherical panorama projection model;
FIG. 4 is a schematic view of a type A calibration plate of the second calibration plate;
FIG. 5 is a schematic structural view of a type B calibration plate of the second calibration plate;
FIG. 6 is a schematic diagram of a calibration structure for image stitching of a fisheye camera;
FIG. 7 is a schematic view of a fisheye camera projection;
fig. 8 is a schematic diagram of a second feature point homonymy group of the fisheye camera obtained by using the second calibration plate;
FIG. 9 is a schematic diagram of using homonymous image point differences in a common field of view to optimize external referencing;
fig. 10 is a flow chart of fisheye camera panoramic video generation.
Reference numerals: 1. an original planar model; 2. a field-of-view spherical model; 21. a first calibration plate; 3. a reference spherical model; 110. a cross bar; 120. a central support rod; 130. a side support bar; 140. a second calibration plate; 141. a base plate; 142. a chessboard pattern; 143. identifying the graph; 200. a fisheye camera; 210. fisheye lens.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
The invention is further described with reference to the following drawings and specific examples, which are not intended to be limiting.
An image generation method based on a fisheye camera 200 provides at least two fisheye cameras 200 with coincident fields of view, each fisheye camera 200 has initial internal parameters and external parameters, and an original plane model 1 for imaging, adjacent fisheye cameras 200 have initial relative pose relations,
the initial internal parameters of the fisheye camera refer to parameters inherent to the fisheye camera, such as the center coordinates (cx, cy) of the fisheye image, that is, the center coordinates of the original plane model, the radius R (the shape is circular) of the original plane model, the focal length f of the fisheye camera lens, the field angle FOV of the fisheye camera lens, and the specific field spherical model and aberration coefficient, etc., whereas in the method of the present invention, the generation of the field spherical model is implemented in step S1, and the present invention is explained by taking partial internal parameters as an example.
For calculating the central coordinate and the radius of the original plane model, firstly, a fisheye camera is used for collecting an image, a threshold value method is used for matting the image, then, the circle center and the radius of the original plane model are found out by a hugh transformation method or a tangent method, the coordinate of the circle center O is set as (cx, cy), and the radius is set as R; then, coordinate system transformation is carried out on all image points (u, v) on the fisheye image: u-cx and v-cy, so as to transform the coordinate system origin from the upper left corner point of the image to the coordinate system origin with the circle center O as the origin; thus, the original planar model is established. And then, acquiring an image of the calibration plate by a fisheye camera, and calibrating the focal length f of the fisheye camera lens and the field angle FOV of the fisheye camera lens by an internal parameter calibration method such as a Zhang's calibration method, an Ocam method and the like.
The extrinsic parameters include a rotation matrix R and a translation parameter t, and the initial extrinsic parameters can be obtained from the center position of the actual fisheye camera 200 and the corresponding original plane model 1 through the rotation matrix and the translation parameter.
Step S1, establishing a corresponding field-of-view spherical model 2 for each fisheye camera 200 according to the initial internal parameters, the establishing method is common knowledge and is not described herein; since the field of view of the single fisheye camera 200 cannot be 360 °, for example, if the field of view is 210 °, only 210 ° of the field of view is present on the spherical viewing surface, and other spherical areas without the image are transparent.
The spherical reprojection process is as shown in fig. 1, and is the center of a circle of the original plane model 1 and the spherical center of the spherical surface of the field-of-view spherical model 2, and the image point on the fisheye image is projected onto the spherical surface of the field-of-view spherical model 2 according to the Zhang's scaling method, and if the FOV is the field angle of the lens of the fisheye camera 200, the original plane model 1R is located at a small circle corresponding to the FOV on the spherical surface of the field-of-view spherical model 2. Let the spherical radius of the spherical model of the field of view be r, the original image point (u, v) on the fisheye imageTThe three-dimensional image point projected on the standard spherical surface is (x, y, z)T，(u,v)TA distance r from the center O of the spheredThree-dimensional image point (x, y, z)TAnd the optical axis
Is θ, as shown in fig. 1:
taking an equidistant projection model as an example, the internal parameter f of the equidistant projection model obtained in the step 1) can be obtained as follows:
further, it can be based on the original image point (u, v)TFinding a three-dimensional image point (x, y, z)TThe value of (c):
all image points on the view field spherical model can be obtained by solving the formula, so that the projection of the original plane model on the view field spherical model according to the mapping relation is completed.
Step S2, the initial internal parameters of each fisheye camera are corrected by a preset first calibration board to obtain corrected internal parameters, which are as follows:
providing a first calibration plate 21, namely a calibration plate for calibration, and correcting the initial internal parameters of each fisheye camera 200 through the first calibration plate 21 to obtain corrected internal parameters;
the first calibration plate 21 includes a plurality of first feature points, and the step S2 includes a plurality of correction steps, and the correction steps include
Step S21, calculating a theoretical image point position of the first feature point in the original plane model 1 of the fisheye camera 200 according to the relative pose relationship between the first calibration plate 21 and the corresponding fisheye camera 200, and the initial internal and external parameters of the fisheye camera 200, because each image point imaged by the fisheye camera 200 has a corresponding position, i.e., an actual image point position, and the position obtained by the calculation of the original plane model is called an actual image point position;
step S22, calculating the actual image point position of the first feature point in the original plane model 1 of the fisheye camera 200;
in step S23, the deviation between the theoretical pixel position and the actual pixel position of each first feature point is calculated, and the initial internal parameters of the corresponding fisheye camera 200 are corrected according to the deviation to make the theoretical pixel position approach the actual pixel position.
In step S21, the relative positional relationship between the first calibration plate 21 and the corresponding fisheye camera 200 is different.
Specifically, as follows, in general, the initial internal parameters are not accurate and further correction is required. According to the method, the fisheye lens 210 is used for collecting a plurality of first calibration plate 21 images with different distances and different postures, the constraint relation between an actual image point and a theoretical image point obtained by a projection model built by internal parameters of the fisheye camera 200 is constructed, and therefore the initial internal parameters are corrected by using a block method.
As shown in FIG. 2, it is convenient to identify N first feature points (corner points) on the calibration plate, namely the corner points where the white square pattern and the black square pattern coincide, and the coordinate of the ith first feature point in the coordinate system of the first calibration plate is [ x ]i，yi，0]TThe corresponding characteristic point coordinate in the view field spherical model is
The coordinate of the spherical image point when the spherical image point is mapped on the spherical model of the field of view is [ X ]i，Yi，Zi]TThe corresponding two-dimensional image point coordinate on the original plane model is [ u ]i，vi]TThen, for the k frame calibration image, the following mapping relationship exists: 1. mapping three-dimensional coordinates from a calibration plate coordinate system to a spherical coordinate system:
2. mapping of three-dimensional coordinates of the visual spherical coordinate system to coordinates of the visual field spherical model:
wherein
3. Mapping of the field of view spherical model coordinates to the original planar model coordinates: [ X ]i，Yi，Zi]T→[ui，vi]T。
If the mapping from the original plane model image point to the view field spherical model coordinate system image point is f, then:
wherein the sum of the values of cx, cy,
f, FOV is equal to the internal parameter to be corrected, and then the view field spherical model coordinates are converted to the original planeThe mapping of the model coordinates is the inverse of the mapping described above, i.e.:
if mark the plate image coordinate [ x [ ]i，yi，0]TCoordinates [ u ] of image points on the original plane modeli，vi]TIs g, then the formula (x) has:
as can be seen from the equation, for a single frame of the calibration plate image, N constraints and the values of cx, cy,
f, FOV, and Rk、tkAnd 6 external parameter unknowns are added, but the internal parameters are constant, so that only 6 external parameter unknowns are added for each frame of calibration image, a balancing constraint equation can be listed according to the formula, and the internal parameters are further corrected by using a balancing method, wherein the error amount in balancing is the error between the theoretical image point and the actual image point obtained according to the projection model shown in the formula, the balancing amount is cx, cy,
f, FOV, etc. 5 internal parameters.
Step S3, establishing a reference spherical model, and respectively determining the reference mapping relation between the field of view spherical model and the reference spherical model of each fisheye camera according to the initial external parameters; the method comprises the following specific steps:
establishing a reference spherical model 3, wherein the reference spherical model 3 is a spherical surface and is used for generating a spherical panoramic image, and the reference mapping relation of the field-of-view spherical model 2 and the reference spherical model 3 of each fisheye camera 200 is respectively determined according to initial external parameters, and the reference mapping relation reflects the relative position relation of the field-of-view spherical model 2 and the reference spherical model 3, so that each image point can find the corresponding position on the other corresponding spherical surface; in one of the embodiments, the center point of the reference spherical model 3 coincides with the center point of one of the field-of-view spherical models 2. In another embodiment, the center point of the reference spherical model 3 is the midpoint between the center points of all the field-of-view spherical models 2.
A spherical panorama is the closest panorama description to the human eye model. The panoramic stitching idea of the present patent is to project images obtained by different fisheye cameras 200 onto a spherical surface under a common coordinate system, so as to establish a reference spherical model 3, thereby forming a spherical panoramic view, as shown in fig. 3.
Firstly, according to a formula, the original fisheye images obtained by two fisheye cameras are respectively projected onto corresponding view field spherical models according to projection models thereof, namely fisheye image SAProjected onto a spherical surface OA-xAyAzAUpper, fish eye image SBProjected onto a spherical surface OB-xByBzBThe above step (1);
let SAOriginal image point (u) ofA,vA)TThe three-dimensional image point projected on the standard spherical surface is (x)A,yA,zA)T，SBOriginal image point (u) ofB,vB)TThe three-dimensional image point projected on the standard spherical surface is (x)B,yB,zB)TThus, there are:
the unit sphere (namely the spherical surface model of the field of view) O can be obtained through a formulaB-xByBzBThe image point on the image is remapped to a unit sphere (i.e. a view field spherical model) OA-xAyAzAThe above.
For a common fish-eye image set with a relatively close optical center, the fish-eye image set can be directly projected onto a viewing sphere corresponding to one of the fish-eye images (for example, the viewing sphere B corresponding to the fish-eye image set B is re-projected onto the viewing sphere BOn the viewing sphere a corresponding to the camera a). However, if the optical center distance between the two fisheye images is large, the two fisheye images are projected to the coordinate system of one fisheye image uniformly to generate the "decentration" phenomenon, and to avoid this problem, the two cameras can be projected to the unit spherical surface O at the middle position of the two cameras uniformly according to the method shown in fig. 3O-xOyOzOWherein, in the step (A),
the posture is consistent with the fisheye camera A, and then:
and a reference mapping relation is obtained, and images acquired by two or more fisheye cameras can be projected onto the reference spherical model from the original plane model according to the reference mapping relation so as to splice panoramic images.
Step S4, correcting the initial external parameters of each fisheye camera through a preset second calibration plate to obtain corrected external parameters; the method comprises the following specific steps:
providing a plurality of second calibration plates 140 in the overlapped view field, wherein the second calibration plates are also calibration plates for calibration, each second calibration plate 140 comprises a plurality of second feature points, the second feature points are angle points between color blocks of the second calibration plates, each fisheye camera 200 respectively collects the second feature points and projects the feature points to the reference view field spherical model 2 according to the corresponding reference mapping relation, the same second feature points are determined to be homonymy groups in the reference view field spherical model 2, because the two fisheye cameras respectively collect the same image point, when the two image points are projected to the reference view field spherical model 2, two corresponding second feature points will appear, each two feature points are a group of homonymy groups, the subsequent processing is convenient, the deviation distance between the second feature points in each homonymy group is calculated, the initial external parameters of each fisheye camera 200 are corrected until the sum of the deviation distances of all homonymy groups is minimum, obtaining corrected external parameters;
step S4 further includes step S41 of determining that the same second feature point is in a same group;
step S41 includes establishing a common field of view model, which has a mapping relationship with the field of view spherical model 2 of each fisheye camera 200; for any first fisheye camera 200 and any second fisheye camera 200 with coincident fields of view, respectively mapping second feature points acquired by the first fisheye camera 200 and the second fisheye camera 200 onto a common field of view model, and if a second feature point corresponding to the first fisheye camera 200 and a second feature point corresponding to the second fisheye camera 200 are the closest second feature points, determining the two second feature points in the same group. Each second calibration plate 140 is provided with an identification pattern, and the fisheye camera 200 acquires an image of the identification pattern to obtain an identification code for distinguishing the second calibration plate 140. The second calibration board 140 includes several black and white square patterns, and the corner point of each two adjacent square patterns, which is coincident, is a second feature point. The fisheye cameras 200 are arranged in two, the overlapped fields of view of the two fisheye cameras 200 are annular fields of view, and the second calibration plates 140 are uniformly arranged along the annular fields of view. The shape of the common view field model is an annular strip-shaped curved surface.
In order to realize the external parameter correction, an image stitching calibration structure of the fisheye lens 210 needs to be designed correspondingly, and the image stitching calibration structure is specifically as follows: a special annular calibration field is designed for a calibration model consisting of two fisheye cameras 200, and a specific second calibration plate 140 is arranged in a superposed field area of each fisheye camera 200 (the invention designs two forms of second calibration plates 140, namely, A type is a grid plate directly consisting of two-dimensional codes, and B type is a two-dimensional code graph for identification assisted by a common chessboard.
The A-type calibration board is directly formed by two-dimensional codes (Aruco codes), so that four corner points of each Aruco code can be used as homonymous points, the ID number of each Aruco code can be recognized firstly during specific use, then, a digital image processing method (such as morphological operation and threshold binarization) is respectively applied to a square area where each Aruco code is located, the Aruco code area is converted into a black square, the calibration board is changed into a common chessboard grid board, and finally, the corner point coordinates are extracted;
the type B calibration board is formed by combining a chessboard grid pattern and a two-dimensional code pattern (for example, an Aruco two-dimensional code can be pasted on the upper side and the lower side of the chessboard grid pattern), angular points on the chessboard grid pattern are used as points with the same name in the type B calibration board, and the two-dimensional code in the type B calibration board only plays a role of identifying the ID number of the chessboard grid pattern, but is not used as the points with the same name.
The fisheye lens 210 image stitching calibration structure is formed by fixedly connecting a plurality of the above-mentioned a-type or B-type cooperative sign boards, wherein all the cooperative sign boards are located in the common field of view area of the adjacent cameras (note: it is not necessarily required that the whole sign board is located in the common field of view, as long as the same name point is ensured in the common field of view).
The fisheye lens 210 image splicing calibration structure comprises a support chassis, the support chassis comprises a regular hexagon structure formed by combining six cross bars 110, the center extends upwards to form a center support bar 120, the center support bar 120 is used for fixing the assembly of the fisheye camera 200 to realize the calibration function, each cross bar 110 is outwards fixed with a side support bar 130, the side support bars 130 extend upwards perpendicular to the ground and are fixed with a bottom plate 141, the bottom plate 141 is fixed with a calibration pattern, the calibration pattern can be set as an A type or a B type, the bottom plate 141 can be formed by an acrylic plate, the bottom plate 141 and the calibration pattern form a second calibration plate 140, the fixing modes among the side support bars 130, the center support bar 120 and the cross bars 110 can be fixing modes such as welding, threaded connection, integral forming and the like, and are not limited, while in another embodiment, the side support bars 130 can be parallel to the ground and are perpendicular to the cross bars 110, the bottom plate 141 is fixed above the side support bar 130, and the fixing manner of the bottom plate 141 and the side support bar 130 may be a fixing connection manner such as adhesion, screw connection, etc., without limitation.
And in the second step, the detection and extraction of the second feature point are required to be realized. Because the target image obtained by the fisheye camera 200 has large distortion, especially the distortion at the edge part is serious, even if SIFT feature matching with strong adaptability can not directly match the extracted second feature point on the original plane model 1, the invention provides a method for projecting the images of the two fisheye cameras 200 onto the view field spherical model 2, then projecting the images onto the annular strip-shaped curved surface of the view field spherical model 2 by a certain small window to obtain an image with small distortion, and finally performing feature point matching based on the image with small distortion. An image point P on the original plane model I is firstly projected to a point K on a view field spherical model C according to a fisheye projection model, and then the point K on the view field spherical model C is projected to a point Q on an annular circular strip-shaped curved surface S tangent to the view field spherical model.
Then matching the same-name feature points in the adjacent fisheye images, placing a plurality of second calibration plates in the common view field area (common view field model) of the two adjacent panoramic cameras, wherein the second calibration plates are positioned in the common view field model, the positions of the calibration plates are uniformly distributed in all directions to avoid the situation of being in local optimum during posture optimization, and C1、C2The imaging device is a field of view spherical model projected by two fisheye image machines, the annular circular strip-shaped curved surface (common field of view model) S is a common visual area of the two cameras, and six chessboard for chessboard angular point detection are preferably placed in six directions of the annular belt part. And (4) extracting corner points of the chessboard from the images acquired by the fisheye cameras respectively, wherein the same corner points of the chessboard at the same position are the same-name image points.
Finally, in order to ensure the reliability of the same-name image point pairs in the matching process, the RANSCA method and the bidirectional matching verification are preferably selected to eliminate wrong matching points. Wherein, the bidirectional matching refers to the fish-eye image SAPoint P in1In fisheye image SBThe best matching point in (1) is point P2Then P is1And P2The condition being a dotted image is a fisheye image SBPoint P in2In the image SAThe best matching point in (1) is also P1Then the two second feature points are divided into the sameA group of same names. In the RANSAC method, in order to eliminate mismatching points, the relative position and posture relationship (rotation matrix R and translation vector t) between adjacent fisheye cameras is used as a check condition of an inner point, and the homonymous image points on the corresponding view sphere of the adjacent fisheye cameras are respectively set as PiAnd
then P isiAnd
the constraint relationship between them is as follows:
after the homonymy group of the calibration plates in the common view field of the adjacent fisheye cameras is obtained, the relative pose relationship between the fisheye cameras can be optimized according to the aberration of the second characteristic point of the homonymy group in the reference spherical model.
Let I1And I2The original fisheye images P of two fixedly connected fisheye lenses1And P2Are respectively I1And I2A pair of corresponding image points, the image point P1And P2Respectively projecting the internal parameters to the respective view field spherical models S according to a middle method1And S2Obtaining an image point K on the viewing sphere of the spherical model corresponding to the viewing field1And K2Projecting the external parameters (rotation matrix R and translation vector t) on a uniform reference spherical model S to obtain a corresponding image point Q1And Q2Then the image point with the same name (the second feature point of the same name group) Q1And Q2The principle of adjusting and optimizing the external parameters is to minimize the misalignment of all the homonymous image points, and the optimized objective function is:
wherein
Representing an image point Q1iAnd Q2iI is the index number of the homonymous point, and n is the number of the homonymous points. When the objective function reaches a minimum value, the optimal rotation and translation parameters can be obtained, which are marked as R 'and t', and are the corrected external parameters.
Step S5, projecting the image collected by each fisheye lens 210 to the reference spherical model 3 according to the corrected internal parameters and the corrected external parameters to obtain a panoramic image.
Referring to the flowchart, initial internal and external parameters are used to establish a field-of-view spherical model 2 for each fisheye camera 200, modify the internal parameters according to step S2, establish a reference spherical model 3 according to step S3 and modify the external parameters according to the method of step S4 to obtain optimized new external parameters, and two fisheye images are projected from an original plane model 1 to a reference spherical model 3 in a common coordinate system to obtain a spherical panorama.
Although the camera can be accurately aligned through the optimized rotation and translation parameters, so that the splicing alignment of the panoramic image is more accurate, the images of the overlapped part can generate traces at the spliced part inevitably due to exposure and other reasons, and the visual effect of the final panoramic image is influenced.
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.
Claims (7)
1. An image generation method for correcting a fisheye camera based on extrinsic parameters, providing two fisheye cameras with coincident fields of view, each fisheye camera having initial intrinsic and extrinsic parameters, and an original planar model for imaging, comprising:
step S1, respectively establishing a corresponding field-of-view spherical model for each fisheye camera according to the initial internal parameters;
step S2, providing a first calibration plate, and correcting the initial internal parameters of each fisheye camera through the first calibration plate to obtain corrected internal parameters;
step S3, establishing a reference spherical model, and respectively determining the reference mapping relation between the field of view spherical model of each fisheye camera and the reference spherical model according to the initial external parameters;
step S4, providing a plurality of second calibration plates in the coincidence field, wherein each second calibration plate comprises a plurality of second feature points, each fisheye camera respectively collects the second feature points and projects the feature points to a reference spherical model according to a corresponding reference mapping relation, the same second feature points are determined to be homonymy groups in the reference spherical model, the deviation distance between the second feature points in each homonymy group is calculated, and the initial external parameters of each fisheye camera are corrected until the sum of the deviation distances of all homonymy groups is minimum, so that corrected external parameters are obtained;
step S5, projecting the image collected by each fisheye camera to the reference spherical model according to the corrected internal parameters and the corrected external parameters to obtain a panoramic image;
the step S4 further includes a step S41 of determining that the same second feature point is in a same name group;
the step S41 includes establishing a common view field model, where the common view field model and the view field spherical model of each fisheye camera have a mapping relationship; for any first fisheye camera and any second fisheye camera with coincident fields of view, mapping the second feature points acquired by the first fisheye camera and the second fisheye camera to the common field of view model respectively, and if a second feature point corresponding to the first fisheye camera and a second feature point corresponding to the second fisheye camera are the closest second feature points on the common field of view model, determining the two second feature points in the same group.
2. The method as claimed in claim 1, wherein the center point of the reference spherical model coincides with the center point of one of the spherical models of the field of view.
3. The method as claimed in claim 1, wherein the center point of the reference spherical model is a midpoint between the center points of all the spherical models of the field of view.
4. The method as claimed in claim 1, wherein the fisheye cameras are two, the overlapped fields of view of the two fisheye cameras are annular fields of view, and the second calibration plates are uniformly arranged along the annular fields of view.
5. The method as claimed in claim 4, wherein an identification pattern is disposed on each of the second calibration plates, and the fisheye camera captures an image of the identification pattern to obtain an identification code for distinguishing the second calibration plates.
6. The method as claimed in claim 1, wherein the common view model is in the shape of a circular strip.
7. The method as claimed in claim 1, wherein the second calibration plate comprises black and white square patterns, and the corner point of each two adjacent square patterns, which is coincident with each other, is the second feature point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710148774.2A CN106952219B (en) | 2017-03-14 | 2017-03-14 | Image generation method for correcting fisheye camera based on external parameters |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710148774.2A CN106952219B (en) | 2017-03-14 | 2017-03-14 | Image generation method for correcting fisheye camera based on external parameters |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106952219A CN106952219A (en) | 2017-07-14 |
| CN106952219B true CN106952219B (en) | 2020-11-06 |
Family
ID=59466786
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710148774.2A Active CN106952219B (en) | 2017-03-14 | 2017-03-14 | Image generation method for correcting fisheye camera based on external parameters |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106952219B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110351494B (en) * | 2018-04-02 | 2020-06-26 | 杭州海康威视数字技术股份有限公司 | Panoramic video synthesis method and device and electronic equipment |
| CN112116661A (en) * | 2019-06-20 | 2020-12-22 | 北京地平线机器人技术研发有限公司 | High-precision map construction method and device |
| CN111210478B (en) * | 2019-12-31 | 2023-07-21 | 重庆邮电大学 | Common-view-free multi-camera system external parameter calibration method, medium and system |
| CN113834638A (en) * | 2021-09-03 | 2021-12-24 | 大连中科创达软件有限公司 | Method, device and equipment for determining field angle and storage medium |
| CN114757824B (en) * | 2022-03-18 | 2023-03-21 | 北京百度网讯科技有限公司 | Image splicing method, device, equipment and storage medium |
| CN115512242B (en) * | 2022-07-22 | 2023-05-30 | 北京微视威信息科技有限公司 | Scene change detection method and flight device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102005039A (en) * | 2010-08-24 | 2011-04-06 | 浙江大学 | Fish-eye camera stereo vision depth measuring method based on Taylor series model |
| CN102081798A (en) * | 2010-12-24 | 2011-06-01 | 北京控制工程研究所 | Epipolar rectification method for fish-eye stereo camera pair |
| CN104268867A (en) * | 2014-09-22 | 2015-01-07 | 国家电网公司 | Self-adaptive and rapid correcting method for fish-eye lens |
| CN105678787A (en) * | 2016-02-03 | 2016-06-15 | 西南交通大学 | Heavy-duty lorry driving barrier detection and tracking method based on binocular fisheye camera |
| CN106327478A (en) * | 2016-08-31 | 2017-01-11 | 北京的卢深视科技有限公司 | Fisheye panoramic image synthesis processing method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9402026B2 (en) * | 2013-01-05 | 2016-07-26 | Circular Logic Systems, Inc. | Spherical panoramic image camera rig |
| US9916660B2 (en) * | 2015-01-16 | 2018-03-13 | Magna Electronics Inc. | Vehicle vision system with calibration algorithm |
-
2017
- 2017-03-14 CN CN201710148774.2A patent/CN106952219B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102005039A (en) * | 2010-08-24 | 2011-04-06 | 浙江大学 | Fish-eye camera stereo vision depth measuring method based on Taylor series model |
| CN102081798A (en) * | 2010-12-24 | 2011-06-01 | 北京控制工程研究所 | Epipolar rectification method for fish-eye stereo camera pair |
| CN104268867A (en) * | 2014-09-22 | 2015-01-07 | 国家电网公司 | Self-adaptive and rapid correcting method for fish-eye lens |
| CN105678787A (en) * | 2016-02-03 | 2016-06-15 | 西南交通大学 | Heavy-duty lorry driving barrier detection and tracking method based on binocular fisheye camera |
| CN106327478A (en) * | 2016-08-31 | 2017-01-11 | 北京的卢深视科技有限公司 | Fisheye panoramic image synthesis processing method and device |
Non-Patent Citations (1)
| Title |
|---|
| Automatic Spherical Panorama Generation with Two Fisheye Images;Xiaoming Deng, ET AL.;《Proceedings of the 7th World Congress on Intelligent Control and Automation》;20081231;第5955-5959页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106952219A (en) | 2017-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106952219B (en) | Image generation method for correcting fisheye camera based on external parameters | |
| CN106886976B (en) | Image generation method for correcting fisheye camera based on internal parameters | |
| CN106875339B (en) | Fisheye image splicing method based on strip-shaped calibration plate | |
| CN110197466B (en) | Wide-angle fisheye image correction method | |
| WO2017092631A1 (en) | Image distortion correction method for fisheye image, and calibration method for fisheye camera | |
| CN109003311B (en) | Calibration method of fisheye lens | |
| CN106657910B (en) | A kind of panoramic video monitoring method of electricity substation | |
| CN108257183B (en) | Camera lens optical axis calibration method and device | |
| CN110809786B (en) | Calibration device, calibration chart, chart pattern generation device, and calibration method | |
| CN109903227B (en) | Panoramic image splicing method based on camera geometric position relation | |
| CN108122191B (en) | Method and device for splicing fisheye images into panoramic image and panoramic video | |
| CN110717942B (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| CN108734657B (en) | Image splicing method with parallax processing capability | |
| CN107705252B (en) | Method and system suitable for splicing, unfolding and correcting binocular fisheye image | |
| CN111461963B (en) | Fisheye image stitching method and device | |
| CN107527336B (en) | Lens relative position calibration method and device | |
| WO2020232971A1 (en) | Fisheye camera calibration system, method and apparatus, and electronic device and storage medium | |
| WO2023273108A1 (en) | Monocular distance measurement method and apparatus, and intelligent apparatus | |
| CN112734860A (en) | Arc-screen prior information-based pixel-by-pixel mapping projection geometric correction method | |
| CN112465915A (en) | Vehicle-mounted panoramic system calibration method | |
| CN111243034A (en) | Panoramic auxiliary parking calibration method, device, equipment and storage medium | |
| CN114549666B (en) | AGV-based panoramic image splicing calibration method | |
| CN110136205B (en) | Parallax calibration method, device and system of multi-view camera | |
| CN112258581B (en) | On-site calibration method for panoramic camera with multiple fish glasses heads | |
| CN111383264A (en) | Positioning method, positioning device, terminal and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |