CN110660105B - Calibration parameter optimization method and device for panoramic looking-around system - Google Patents

Calibration parameter optimization method and device for panoramic looking-around system Download PDF

Info

Publication number
CN110660105B
CN110660105B CN201810712870.XA CN201810712870A CN110660105B CN 110660105 B CN110660105 B CN 110660105B CN 201810712870 A CN201810712870 A CN 201810712870A CN 110660105 B CN110660105 B CN 110660105B
Authority
CN
China
Prior art keywords
image
calibration parameters
matching
panoramic
residual error
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810712870.XA
Other languages
Chinese (zh)
Other versions
CN110660105A (en
Inventor
谢小燕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Hikvision Digital Technology Co Ltd
Original Assignee
Hangzhou Hikvision Digital Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Hikvision Digital Technology Co Ltd filed Critical Hangzhou Hikvision Digital Technology Co Ltd
Priority to CN201810712870.XA priority Critical patent/CN110660105B/en
Publication of CN110660105A publication Critical patent/CN110660105A/en
Application granted granted Critical
Publication of CN110660105B publication Critical patent/CN110660105B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/04—Context-preserving transformations, e.g. by using an importance map
    • G06T3/047—Fisheye or wide-angle transformations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/757—Matching configurations of points or features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

The application provides a calibration parameter optimization method and a calibration parameter optimization device for a panoramic looking-around system, wherein the method comprises the following steps: acquiring a group of regional images acquired by a panoramic all-round system at the same time; extracting feature points in the overlapping area of the adjacent area images aiming at each pair of adjacent area images in the group of area images, and matching the feature points of the extracted adjacent area images to obtain matched feature point pairs; and after the matching characteristic point pairs corresponding to each pair of adjacent area images are obtained, optimizing the calibration parameters of the panoramic all-around system according to the obtained matching characteristic point pairs. The method is characterized in that characteristic points are selected in the overlapping area of the area images to calibrate parameters for optimization, and the external parameters of all cameras in the panoramic all-around system are associated together for optimization, so that the splicing effect of the panoramic images is improved, and the problem of closure of a closed loop is solved.

Description

Calibration parameter optimization method and device of panoramic looking-around system
Technical Field
The application relates to the field of computer vision, in particular to a calibration parameter optimization method and device for a panoramic looking-around system.
Background
The panoramic all-round looking system is characterized in that four or more fisheye cameras are arranged on the periphery of a vehicle, pictures shot by the fisheye cameras are spliced in real time and displayed on one picture at the same time, and finally a panoramic image at an aerial overlooking angle can be obtained. As shown in fig. 1, a panoramic image of the vehicle is obtained.
Because the image provided by the fisheye camera has distortion, namely, except the scenery in the center of the picture is kept unchanged, other horizontal or vertical scenery has corresponding changes, the fisheye camera needs to be calibrated, then the distorted image is corrected into a distortion-free image by utilizing calibration parameters, and then a plurality of distortion-free images are spliced into a panoramic image around the vehicle.
The essence of the calibration is to establish the relationship between the image pixel position and the real scene position. From the real scene coordinates to the image coordinates, a plurality of coordinate system projection transformations are required. To this end, 4 coordinate systems are established: a world coordinate system, a camera coordinate system, an imaging plane coordinate system, and an image coordinate system. The calibration parameters comprise camera external parameters and camera internal parameters; the projective transformation relationship from the world coordinate system to the camera coordinate system is related only to the installation position and the installation manner of the camera, and is called as camera external parameter. The transformation from camera coordinates to image coordinates is related to the chip and the process of the camera, and once the camera is assembled, the transformation is not influenced by factors such as the installation position of the camera, the use environment and the like, and is called as camera internal reference. The use environment of the camera is irrelevant to the installation mode, once the design is finished, the camera can be regarded as a constant parameter, and the external parameter can be greatly changed due to the influence of the installation, a reference object and the use environment and needs to be calibrated according to the actual condition.
Disclosure of Invention
The application provides a calibration parameter optimization method and device of a panoramic looking-around system, which are used for improving the precision of calibration parameters and improving the splicing effect of panoramic images of vehicles.
Specifically, the method is realized through the following technical scheme:
in a first aspect of the present application, a calibration parameter optimization method for a panoramic all-around view system is provided, including:
acquiring a group of regional images acquired by a panoramic looking-around system at the same time;
extracting feature points in the overlapping area of the adjacent area images aiming at each pair of adjacent area images in the group of area images, and matching the feature points of the extracted adjacent area images to obtain matched feature point pairs;
and after the matching characteristic point pairs of each pair of adjacent area images are obtained, optimizing the calibration parameters of the panoramic all-around system according to the obtained matching characteristic point pairs.
In a second aspect of the present application, a calibration parameter optimization apparatus for a panoramic looking-around system is provided, which has the function of implementing the method provided in the first aspect. The functions can be realized by hardware, and the functions can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules or units corresponding to the above functions.
In one implementation, the apparatus may include:
the acquisition module is used for acquiring a group of regional images acquired by the panoramic all-round looking system at the same moment;
the matching module is used for extracting feature points in the overlapping area of the adjacent area images aiming at each pair of adjacent area images in the group of area images, and matching the feature points of the extracted adjacent area images to obtain matched feature point pairs;
and the optimization module is used for optimizing the calibration parameters of the panoramic looking-around system according to the obtained matching characteristic point pairs after the matching characteristic point pairs of each pair of adjacent area images are obtained.
In another implementation, the apparatus may include a processor, a memory, and a bus, where the processor and the memory are connected to each other through the bus; the memory stores machine-readable instructions, and the processor executes the method provided by the first aspect of the present application by calling the machine-readable instructions.
According to the method, the calibration parameters are optimized by selecting the characteristic points in the overlapped areas of the area images aiming at the area images shot by the fish-eye cameras on the vehicle at the same moment, and the external parameters of all the cameras in the panoramic all-around system are associated together for optimization in a pairwise association mode between the cameras, so that the splicing effect of the panoramic images is improved, and the problem of closure of a closed loop is solved.
Drawings
FIG. 1 is a schematic illustration of a panoramic image of a vehicle;
FIG. 2 is a schematic illustration of a panoramic image of an unclosed vehicle;
FIG. 3 is a flow chart of a method provided by an embodiment of the present application;
fig. 4 is a schematic diagram of a panoramic image composed of four area images according to an embodiment of the present application;
FIG. 5 is a detailed flowchart of calibration parameter optimization provided by an embodiment of the present application;
fig. 6 is a block diagram of device modules provided in an embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
Aiming at the problems of low calibration precision, efficiency and applicability of the calibration scheme of the existing panoramic looking-around system, the patent scheme with the publication number of CN105608693A and the invention name of vehicle-mounted panoramic looking-around calibration system and method provides an optimization method of calibration parameters, and the method comprises the following steps: defining an optimization target, namely an error function, and when the error function reaches the minimum, indicating that the current calibration parameter is the optimal solution under the current information; the error function is:
Figure BDA0001716923420000041
where m denotes the total number of cameras in the panoramic looking-around system, n denotes the total number of feature points of the feature, PkiRepresents the projection position of the ith image feature point in the kth camera, KkRepresenting the Kth camera projectionShadow matrix, PiThe three-dimensional coordinates of the i-th actual feature point in the global coordinate system are represented, and the function D (,) represents the euclidean distance in the image space.
According to the formula, all feature points in an image participate in calculation, the reprojection error of the feature points is used as a constraint function, and internal and external parameters of the camera are obtained through iterative optimization; the method does not establish the relation between the cameras per se, namely, the internal and external parameter optimization between the cameras are mutually independent, so that the problem that the places where the panoramic images are connected end to end are not closed is probably caused, and the fact that the accuracy of the calibration parameters is required to be checked after the optimized calibration parameters are obtained in the patent scheme can be proved.
As shown in fig. 2, the panoramic image of the vehicle is an unclosed vehicle panoramic image, and the unclosed vehicle panoramic image shows that a straight line on the left side of the vehicle in the image is displaced. The unclosed panoramic image brings inconvenience and even danger to the driver during the running process of the vehicle.
In order to solve the problem that spliced panoramic images are not closed, the invention provides a calibration parameter optimization scheme applied to a panoramic all-around system, aiming at regional images shot by a fisheye camera on a vehicle at the same time, and selecting characteristic points in the overlapping regions of the regional images to optimize the calibration parameters, which is equivalent to associating external parameters of all cameras in the panoramic all-around system together for optimization, so that the splicing effect of the panoramic images can be improved.
The technical scheme provided by the application can be applied to the calibration process when the vehicle leaves the factory and also can be applied to the recalibration process when the vehicle is maintained. The execution main body of the technical scheme provided by the application can be an on-vehicle controller, and can also be a controller externally connected to a vehicle in a wired or wireless mode.
Referring to fig. 3, in one embodiment, the calibration parameter optimization method for a panoramic looking-around system provided by the present application may include the following steps:
step 301: and acquiring a group of regional images acquired by the panoramic all-round looking system at the same time.
The number of the area images acquired by the panoramic looking-around system at the same time is equal to the number of the fisheye cameras arranged around the vehicle, at present, four or more fisheye cameras are generally arranged around the vehicle in the panoramic looking-around system, and therefore a group of area images acquired here usually at least comprises four area images.
Step 302: and extracting characteristic points in the overlapping region of the adjacent region images aiming at each pair of adjacent region images in the group of region images, and matching the characteristic points of the extracted adjacent region images to obtain matched characteristic point pairs.
The matching characteristic point pairs represent the positions of the same point in space on the images of the two adjacent areas.
The implementation of step 302 is illustrated here with a panoramic image as shown in fig. 4. The panoramic image is formed by splicing four area images, the rectangular area at the center represents a vehicle, the rectangular area at the periphery of the vehicle represents an area image shot by a fisheye camera, and the overlapping area between the area images is shown in a darker color in fig. 4. For each pair of adjacent region images, such as region image 1 and region image 2, the following steps may be performed:
firstly, confirming an overlapped area of an area image 1 and an area image 2;
secondly, respectively extracting characteristic points in the overlapped area on the area image 1 and the area image 2;
and thirdly, matching the extracted feature points to obtain matched feature point pairs. The extraction and matching of the image feature points are basic problems in computer vision, common feature point algorithms include SIFT, SURF, FAST and the like, and the extraction and matching process of the feature points is not described in detail here.
Through the steps, the matching characteristic point pairs of the area image 1 and the area image 2 in the overlapping area can be obtained. For example, in fig. 4, point a and point B are a pair of matching feature point pairs in the overlapping region of region image 1 and region image 2; ideally, the point a and the point B should be overlapped to a point on the stitched panoramic image, but in practice, the point a and the point B are misaligned due to reasons such as inaccurate external reference calibration.
Through similar processing, matching feature point pairs of the region image 1 and the region image 4, the region image 2 and the region image 3, and the region image 3 and the region image 4 can be obtained, and the feature points constituting the matching feature point pairs are all located in the overlapping region in fig. 4.
Step 303: and after the matching characteristic point pairs of each pair of adjacent area images are obtained, optimizing the calibration parameters of the panoramic all-around system according to the obtained matching characteristic point pairs.
In an alternative embodiment, the calibration parameter optimization process described in step 303 can be implemented by the method shown in fig. 5:
step 501: and calculating the residual error of each matched characteristic point pair based on the initial calibration parameters, wherein the residual error is the error between the projection coordinates of the two characteristic points when the two characteristic points forming the matched characteristic point pair are projected into the same image.
The better the coincidence of the two characteristic points forming the matched characteristic point pair, the smaller the absolute value of the residual error obtained by calculation. When the two feature points are completely coincident, the corresponding residual value is zero.
As an implementation manner, assuming that two feature points a and B constituting a matching feature point pair are located on an image i and an image j, respectively, if the B point is projected into the image i, a residual error of the matching feature point pair can be calculated by the following formula:
Figure BDA0001716923420000061
wherein the content of the first and second substances,
Figure BDA0001716923420000062
is the coordinate of point a in image i,
Figure BDA0001716923420000063
representing the projection coordinates of point B into image i,
Figure BDA0001716923420000064
the calculation formula is as follows:
Figure BDA0001716923420000065
wherein the content of the first and second substances,
Figure BDA0001716923420000066
is the coordinate of point B in image j, KiAnd [ R ]iTi]Internal and external references, K, respectively, of the camera responsible for taking the image i with respect to the world coordinate systemjAnd [ R ]jTj]Respectively the internal and external reference of the camera responsible for taking the image j with respect to the world coordinate system,
Figure BDA0001716923420000068
and [ R ]jTj]-1Is KjAnd [ R ]jTj]The inverse matrix of (c).
For camera reference matrix (such as K described above)iAnd Kj) The form can be expressed as follows:
Figure BDA0001716923420000067
wherein f isuAnd fvIs a focal length of u0And v0The principal point coordinates are γ (═ 0) which is a skewness coefficient. These are camera intrinsic properties, one camera intrinsic parameter matrix for each camera.
With respect to the rotation matrix R in the camera profile (such as R mentioned above)iAnd Rj) The form can be expressed as follows:
Figure BDA0001716923420000071
for translation vector T in camera external parameters (such as T described above)iAnd Tj) The form can be expressed as follows:
Figure BDA0001716923420000072
it should be noted that the above formula (1) and formula (2) are derived mainly based on the premise that the point B is projected to the image i where the point a is located; further, if the B point and the a point are projected onto another image other than the image i and the image j to calculate an error between the projected coordinates of the a point and the B point, a correlation modification of the above formula is required. Moreover, all the formulas mentioned in the technical scheme of the application can have more logical variations.
Step 502: and optimizing the initial calibration parameters according to the residual error of each matched characteristic point pair calculated based on the initial calibration parameters to obtain new calibration parameters.
As an implementation manner, the new calibration parameters can be obtained through the following steps:
s5021: and accumulating the residual errors of each matched characteristic point pair calculated based on the initial calibration parameters to obtain the residual error sum.
For example, the residual error of each matching feature point pair calculated based on the initial calibration parameters may be substituted into the following formula to obtain a residual error sum:
Figure BDA0001716923420000073
wherein P istThe calibration parameters representing the panoramic looking-around system, i.e. the calibration parameters of all cameras in the system, can be expressed as a 1 x 10 dimensional matrix, and the internal reference K and the external reference of each camera are expressed [ R T ]]After the combination modification, a calibration parameter matrix of the camera in 1 x 10 dimensions can be obtained.
e(Pt) Representation pair based on PtCalculating the residual error of each matched characteristic point, and accumulating to obtain a residual error sum; n is the number of regional images shot by the panoramic looking-around system at the same time,
Figure BDA0001716923420000074
a matching image representing the image i,
Figure BDA0001716923420000075
representing pairs of matching feature points of image i and image j,
Figure BDA0001716923420000076
is a residual error
Figure BDA0001716923420000077
Is used as the error iteration function.
Here, the
Figure BDA0001716923420000081
There are many forms, two of which are simply enumerated here.
In the first kind of the method, the first,
Figure BDA0001716923420000082
can be equal to
Figure BDA0001716923420000083
Based on the above formula (6), it is equivalent to directly summing the residual errors of each matching feature point pair, and the obtained sum value is the residual error sum.
In the second type of the above-mentioned methods,
Figure BDA0001716923420000084
the following formula requirements can be met:
Figure BDA0001716923420000085
where σ is a known constant.
S5022: calculating an iteration increment matrix delta P according to the obtained residual sumt。
Wherein the iterative delta matrix Δ PtThe following formula requirements can be met:
ΔPt＝[JT(Pt)J(Pt)+μC]-1JT(Pt)e(Pt) Formula (8)
Wherein J: (Pt) Is PtJacobian matrix of, JT(Pt) Is J (P)t) And μ C is a confidence matrix used to control the iteration speed.
The confidence matrix C is of the form:
Figure BDA0001716923420000086
σθis composed of
Figure BDA0001716923420000087
σfIs composed of
Figure BDA0001716923420000088
Both are constants; if the residual sum obtained by the next calculation is smaller than that obtained by the previous calculation in the iteration process, namely e (P)t)<e(Pt+1) Then μ is decreased, otherwise it is increased (μ is a coefficient).
S5023: from the iterative delta matrix Δ PtAnd an initial calibration parameter PtNew calibration parameter P can be obtainedt+1＝Pt+ΔPt。
Step 503: and judging iteration termination conditions.
The iteration termination condition may be that the number of iterations reaches a set upper limit value, or a calibration parameter P obtained by two adjacent iterationst+1And PtThe difference between them is less than a set threshold. As long as any one of the above conditions is satisfied, the iteration termination condition is considered to be satisfied.
If the iteration termination condition is met, the new calibration parameters can be used as the final calibration parameters of the panoramic looking-around system. If the iteration termination condition is not satisfied, the new calibration parameter needs to be optimized continuously, that is, returning to step 501, taking the new calibration parameter as the initial calibration parameter, and re-executing steps 501-503.
After the final calibration parameters of the panoramic looking-around system are obtained, the panoramic looking-around system can be set by using the external parameters in the final calibration parameters.
To sum up, the technical scheme of the application selects characteristic points from the overlapping areas of the area images shot by the fish-eye cameras on the vehicle at the same time to optimize the calibration parameters, and associates the external parameters of all the cameras in the panoramic all-around system together in a pairwise association mode between the cameras to optimize, so that the splicing effect of the panoramic images is improved, and the problem of closure of a closed loop is solved.
The methods provided herein are described above. The apparatus provided in the present application is described below.
Referring to fig. 6, fig. 6 is a functional block diagram of a calibration parameter optimization apparatus of a panoramic looking-around system provided in the present application. As shown in fig. 6, the apparatus includes:
the obtaining module 601 is configured to obtain a group of area images collected by the panoramic looking-around system at the same time.
The matching module 602 is configured to, for each pair of adjacent region images in the group of region images, extract feature points in a coincidence region of the adjacent region images, and perform matching on the feature points of the extracted adjacent region images to obtain matched feature point pairs.
And the optimizing module 603 is configured to, after obtaining the matching feature point pairs of each pair of neighboring area images, optimize calibration parameters of the panoramic all-around system according to the obtained matching feature point pairs.
In one embodiment, the optimizing module 603 is configured to calculate a residual error of each matching feature point pair based on the initial calibration parameter, where the residual error is an error between projection coordinates of two feature points constituting the matching feature point pair when the two feature points are projected onto the same image; optimizing the initial calibration parameters according to the residual error of each matched characteristic point pair calculated based on the initial calibration parameters to obtain new calibration parameters; judging iteration termination conditions, and if the iteration termination conditions are met, taking the new calibration parameters as final calibration parameters of the panoramic all-around system; if the iteration termination condition is not met, the new calibration parameters are continuously optimized until the iteration termination condition is met.
In one embodiment, the optimization module 603 is configured to accumulate residuals of each matching feature point calculated based on the initial calibration parameter to obtain a residual sum; calculating an iteration increment matrix delta P according to the obtained residual sumt(ii) a From the iterative delta matrix Δ PtAnd an initial calibration parameter PtTo obtain a new calibration parameter Pt+1＝Pt+ΔPt。
In one embodiment, the residual error of each matching feature point pair satisfies the following formula requirement:
assuming that two feature points a and B constituting a matching feature point pair are located on image i and image j, respectively, the residual error of the matching feature point pair is:
Figure BDA0001716923420000101
wherein, the first and the second end of the pipe are connected with each other,
Figure BDA0001716923420000102
is the coordinate of point a in image i,
Figure BDA0001716923420000103
representing the projection coordinates of point B into image i,
Figure BDA0001716923420000104
the calculation formula is as follows:
Figure BDA0001716923420000105
wherein the content of the first and second substances,
Figure BDA0001716923420000106
is the coordinate of point B in image j, KiAnd [ R ]iTi]Internal and external references, K, respectively, of the camera responsible for taking the image i with respect to the world coordinate systemjAnd [ R ]jTj]With respect to the world coordinate system, each camera being responsible for taking image jThe internal and external ginseng are used as raw materials,
Figure BDA0001716923420000107
and [ R ]jTj]-1Is KjAnd [ R ]jTj]The inverse matrix of (c).
In one embodiment, the residual sum satisfies the following formula:
Figure BDA0001716923420000108
wherein P istCalibration parameters for panoramic looking-around systems, e (P)t) To be based on PtCalculating residual errors of each matched characteristic point pair, and accumulating to obtain residual error sum; n is the number of region images included in the set of region images,
Figure BDA0001716923420000109
a matching image representing the image i is shown,
Figure BDA00017169234200001010
representing pairs of matching feature points of image i and image j,
Figure BDA00017169234200001011
is a residual error
Figure BDA00017169234200001012
Is used as the error iteration function.
In one embodiment, the iterative delta matrix Δ PtThe following formula requirements are met:
ΔPt＝[JT(Pt)J(Pt)+μC]-1JT(Pt)e(Pt)
wherein, J (P)t) Is PtJacobian matrix of, JT(Pt) Is J (P)t) And μ C is a confidence matrix used to control the iteration speed.
In one embodiment, theAs described in
Figure BDA0001716923420000111
The following formula requirements are met:
Figure BDA0001716923420000112
where σ is a known constant.
In one embodiment, the iteration termination condition is: the iteration times reach a set upper limit value; or the difference value between the calibration parameters obtained by two adjacent iterations is smaller than a set threshold value.
It should be noted that the division of the unit in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation. The functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The specific details of the implementation process of the functions and actions of each unit in the above device are the implementation processes of the corresponding steps in the above method, and are not described herein again.
The description of the apparatus shown in fig. 5 is thus completed.
The application also provides a calibration parameter optimization device of the panoramic looking-around system, which comprises a processor, a memory and a bus, wherein the processor and the memory are connected with each other through the bus; the memory stores machine-readable instructions that the processor invokes to implement the method shown in fig. 3.
Additionally, a machine-readable storage medium is provided that stores machine-readable instructions which, when invoked and executed by a processor, cause the processor to implement the method illustrated in fig. 3.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims (16)

1. A calibration parameter optimization method of a panoramic looking-around system is characterized by comprising the following steps:
acquiring a group of regional images acquired by a panoramic looking-around system at the same time;
extracting feature points in the overlapping area of the adjacent area images aiming at each pair of adjacent area images in the group of area images, and matching the feature points of the extracted adjacent area images to obtain matched feature point pairs;
after the matched characteristic point pairs of each pair of adjacent area images are obtained, the calibration parameters of the panoramic looking-around system are optimized according to the residual errors of the obtained matched characteristic point pairs;
wherein, the residual error of each matching feature point pair meets the following formula requirement:
assuming that two feature points a and B constituting a matching feature point pair are located on image i and image j, respectively, the residual error of the matching feature point pair is:
Figure FDA0003541575070000011
wherein the content of the first and second substances,
Figure FDA0003541575070000012
is the coordinate of point a in image i,
Figure FDA0003541575070000013
representing the projection coordinates of point B into image i,
Figure FDA0003541575070000014
the calculation formula is as follows:
Figure FDA0003541575070000015
wherein the content of the first and second substances,
Figure FDA0003541575070000016
is the coordinate of point B in image j, KiAnd [ R ]iTi]Internal and external references, K, respectively, of the camera responsible for taking the image i with respect to the world coordinate systemjAnd [ R ]jTj]Respectively the internal and external references of the camera responsible for taking the image j with respect to the world coordinate system,
Figure FDA0003541575070000017
and [ R ]jTj]-1Is KjAnd [ R ]jTj]The inverse matrix of (c).
2. The method according to claim 1, wherein the residual is an error between projection coordinates of two feature points constituting a matching feature point pair when the two feature points are projected into the same image;
according to the obtained residual error of the matching characteristic point pair, the calibration parameters of the panoramic looking-around system are optimized, and the method comprises the following steps:
optimizing the initial calibration parameters according to the residual error of each matched characteristic point pair calculated based on the initial calibration parameters to obtain new calibration parameters;
judging iteration termination conditions, and if the iteration termination conditions are met, taking the new calibration parameters as final calibration parameters of the panoramic all-around system; and if the iteration termination condition is not met, continuously optimizing the new calibration parameter until the iteration termination condition is met.
3. The method as claimed in claim 2, wherein said optimizing the initial calibration parameters according to the residual error of each matched feature point pair calculated based on the initial calibration parameters to obtain new calibration parameters comprises:
accumulating the residual error of each matched characteristic point calculated based on the initial calibration parameters to obtain the residual error sum;
calculating an iteration increment matrix delta P according to the obtained residual sumt；
According to an iterative delta matrix Δ PtAnd an initial calibration parameter PtTo obtain a new calibration parameter Pt+1＝Pt+ΔPt。
4. The method of claim 1, wherein the residual sum satisfies the following formula requirement:
Figure FDA0003541575070000021
wherein P istCalibration parameters for panoramic looking-around systems, e (P)t) To be based on PtCalculating residual errors of each matched characteristic point pair, and accumulating to obtain residual error sum; n is the number of region images included in the set of region images,
Figure FDA0003541575070000022
a matching image representing the image i is shown,
Figure FDA0003541575070000023
representing pairs of matching feature points of image i and image j,
Figure FDA0003541575070000024
is a residual error
Figure FDA0003541575070000025
Is used as the error iteration function.
5. The method of claim 3, in which the iterative delta matrix Δ PtThe following formula requirements are met:
ΔPt＝[JT(Pt)J(Pt)+μC]-1JT(Pt)e(Pt)
wherein, J (P)t) Is PtJacobian matrix of JT(Pt) Is J (P)t) And μ C is a confidence matrix used to control the iteration speed.
6. The method of claim 4, wherein the method is as set forth in claim 4
Figure FDA0003541575070000026
The following formula requirements are met:
Figure FDA0003541575070000027
where σ is a known constant.
7. The method of claim 2, wherein the iteration termination condition is:
the iteration times reach a set upper limit value; or
And the difference value between the calibration parameters obtained by two adjacent iterations is smaller than a set threshold value.
8. A calibration parameter optimization device of a panoramic looking-around system is characterized by comprising:
the acquisition module is used for acquiring a group of regional images acquired by the panoramic all-round looking system at the same moment;
the matching module is used for extracting feature points in the overlapping area of the adjacent area images aiming at each pair of adjacent area images in the group of area images, and matching the feature points of the extracted adjacent area images to obtain matched feature point pairs;
the optimization module is used for optimizing the calibration parameters of the panoramic all-around system according to the residual error of the obtained matching characteristic point pairs after the matching characteristic point pairs of each pair of adjacent area images are obtained;
wherein, the residual error of each matching feature point pair meets the following formula requirement:
assuming that two feature points a and B constituting a matching feature point pair are located on image i and image j, respectively, the residual error of the matching feature point pair is:
Figure FDA0003541575070000031
wherein, the first and the second end of the pipe are connected with each other,
Figure FDA0003541575070000032
is the coordinate of point a in image i,
Figure FDA0003541575070000033
representing the projection coordinates of point B into image i,
Figure FDA0003541575070000034
the calculation formula is as follows:
Figure FDA0003541575070000035
wherein the content of the first and second substances,
Figure FDA0003541575070000036
is the coordinate of point B in image j, KiAnd [ R ]iTi]Internal and external references, K, respectively, of the camera responsible for taking the image i with respect to the world coordinate systemjAnd [ R ]jTj]Respectively the internal and external references of the camera responsible for taking the image j with respect to the world coordinate system,
Figure FDA0003541575070000037
and [ R ]jTj]-1Is KjAnd [ R ]jTj]The inverse matrix of (c).
9. The apparatus according to claim 8, wherein the residual is an error between projection coordinates of two feature points constituting a matching feature point pair when the two feature points are projected into the same image;
the optimization module is used for optimizing the initial calibration parameters according to the residual error of each matched feature point pair calculated based on the initial calibration parameters to obtain new calibration parameters; judging iteration termination conditions, and if the iteration termination conditions are met, taking the new calibration parameters as final calibration parameters of the panoramic all-around system; and if the iteration termination condition is not met, continuously optimizing the new calibration parameter until the iteration termination condition is met.
10. The apparatus of claim 9,
the optimization module is used for accumulating the residual error of each matching characteristic point calculated based on the initial calibration parameters to obtain the residual error sum; calculating an iteration increment matrix delta P according to the obtained residual sumt(ii) a From the iterative delta matrix Δ PtAnd an initial calibration parameter PtTo obtain a new calibration parameter Pt+1＝Pt+ΔPt。
11. The apparatus of claim 8, wherein the sum of residuals satisfies the following formula requirement:
Figure FDA0003541575070000041
wherein P istCalibration parameters for panoramic looking-around systems, e (P)t) To be based on PtCalculating residual errors of each matched characteristic point pair, and accumulating the residual errors to obtain residual error sum; n is the number of region images included in the set of region images,
Figure FDA0003541575070000042
a matching image representing the image i,
Figure FDA0003541575070000043
representing pairs of matching feature points of image i and image j,
Figure FDA0003541575070000044
is a residual error
Figure FDA0003541575070000045
Is used as the error iteration function.
12. The apparatus of claim 10, in which the iterative delta matrix Δ ΡtThe following formula requirements are met:
ΔPt＝[JT(Pt)J(Pt)+μC]-1JT(Pt)e(Pt)
wherein, J (P)t) Is PtJacobian matrix of JT(Pt) Is J (P)t) And μ C is a confidence matrix used to control the iteration speed.
13. The apparatus of claim 11, wherein the apparatus is characterized in that
Figure FDA0003541575070000046
The following formula requirements are met:
Figure FDA0003541575070000047
where σ is a known constant.
14. The apparatus of claim 9, wherein the iteration termination condition is:
the iteration times reach a set upper limit value; or
And the difference value between the calibration parameters obtained by two adjacent iterations is smaller than a set threshold value.
15. A calibration parameter optimization device of a panoramic looking-around system is characterized by comprising a processor, a memory and a bus, wherein the processor and the memory are connected with each other through the bus;
the memory has stored therein machine-readable instructions, the processor executing the method of any of claims 1 to 7 by calling the machine-readable instructions.
16. A machine-readable storage medium having stored thereon machine-readable instructions which, when invoked and executed by a processor, cause the processor to carry out the method of any one of claims 1 to 7.
CN201810712870.XA 2018-06-29 2018-06-29 Calibration parameter optimization method and device for panoramic looking-around system Active CN110660105B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810712870.XA CN110660105B (en) 2018-06-29 2018-06-29 Calibration parameter optimization method and device for panoramic looking-around system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810712870.XA CN110660105B (en) 2018-06-29 2018-06-29 Calibration parameter optimization method and device for panoramic looking-around system

Publications (2)

Publication Number Publication Date
CN110660105A CN110660105A (en) 2020-01-07
CN110660105B true CN110660105B (en) 2022-05-31

Family

ID=69027788

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810712870.XA Active CN110660105B (en) 2018-06-29 2018-06-29 Calibration parameter optimization method and device for panoramic looking-around system

Country Status (1)

Country Link
CN (1) CN110660105B (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113393529B (en) * 2020-03-12 2024-05-10 浙江宇视科技有限公司 Method, device, equipment and medium for calibrating camera
CN111652802B (en) * 2020-05-19 2024-03-05 杭州海康威视数字技术股份有限公司 Panorama making method, interaction method and device based on panorama
CN113706624A (en) * 2020-05-20 2021-11-26 杭州海康威视数字技术股份有限公司 Camera external parameter correction method and device and vehicle-mounted all-round-looking system
CN111445537B (en) * 2020-06-18 2020-09-29 浙江中控技术股份有限公司 Calibration method and system of camera
CN112785652A (en) * 2020-12-24 2021-05-11 广州小鹏自动驾驶科技有限公司 Panoramic calibration method and device
CN112767496B (en) * 2021-01-22 2023-04-07 阿里巴巴集团控股有限公司 Calibration method, device and system
CN113139490B (en) * 2021-04-30 2024-02-23 中德（珠海）人工智能研究院有限公司 Image feature matching method and device, computer equipment and storage medium
CN115082573B (en) * 2022-08-19 2023-04-11 小米汽车科技有限公司 Parameter calibration method and device, vehicle and storage medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN205563716U (en) * 2016-03-30 2016-09-07 广州市盛光微电子有限公司 Panoramic camera calibration device based on many camera lenses multisensor
CN106846415A (en) * 2017-01-24 2017-06-13 长沙全度影像科技有限公司 A kind of multichannel fisheye camera binocular calibration device and method

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101877140A (en) * 2009-12-18 2010-11-03 北京邮电大学 Panorama-based panoramic virtual tour method
CN104933755B (en) * 2014-03-18 2017-11-28 华为技术有限公司 A kind of stationary body method for reconstructing and system
CN103985118A (en) * 2014-04-28 2014-08-13 无锡观智视觉科技有限公司 Parameter calibration method for cameras of vehicle-mounted all-round view system
US9928594B2 (en) * 2014-07-11 2018-03-27 Agt International Gmbh Automatic spatial calibration of camera network
CN105608693B (en) * 2015-12-18 2018-08-28 上海欧菲智能车联科技有限公司 The calibration system and method that vehicle-mounted panoramic is looked around
CN106097300B (en) * 2016-05-27 2017-10-20 西安交通大学 A kind of polyphaser scaling method based on high-precision motion platform
CN106651767A (en) * 2016-12-30 2017-05-10 北京星辰美豆文化传播有限公司 Panoramic image obtaining method and apparatus
CN108171759A (en) * 2018-01-26 2018-06-15 上海小蚁科技有限公司 The scaling method of double fish eye lens panorama cameras and device, storage medium, terminal

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN205563716U (en) * 2016-03-30 2016-09-07 广州市盛光微电子有限公司 Panoramic camera calibration device based on many camera lenses multisensor
CN106846415A (en) * 2017-01-24 2017-06-13 长沙全度影像科技有限公司 A kind of multichannel fisheye camera binocular calibration device and method

Also Published As

Publication number Publication date
CN110660105A (en) 2020-01-07

Similar Documents

Publication Publication Date Title
CN110660105B (en) Calibration parameter optimization method and device for panoramic looking-around system
CN109902637B (en) Lane line detection method, lane line detection device, computer device, and storage medium
CN108805934B (en) External parameter calibration method and device for vehicle-mounted camera
CN105096329B (en) Method for accurately correcting image distortion of ultra-wide-angle camera
US20140085409A1 (en) Wide fov camera image calibration and de-warping
CN109741241B (en) Fisheye image processing method, device, equipment and storage medium
CN112257698B (en) Method, device, equipment and storage medium for processing annular view parking space detection result
CN111899282A (en) Pedestrian trajectory tracking method and device based on binocular camera calibration
CN105488766A (en) Fish-eye lens image correcting method and device
JP5228614B2 (en) Parameter calculation apparatus, parameter calculation system and program
CN114494462A (en) Binocular camera ranging method based on Yolov5 and improved tracking algorithm
CN113034616A (en) Camera external reference calibration method and system for vehicle all-round looking system and all-round looking system
CN112465915A (en) Vehicle-mounted panoramic system calibration method
KR101770668B1 (en) Automatic calibration method based on the simplified pattern for the vehicle image registration and the method thereof
CN113793266A (en) Multi-view machine vision image splicing method, system and storage medium
WO2015015542A1 (en) Vehicle-mounted stereo camera system and calibration method therefor
CN109658451B (en) Depth sensing method and device and depth sensing equipment
CN110097064B (en) Picture construction method and device
CN111383264B (en) Positioning method, positioning device, terminal and computer storage medium
KR101697229B1 (en) Automatic calibration apparatus based on lane information for the vehicle image registration and the method thereof
CN111652937B (en) Vehicle-mounted camera calibration method and device
CN108520541B (en) Calibration method of wide-angle camera
TWI424259B (en) Camera calibration method
CN110910311A (en) Automatic splicing method for multi-channel panoramic camera based on two-dimensional code
Geiger Monocular road mosaicing for urban environments

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant