CN115311336A - Image registration method, device and equipment of multiple cameras and storage medium - Google Patents
Image registration method, device and equipment of multiple cameras and storage medium Download PDFInfo
- Publication number
- CN115311336A CN115311336A CN202210946257.0A CN202210946257A CN115311336A CN 115311336 A CN115311336 A CN 115311336A CN 202210946257 A CN202210946257 A CN 202210946257A CN 115311336 A CN115311336 A CN 115311336A
- Authority
- CN
- China
- Prior art keywords
- image
- camera
- calibration
- parameter
- cost function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
- G06T2207/30208—Marker matrix
Abstract
The application discloses a method, a device and equipment for image registration of multiple cameras and a storage medium, and relates to the technical field of three-dimensional reconstruction. The technical scheme includes that a first image collected by a first camera and a second image collected by a second camera are obtained; registering the first image and the second image based on a first internal reference parameter of the first camera, a second internal reference parameter of the second camera and an external reference parameter between the first camera and the second camera which are calibrated in advance; the cost function adopted when the first internal reference parameter, the second internal reference parameter and the external reference parameter are calibrated in advance comprises a regularization cost function, and the regularization cost function is constructed on the basis of an infinite homography parameter between the first camera and the second camera. By the technical means, the calibration error is reduced to improve the accuracy of the real-time image registration, so that the problem that the real-time image registration error in the prior art cannot meet the requirement is solved.
Description
Technical Field
The present application relates to the field of image registration technologies, and in particular, to a method, an apparatus, a device, and a storage medium for image registration of multiple cameras.
Background
The real-time image registration refers to a process of converting a plurality of images shot at the same time into the same pixel space while controlling the multiple cameras to shoot the images. At present, a relatively mature image registration technology is based on block or feature point registration, but the two registration methods are too large in calculation amount and low in registration efficiency, and are not suitable for real-time registration scenes.
In the prior art, images shot by each camera are uniformly converted into a pixel space of a certain camera through an internal reference matrix and an external reference matrix of a plurality of cameras which are calibrated in advance. The image registration method based on the calibrated internal and external parameters does not need to match the characteristic points or blocks, greatly saves the time consumed by registration, and realizes the real-time registration of the images of multiple cameras. When the internal and external parameters of the multi-camera are calibrated in advance, the internal and external parameters are calibrated by collecting images of the checkerboards through the multi-camera, but the multi-camera is closer to the checkerboards during calibration, and the multi-camera is farther from a shooting object during actual operation. Therefore, when the real-time image registration is carried out, calibration errors are introduced by using internal and external parameters based on checkerboard calibration, so that the image registration errors cannot meet the requirements, and the image registration fails.
Disclosure of Invention
The application provides a multi-camera image registration method, a multi-camera image registration device and a multi-camera image registration storage medium, so that calibrated internal and external reference models simultaneously meet geometric constraint of a calibration board and space constraint of a long-distance shooting scene, calibration errors are reduced to improve the accuracy of real-time image registration, and the problem that the real-time image registration errors in the prior art cannot meet requirements is solved.
In a first aspect, the present application provides a method for image registration of multiple cameras, comprising:
acquiring a first image acquired by a first camera and a second image acquired by a second camera;
registering the first image and the second image based on a first internal reference parameter of a first camera, a second internal reference parameter of a second camera and an external reference parameter between the first camera and the second camera which are calibrated in advance; the cost function adopted when the first internal reference parameter, the second internal reference parameter and the external reference parameter are calibrated in advance comprises a regularization cost function, and the regularization cost function is constructed on the basis of an infinite homography parameter between the first camera and the second camera.
In a second aspect, the present application provides an image registration apparatus of multiple cameras, comprising:
an image acquisition module configured to acquire a first image captured by a first camera and a second image captured by a second camera;
the image registration module is configured to register the first image and the second image based on a first internal reference quantity of a first camera, a second internal reference quantity of a second camera and an external reference quantity between the first camera and the second camera which are calibrated in advance; the cost function adopted when the first internal reference parameter, the second internal reference parameter and the external reference parameter are calibrated in advance comprises a regularization cost function, and the regularization cost function is constructed on the basis of an infinite homography parameter between the first camera and the second camera.
In a third aspect, the present application provides an image registration apparatus of multiple cameras, comprising:
one or more processors; a storage device storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method for image registration of multiple cameras as described in the first aspect.
In a fourth aspect, the present application provides a storage medium containing computer executable instructions for performing the method of image registration of multiple cameras as described in the first aspect when executed by a computer processor.
The method and the device have the advantages that the regular term is added on the basis of the calibration cost function to restrict the solution space of the calibration cost function, and the internal and external reference models calculated based on the calibration cost function are prevented from fitting the close-range shooting scene. The regularization cost function is constructed through the infinite homography parameters calibrated remotely, and the solution space of the regularization cost function is constrained to tend to the infinite homography parameters through the regularization cost function, so that the internal and external parameter models calculated by the regularization cost function meet the physical model requirements of the remote shooting scene in actual operation, and the generalization capability of the internal and external parameter models is improved. Therefore, the cost function adopted when the internal and external parameters of the multi-camera are calibrated in advance comprises a calibration cost function constructed based on the calibration plate and a regularization function constructed based on the infinite homography parameter, so that the calibrated internal and external parameter models simultaneously meet the geometric constraint of the calibration plate and the spatial constraint of a long-distance shooting scene, the calibration error of the internal and external parameters of the multi-camera is reduced, the accuracy of real-time image registration is further improved, and the problem that the real-time image registration error in the prior art cannot meet the requirement is solved.
Drawings
Fig. 1 is a flowchart of an image registration method for multiple cameras provided in an embodiment of the present application;
FIG. 2 is a schematic diagram of a multispectral camera provided by an embodiment of the present application;
fig. 3 is a flowchart of pre-calibrating multiple cameras according to an embodiment of the present disclosure;
FIG. 4 is a flow chart for determining an infinity homography parameter as provided by an embodiment of the present application;
fig. 5 is a flowchart for constructing a calibration cost function according to an embodiment of the present application;
fig. 6 is a flowchart of image registration based on internal and external parameters according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an image registration apparatus with multiple cameras according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of an image registration apparatus with multiple cameras according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, specific embodiments of the present application are described in detail below with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some but not all of the relevant portions of the present application are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. A process may be terminated when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application may be practiced in sequences other than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of terms, e.g., the first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.
The multi-camera image registration method provided in this embodiment may be performed by a multi-camera image registration apparatus, where the multi-camera image registration apparatus may be implemented in a software and/or hardware manner, and the multi-camera image registration apparatus may be formed by two or more physical entities or may be formed by one physical entity. For example, the image registration device with multiple cameras may be a smart device equipped with multiple cameras, such as an unmanned device, or the like, or may be a processor of the smart device. The unmanned equipment refers to equipment such as an unmanned aerial vehicle and the like which can be automatically executed based on preset tasks. The multi-camera may be a multi-spectral camera or a dual-light camera, etc.
The image registration device of the multi-camera is provided with at least one type of operating system, the image registration device of the multi-camera can be provided with at least one application program based on the operating system, and the application program can be an application program carried by the operating system or an application program downloaded from a third party device or a server. In this embodiment, the image registration device of the multiple cameras has at least an application program that can execute the image registration method of the multiple cameras, and therefore, the image registration device of the multiple cameras may also be the application program itself.
For ease of understanding, the present embodiment is described by taking an unmanned aerial vehicle as an example of a subject that performs the image registration method of multiple cameras.
In one embodiment, the internal reference and the external reference of the multi-camera are calibrated in advance through a Zhang Yong calibration algorithm. And controlling the multiple cameras to shoot images of the surveying and mapping area during the high-altitude operation of the unmanned aerial vehicle, and registering the images based on the internal reference and the external reference of the multiple cameras which are calibrated in advance. Zhang Zhengyou calibration algorithm is to shoot calibration images on checkerboards in a short distance through multiple cameras, and a calibration cost function is constructed based on checkerboard coordinate points in multiple calibration images. And solving a calibration cost function to obtain the internal parameters and the external parameters of the multiple cameras which meet the geometric constraint of the checkerboard and the spatial constraint of the close-range shooting scene. However, when the unmanned aerial vehicle works aloft, images of a surveying and mapping area are shot remotely, and when the images shot at the high altitude are aligned by the internal parameters and the external parameters of the pre-calibrated multi-camera, a large calibration error is introduced because the spatial constraint of a long-distance shooting scene during the high altitude is not met, so that the image alignment error cannot meet the requirement, and further the image alignment failure is caused. If the mapping area is a farmland area, the registration error at the pixel level may cause a crop growth analysis error in the farmland, so that when the image registration error does not meet the requirement, the image needs to be re-registered, and the efficiency of real-time image registration is affected.
In order to solve the above problem, the present embodiment provides an image registration method for multiple cameras, so as to reduce calibration errors and improve the accuracy of image registration.
Fig. 1 is a flowchart of an image registration method for multiple cameras according to an embodiment of the present disclosure. Referring to fig. 1, the image registration method of multiple cameras specifically includes:
and S110, acquiring a first image acquired by the first camera and a second image acquired by the second camera.
The first camera is any one of the multiple cameras, and the second camera is the other cameras except the first camera. The present embodiment is described by taking a multi-camera as the multispectral camera as an example. Fig. 2 is a schematic diagram of a multispectral camera provided by an embodiment of the present application. As shown in fig. 2, the multispectral camera includes a first spectral camera 11, a second spectral camera 12, a third spectral camera 13, and a fourth spectral camera 14, the first spectral camera may be considered the first camera, and the second spectral camera, the third spectral camera, and the fourth spectral camera may be considered the second camera.
When the unmanned aerial vehicle works high above the ground, the first spectrum camera, the second spectrum camera, the third spectrum camera and the fourth spectrum camera are controlled to shoot the surveying and mapping area at the same time, and the first spectrum image, the second spectrum image, the third spectrum image and the fourth spectrum image are correspondingly obtained. The first spectral image is taken as a first image, and the second spectral image, the third spectral image and the fourth spectral image are taken as second images.
S120, registering the first image and the second image based on a first internal reference parameter of the first camera, a second internal reference parameter of the second camera and an external reference parameter between the first camera and the second camera which are calibrated in advance; the cost function adopted when the first internal reference parameter, the second internal reference parameter and the external reference parameter are calibrated in advance comprises a regularization cost function, and the regularization cost function is obtained by construction based on an infinite homography parameter between the first camera and the second camera.
In this embodiment, the first internal reference parameters are internal reference parameters of the first spectrum camera, and the second internal reference parameters include internal reference parameters of the second spectrum camera, the third spectrum camera, and the fourth spectrum camera. The external parameters comprise external parameters between the first spectrum camera and the second spectrum camera, external parameters between the first spectrum camera and the third spectrum camera, and external parameters between the first spectrum camera and the fourth spectrum camera.
When the first spectral image and the second spectral image are registered, the second spectral image is converted into a pixel space of the first spectral image through the pre-calibrated internal reference parameter of the first spectral camera, the pre-calibrated internal reference parameter of the second spectral camera and the pre-calibrated first external reference parameter, so that the first spectral image and the second spectral image are registered. Similarly, the registration process of the first spectral image and the third spectral image, and the registration process of the first spectral image and the fourth spectral image are the same as above.
The regularization cost function is a regularization term added in the calibration cost function, and the calibration cost function is constructed when the calibration board is used for internal reference and external reference calibration, for example, the calibration cost function in the embodiment can be constructed based on a Zhang Zhengyou calibration algorithm. Since the internal and external parameter models solved by the calibration cost function satisfy the geometric constraint of the calibration board and the spatial constraint of the close-range shooting scene, the embodiment proposes to calibrate the solution space of the cost function by the regular term constraint, so as to prevent the internal and external parameter models solved by the calibration cost function from fitting the close-range shooting scene.
In this embodiment, when the multispectral camera performs aerial work to shoot the image of the mapping region, since the baseline of the multispectral camera is short, the distance between the mapping region and the multispectral camera exceeds the measurable depth distance of the multispectral camera, and the multispectral camera cannot know the depth of the image, the mapping region in the image is regarded as an infinite plane. The homography parameter calibrated based on the image containing the infinite plane, that is, the image remotely shot by the multispectral camera, is the infinite homography parameter in the embodiment. Wherein, the homography parameter refers to a relative transformation matrix of pixel spaces of the two cameras, and the expression of the homography parameter is H = K 1 R 2,1 K 2 -1 . H is a homography parameter, K 1 Is an internal reference parameter of the first camera, K 2 And R is an external reference rotation parameter between the first camera and the second camera. In conclusion, the internal parameters and the external parameters of the multispectral camera in the homography parameters at infinity meet the space constraint of a long-distance shooting scene when the multispectral camera works at high altitude. In contrast, in the embodiment, the regular term added by the calibration cost function is constructed based on the infinite homography parameter to constrain the solution space of the calibration cost function to tend to the infinite homography parameter, so that the internal and external parameter models calculated by the calibration cost function meet the physical model requirement of the long-distance shooting scene in actual operation, and the generalization capability of the internal and external parameter models is improved.
It should be noted that, because the infinity homography parameter only satisfies the spatial constraint of the long-distance shooting scene but does not satisfy the geometric constraint of the calibration board, and the calibration error of the infinity homography parameter is greater than the calibration error of the calibration cost function, the infinity homography parameter is suitable for being used as the regular term of the calibration cost function to constrain the solution space thereof, and is not suitable for being used as the dominant cost function.
In one embodiment, fig. 3 is a flowchart illustrating pre-calibration of a multi-spectral camera according to an embodiment of the present disclosure. As shown in fig. 3, the step of performing the pre-selection calibration on the multispectral camera specifically includes steps S210 to S270:
s210, a first calibration image and a second calibration image are obtained, wherein the first calibration image is obtained by shooting at a preset height through a first camera, and the second calibration image is obtained by shooting at a preset height through a second camera.
In this embodiment, the preset height is greater than the range depth of the multispectral camera, which is typically set to 100 meters. Illustratively, in the pre-calibration stage of the multispectral camera, the unmanned aerial vehicle can sail at an altitude 100 meters away from the ground, and the first spectral camera, the second spectral camera, the third spectral camera and the fourth spectral camera are controlled to continuously shoot down the ground, so as to correspondingly obtain a first spectral image sequence, a second spectral image sequence, a third spectral image sequence and a fourth spectral image sequence. The spectral image sequence comprises a plurality of continuously shot calibration spectral images, the first calibration spectral image is a first calibration image, and the second calibration spectral image, the third calibration spectral image and the fourth calibration spectral image are second calibration images.
S220, determining an infinite homography parameter according to the first calibration image and the second calibration image.
Illustratively, the infinity homography parameter between the first spectral camera and the second spectral camera may be determined based on a plurality of first calibration spectral images in the first spectral image sequence and a plurality of second calibration spectral images in the second spectral image sequence. Similarly, the determination process of the infinity homography parameter between the first spectral camera and the third spectral camera and the determination process of the infinity homography parameter between the first spectral camera and the fourth spectral camera are the same as above.
In one embodiment, fig. 4 is a flowchart for determining an infinity homography parameter provided in an embodiment of the present application. As shown in fig. 4, the step of determining the infinity homography parameter specifically includes S2201-S2203:
s2201, extracting a first characteristic point in the first calibration image and a second characteristic point in the second calibration image.
The present embodiment is described taking the example of determining an infinity homography parameter between a first spectral camera and a second spectral camera. And extracting a first Feature point and a second Feature point from the first calibration spectrum image and the second calibration spectrum image respectively based on a Scale Invariant Feature Transform (SIFI) Feature extraction algorithm.
And S2202, matching the first characteristic points with the second characteristic points to determine characteristic matching pairs.
Illustratively, a first calibration spectral image and a second calibration spectral image that are simultaneously captured in the first spectral image sequence and the second spectral image sequence are determined. And matching the first characteristic points of the first calibration spectrum image with the second characteristic points of the second calibration spectrum image shot at the same time to obtain a plurality of groups of characteristic matching pairs.
S2203, determining an infinite homography parameter based on the feature matching pair.
For example, the first calibration spectral image may be regarded as an image obtained by converting the second calibration spectral image captured at the same time by the infinity homography parameter. A cost function during calibration of the homography parameter at infinity can be constructed based on the feature matching pairs. And solving the optimal solution of the cost function to obtain an infinite homography parameter. It should be noted that the more feature matching pairs, the smaller the solution space of the cost function when calibrating the infinity homography parameter, and the smaller the error of the computed infinity homography parameter, so that the sufficient feature matching pairs can be improved by the multi-frame calibration spectral images in the spectral image sequence of the multi-spectral camera.
In another embodiment, feature points in the first calibration image and the second calibration image are extracted through the deep learning model, and the feature points extracted through the deep learning model are matched to obtain a feature matching pair. And constructing a cost function when the infinite homography parameters are calibrated through the characteristic matching pair, and resolving the optimal solution of the cost function to obtain the infinite homography parameters. Because the convolutional neural network in the deep learning model can extract more complex image features, the feature point extraction based on the deep learning model is more accurate. In another embodiment, the infinity homography parameter is learned by the deep learning model to align the first calibration image and the second calibration image, that is, the first calibration image and the second calibration image are input into the deep learning model to obtain the infinity homography parameter output by the deep learning model.
And S230, constructing a regularization cost function based on the infinite homography parameter.
Illustratively, the expression for the regularizing cost function when calibrating the first spectral camera and the second spectral camera is as follows:
wherein f is λ (K 1 ,R 2,1 ,K 2 ) Is a regularization cost function; λ is a regularization parameter, the regularization parameter is used for determining the degree of constraint of the regularization term on the calibration cost function, and an empirical value is generally taken as 0.1; k 1 Is an internal parameter, K, of the first spectral camera 2 Is an internal parameter, R, of the second spectral camera 2,1 Is an extrinsic rotation parameter between the first spectral camera and the second spectral camera,
is an infinite homography parameter between the first spectral camera and the second spectral camera.
Similarly, a regularization cost function of the first spectral camera and the third spectral camera in calibration and a regularization cost function of the first spectral camera and the fourth spectral camera in calibration can be obtained.
And S240, acquiring a third calibration image and a fourth calibration image, wherein the third calibration image is obtained by shooting the preset calibration plate by the first camera, and the fourth calibration image is obtained by shooting the preset calibration plate by the second camera.
The present embodiment describes an example of constructing a calibration cost function based on the Zhang Zhengyou calibration algorithm. In the pre-calibration stage of the multispectral camera, a calibration plate is arranged in the depth ranging range of the multispectral camera, and the first spectral camera, the second spectral camera, the third spectral camera and the fourth spectral camera are controlled to shoot images opposite to the checkerboard, so that a fifth calibration spectral image, a sixth calibration spectral image, a seventh calibration spectral image and an eighth calibration spectral image which comprise checkerboard pictures are correspondingly obtained. The fifth calibration spectral image is a third calibration image, and the sixth calibration spectral image, the seventh calibration spectral image and the eighth calibration spectral image are fourth calibration images.
And S250, constructing a calibration cost function according to the third calibration image and the fourth calibration image.
Illustratively, the calibration cost functions of the first spectral camera and the second spectral camera are constructed based on the fifth calibration spectral image and the sixth calibration spectral image. Similarly, the calibration cost functions of the first spectral camera and the third spectral camera and the calibration cost functions of the first spectral camera and the fourth spectral camera are constructed in the same way.
The present embodiment is described by taking an example of constructing the calibration cost function of the first spectral camera and the second spectral camera. Fig. 5 is a flowchart for constructing a calibration cost function according to an embodiment of the present application. As shown in fig. 5, the step of constructing the calibration cost function specifically includes steps S2501 to S2502:
s2501, constructing a mapping relation between a feature point of a calibration plate in a third calibration image and a feature point of the same calibration plate in a fourth calibration image.
And the characteristic point of the calibration plate is the intersection point of black and white grids in the image. For example, feature point extraction is performed on the fifth calibration spectrum image and the sixth calibration spectrum image, and black and white lattice intersections in the fifth calibration spectrum image and black and white lattice intersections in the sixth calibration spectrum image can be obtained. And matching the black-white grid intersection points in the fifth calibration spectrum image with the black-white grid intersection points in the sixth calibration spectrum image, and determining the black-white grid intersection points at the same position corresponding to the checkerboards in the two images.
Based on the Zhang Zhengyou calibration algorithm, a mapping relation between the intersection point of the black and white grids of the fifth calibration spectrum image after the distortion correction and the intersection point of the black and white grids at the same position of the checkerboards in the sixth calibration spectrum image after the distortion correction is established. The mapping relationship includes the internal parameters of the first spectral camera, the internal parameters of the second spectral camera, and the first external parameters.
S2502, a calibration cost function is constructed according to the mapping relation and the characteristic points of the calibration plates in the third calibration image and the fourth calibration image.
Illustratively, the expressions for the calibrated cost functions of the first spectral camera and the second spectral camera are as follows:
wherein f is d (K 1 ,R 2,1 ,K 2 ) For the calibration of the cost function, M is the total number of black and white intersections on the checkerboard, (u) 1,j ,v 1,j ) The distortion-corrected pixel coordinate of the jth black-and-white intersection point on the checkerboard in the fifth calibration spectrum image is (u) 2,j ,v 2,j ) For the pixel coordinate of the jth black-and-white intersection point on the checkerboard in the sixth calibration spectral image after distortion correction,
the mapping relation of the intersection point of the black and white grids of the fifth calibration spectrum image after distortion correction and the intersection point of the black and white grids at the same position of the checkerboard in the sixth calibration spectrum image after distortion correction is T 2,1 Is an external reference translation parameter between the first spectral camera and the second spectral camera.
And S260, generating a cost function according to the calibration cost function and the regularization cost function.
Illustratively, the expression of the cost function pre-calibrated by the first spectral camera and the second spectral camera is as follows:
wherein f is 2,1 (K 1 ,R 2,1 ,K 2 ) A cost function pre-calibrated for the first spectral camera and the second spectral camera.
And S270, iteratively solving the optimal solution of the cost function based on a preset algorithm to obtain a first internal parameter, a second internal parameter and an external parameter.
Illustratively, an optimal solution of the cost function calibrated in advance by the first spectrum camera and the second spectrum camera is solved to obtain the internal reference parameter of the first spectrum camera, the internal reference parameter of the second spectrum camera and the external reference rotation parameter between the first spectrum camera and the second spectrum camera. Similarly, the determination process of the internal reference parameter of the third spectral camera, the external reference rotation parameters of the first spectral camera and the third spectral camera, and the determination process of the internal reference parameter of the fourth spectral camera and the external reference rotation parameters of the first spectral camera and the fourth spectral camera are the same as above.
In this embodiment, the optimal solution of the cost function may be iteratively calculated based on a newton gaussian algorithm, or may be calculated by a Levenberg-Marquardt (Levenberg-Marquardt) algorithm.
After the pre-calibration of the multispectral camera is completed, the calibrated internal and external parameters can be applied to the real-time image registration of the actual operation. The present embodiment is described by taking a registration process of the first spectral image and the second spectral image as an example. Fig. 6 is a flowchart of image registration based on internal and external parameters according to an embodiment of the present application. As shown in fig. 6, the step of image registration based on the internal and external parameters specifically includes S1201-S1202:
s1201, determining a calibration homography parameter between the first camera and the second camera according to the first internal reference parameter, the external reference parameter and the second internal reference parameter.
Illustratively, according to the expression of the homography parameter, the calibrated homography parameter between the first spectrum camera and the second spectrum camera can be determined through the internal parameter of the first spectrum camera, the internal parameter of the second spectrum camera and the external parameter rotation parameter between the first spectrum camera and the second spectrum camera.
And S1202, converting the second image based on the calibrated homography parameter to obtain a second image which is registered with the first image.
Illustratively, the second spectral image is converted to the pixel space of the first spectral image by a calibrated homography parameter between the first spectral camera and the second spectral camera to achieve registration of the first spectral image and the second spectral image.
In summary, the image registration method for multiple cameras provided by the embodiment of the application restricts the solution space of the calibration cost function by adding the regular term on the basis of the calibration cost function, and prevents the internal and external reference models calculated based on the calibration cost function from overfitting the close-range shooting scene. The regularization cost function is constructed through the infinite homography parameters calibrated remotely, and the solution space of the regularization cost function is constrained to tend to the infinite homography parameters through the regularization cost function, so that the internal and external parameter models calculated by the regularization cost function meet the physical model requirements of the remote shooting scene in actual operation, and the generalization capability of the internal and external parameter models is improved. Therefore, the cost function adopted when the internal and external parameters of the multi-camera are calibrated in advance comprises a calibration cost function constructed based on the calibration plate and a regularization function constructed based on the infinite homography parameter, so that the calibrated internal and external parameter models simultaneously meet the geometric constraint of the calibration plate and the spatial constraint of a long-distance shooting scene, the calibration error of the internal and external parameters of the multi-camera is reduced, the accuracy of real-time image registration is further improved, and the problem that the real-time image registration error in the prior art cannot meet the requirement is solved.
On the basis of the foregoing embodiments, fig. 7 is a schematic structural diagram of an image registration apparatus with multiple cameras according to an embodiment of the present application. Referring to fig. 7, the image registration apparatus for multiple cameras provided in this embodiment specifically includes: an image acquisition module 31 and an image registration module 32.
The image acquisition module is configured to acquire a first image acquired by a first camera and a second image acquired by a second camera;
the image registration module is configured to register the first image and the second image based on a first internal reference quantity of the first camera, a second internal reference quantity of the second camera and an external reference quantity between the first camera and the second camera which are calibrated in advance; the cost function adopted when the first internal reference parameter, the second internal reference parameter and the external reference parameter are calibrated in advance comprises a regularization cost function, and the regularization cost function is obtained by construction based on an infinite homography parameter between the first camera and the second camera.
On the basis of the above embodiment, the image registration apparatus for multiple cameras further includes: the device comprises a first calibration image acquisition module and a second calibration image acquisition module, wherein the first calibration image acquisition module is configured to acquire a first calibration image and a second calibration image, the first calibration image is shot by a first camera at a preset height, and the second calibration image is shot by a second camera at the preset height; the infinity homography parameter determining module is configured to determine an infinity homography parameter according to the first calibration image and the second calibration image; and the regularization cost function construction module is configured to construct a regularization cost function based on the infinity homography parameter.
On the basis of the above embodiment, the infinity homography parameter determination module includes: a feature point extraction unit configured to extract a first feature point in the first calibration image and a second feature point in the second calibration image; the characteristic matching unit is configured to match the first characteristic points with the second characteristic points and determine characteristic matching pairs; an infinity homography parameter determination unit configured to determine an infinity homography parameter based on the feature matching pair.
On the basis of the above embodiment, the image registration apparatus for multiple cameras further includes: the second calibration image acquisition module is configured to acquire a third calibration image and a fourth calibration image, wherein the third calibration image is obtained by shooting the preset calibration plate by the first camera, and the fourth calibration image is obtained by shooting the preset calibration plate by the second camera; the calibration cost function construction module is configured to construct a calibration cost function according to the third calibration image and the fourth calibration image; and the cost function generation module is configured to generate a cost function according to the calibration cost function and the regularization cost function.
On the basis of the above embodiment, the calibration cost function construction module includes: the mapping relation construction unit is configured to construct a mapping relation between the feature point of the calibration plate in the third calibration image and the feature point of the same calibration plate in the fourth calibration image; and the calibration cost function construction unit is configured to construct a calibration cost function according to the mapping relation, the third calibration image and each calibration plate feature point in the fourth calibration image.
On the basis of the above embodiment, the image registration apparatus for multiple cameras further includes: and the cost function calculating module is configured to iteratively calculate the optimal solution of the cost function based on a preset algorithm to obtain a first internal parameter, a second internal parameter and an external parameter.
On the basis of the above embodiment, the image registration module includes: a calibrated homography parameter unit configured to determine a calibrated homography parameter between the first camera and the second camera according to the first internal reference parameter, the external reference parameter and the second internal reference parameter; and the image registration unit is configured to convert the second image based on the calibrated homography parameter to obtain the second image which is registered with the first image.
In the foregoing, the image registration apparatus for multiple cameras provided in the embodiment of the present application restricts a solution space of the calibration cost function by adding the regular term on the basis of the calibration cost function, and prevents the internal and external reference models calculated based on the calibration cost function from overfitting the close-range shooting scene. The regularized cost function is constructed through the infinite homography parameters calibrated remotely, and the solution space of the regularized cost function is constrained to tend to the infinite homography parameters through the regularized cost function, so that the internal and external parameter models calculated by the regularized cost function meet the physical model requirements of a remote shooting scene during actual operation, and the generalization capability of the internal and external parameter models is improved. Therefore, the cost function adopted when the internal and external parameters of the multi-camera are calibrated in advance comprises a calibration cost function constructed based on the calibration plate and a regularization function constructed based on the infinite homography parameter, so that the calibrated internal and external parameter models simultaneously meet the geometric constraint of the calibration plate and the spatial constraint of a long-distance shooting scene, the calibration error of the internal and external parameters of the multi-camera is reduced, the accuracy of real-time image registration is further improved, and the problem that the real-time image registration error in the prior art cannot meet the requirement is solved.
The multi-camera image registration device provided by the embodiment of the application can be used for executing the multi-camera image registration method provided by the embodiment, and has corresponding functions and beneficial effects.
Fig. 8 is a schematic structural diagram of an image registration apparatus with multiple cameras provided in an embodiment of the present application, and referring to fig. 8, the image registration apparatus with multiple cameras includes: processor 41, memory 42, communication device 43, input device 44, and output device 45. The number of processors 41 in the image registration apparatus of the multi-camera may be one or more, and the number of memories 42 in the image registration apparatus of the multi-camera may be one or more. The processor 41, the memory 42, the communication means 43, the input means 44 and the output means 45 of the multi-camera image registration apparatus may be connected by a bus or other means.
The memory 42 serves as a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the image registration method of the multi-camera according to any embodiment of the present application (for example, the image acquisition module 31 and the image registration module 32 in the image registration apparatus of the multi-camera). The memory 42 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the device, and the like. Further, the memory 42 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory may further include memory located remotely from the processor, and these remote memories may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The communication means 43 is used for data transmission.
The processor 41 executes various functional applications of the device and data processing, i.e. implements the above-described image registration method of the multi-camera, by running software programs, instructions and modules stored in the memory 42.
The input device 44 is operable to receive input numeric or character information and to generate key signal inputs relating to user settings and function controls of the apparatus. The output device 45 may include a display device such as a display screen.
The multi-camera image registration apparatus provided above can be used to perform the multi-camera image registration method provided in the above embodiments, and has corresponding functions and advantages.
Embodiments of the present application also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform a method of image registration for multiple cameras, the method of image registration for multiple cameras comprising: acquiring a first image acquired by a first camera and a second image acquired by a second camera; registering the first image and the second image based on a first internal reference parameter of the first camera, a second internal reference parameter of the second camera and an external reference parameter between the first camera and the second camera which are calibrated in advance; the cost function adopted when the first internal reference parameter, the second internal reference parameter and the external reference parameter are calibrated in advance comprises a regularization cost function, and the regularization cost function is constructed on the basis of an infinite homography parameter between the first camera and the second camera.
Storage medium-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations, e.g., in different computer systems connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.
Of course, the storage medium provided in the embodiments of the present application contains computer executable instructions, and the computer executable instructions are not limited to the image registration method of multiple cameras as above, and may also perform related operations in the image registration method of multiple cameras provided in any embodiments of the present application.
The multi-camera image registration apparatus, the storage medium, and the multi-camera image registration device provided in the above embodiments may perform the multi-camera image registration method provided in any embodiment of the present application, and reference may be made to the multi-camera image registration method provided in any embodiment of the present application without detailed technical details described in the above embodiments.
The foregoing is considered as illustrative of the preferred embodiments of the invention and the technical principles employed. The present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the claims.
Claims (10)
1. A method of image registration of multiple cameras, comprising:
acquiring a first image acquired by a first camera and a second image acquired by a second camera;
registering the first image and the second image based on a first internal reference parameter of a first camera, a second internal reference parameter of a second camera and an external reference parameter between the first camera and the second camera which are calibrated in advance; the cost function adopted when the first internal reference parameter, the second internal reference parameter and the external reference parameter are calibrated in advance comprises a regularization cost function, and the regularization cost function is constructed on the basis of an infinite homography parameter between the first camera and the second camera.
2. The method of image registration of multiple cameras of claim 1, further comprising, prior to said registering the first image and the second image:
acquiring a first calibration image and a second calibration image, wherein the first calibration image is obtained by shooting at a preset height by the first camera, and the second calibration image is obtained by shooting at the preset height by the second camera;
determining the infinity homography parameter according to the first calibration image and the second calibration image;
and constructing the regularization cost function based on the infinite homography parameter.
3. The method of image registration of multiple cameras of claim 2, wherein said determining the infinity homography parameter from the first calibration image and the second calibration image comprises:
extracting a first characteristic point in the first calibration image and a second characteristic point in the second calibration image;
matching the first characteristic point with the second characteristic point to determine a characteristic matching pair;
determining the infinity homography parameter based on the feature matching pairs.
4. The multi-camera image registration method according to any one of claims 1-3, further comprising, before said registering the first image and the second image:
acquiring a third calibration image and a fourth calibration image, wherein the third calibration image is obtained by shooting a preset calibration plate by the first camera, and the fourth calibration image is obtained by shooting the preset calibration plate by the second camera;
constructing a calibration cost function according to the third calibration image and the fourth calibration image;
and generating the cost function according to the calibration cost function and the regularization cost function.
5. The multi-camera image registration method of claim 4, wherein the constructing a calibration cost function from the third calibration image and the fourth calibration image comprises:
constructing a mapping relation between the characteristic point of the calibration plate in the third calibration image and the characteristic point of the same calibration plate in the fourth calibration image;
and constructing the calibration cost function according to the mapping relation, the third calibration image and the fourth calibration image.
6. The method of image registration of multiple cameras of claim 1, further comprising, prior to said registering the first image and the second image:
iteratively solving the optimal solution of the cost function based on a preset algorithm to obtain the first internal parameter, the second internal parameter and the external parameter.
7. The method of claim 1, wherein the registering the first image and the second image based on the pre-calibrated first internal reference of the first camera, the second internal reference of the second camera and the external reference between the first camera and the second camera comprises:
determining a calibration homography parameter between the first camera and the second camera according to the first internal reference parameter, the external reference parameter and the second internal reference parameter;
and converting the second image based on the calibrated homography parameter to obtain a second image which is registered with the first image.
8. An image registration apparatus of a plurality of cameras, comprising:
an image acquisition module configured to acquire a first image captured by a first camera and a second image captured by a second camera;
the image registration module is configured to register the first image and the second image based on a first internal reference quantity of a first camera, a second internal reference quantity of a second camera and an external reference quantity between the first camera and the second camera which are calibrated in advance; the cost function adopted when the first internal reference parameter, the second internal reference parameter and the external reference parameter are calibrated in advance comprises a regularization cost function, and the regularization cost function is constructed on the basis of an infinite homography parameter between the first camera and the second camera.
9. An image registration apparatus of multiple cameras, characterized by comprising: one or more processors; a storage device storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the multi-camera image registration method of any of claims 1-7.
10. A storage medium containing computer executable instructions for performing the method for image registration of multiple cameras of any one of claims 1-7 when executed by a computer processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210946257.0A CN115311336A (en) | 2022-08-08 | 2022-08-08 | Image registration method, device and equipment of multiple cameras and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210946257.0A CN115311336A (en) | 2022-08-08 | 2022-08-08 | Image registration method, device and equipment of multiple cameras and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115311336A true CN115311336A (en) | 2022-11-08 |
Family
ID=83861722
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210946257.0A Pending CN115311336A (en) | 2022-08-08 | 2022-08-08 | Image registration method, device and equipment of multiple cameras and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115311336A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115797468A (en) * | 2023-02-03 | 2023-03-14 | 厦门农芯数字科技有限公司 | Automatic correction method, device and equipment for mounting height of fisheye camera |
-
2022
- 2022-08-08 CN CN202210946257.0A patent/CN115311336A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115797468A (en) * | 2023-02-03 | 2023-03-14 | 厦门农芯数字科技有限公司 | Automatic correction method, device and equipment for mounting height of fisheye camera |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112367474B (en) | Self-adaptive light field imaging method, device and equipment | |
| CN108759788B (en) | Unmanned aerial vehicle image positioning and attitude determining method and unmanned aerial vehicle | |
| CN112419374A (en) | Unmanned aerial vehicle positioning method based on image registration | |
| Zheng et al. | Minimal solvers for 3d geometry from satellite imagery | |
| CN112946679B (en) | Unmanned aerial vehicle mapping jelly effect detection method and system based on artificial intelligence | |
| CN113192646A (en) | Target detection model construction method and different target distance monitoring method and device | |
| CN107274441A (en) | The wave band calibration method and system of a kind of high spectrum image | |
| CN114529615B (en) | Radar calibration method, device and storage medium | |
| CN115311336A (en) | Image registration method, device and equipment of multiple cameras and storage medium | |
| WO2021051382A1 (en) | White balance processing method and device, and mobile platform and camera | |
| CN111444923A (en) | Image semantic segmentation method and device under natural scene | |
| CN115018920A (en) | Camera array calibration method and device, electronic equipment and storage medium | |
| CN111325828A (en) | Three-dimensional face acquisition method and device based on three-eye camera | |
| CN116817887B (en) | Semantic visual SLAM map construction method, electronic equipment and storage medium | |
| CN115423863B (en) | Camera pose estimation method and device and computer readable storage medium | |
| CN116597246A (en) | Model training method, target detection method, electronic device and storage medium | |
| CN109919998B (en) | Satellite attitude determination method and device and terminal equipment | |
| CN115272462A (en) | Camera pose estimation method and device and electronic equipment | |
| CN113496505B (en) | Image registration method and device, multispectral camera, unmanned equipment and storage medium | |
| CN114897990A (en) | Camera distortion calibration method and system based on neural network and storage medium | |
| CN114387353A (en) | Camera calibration method, calibration device and computer readable storage medium | |
| CN114966576A (en) | Radar external reference calibration method and device based on prior map and computer equipment | |
| CN110689513A (en) | Color image fusion method and device and terminal equipment | |
| CN115100535B (en) | Satellite remote sensing image rapid reconstruction method and device based on affine camera model | |
| CN115222776B (en) | Matching auxiliary visual target tracking method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |