WO2022120567A1

WO2022120567A1 - Automatic calibration system based on visual guidance

Info

Publication number: WO2022120567A1
Application number: PCT/CN2020/134521
Authority: WO
Inventors: 程俊; 张能波; 郭海光
Original assignee: 深圳先进技术研究院
Priority date: 2020-12-08
Filing date: 2020-12-08
Publication date: 2022-06-16

Abstract

Disclosed in the present invention are an automatic calibration system and method based on visual guidance. The system comprises: a calibration vehicle, a rotation control gimbal, a multi-face calibration target, a plurality of cameras and a data acquisition control module. The multi-face calibration target is connected to the rotation control gimbal, and the data acquisition control module is used for controlling the calibration vehicle to carry the multi-face calibration target to move and controlling corresponding cameras to photograph the multi-face calibration target when the multi-face calibration target is moved to an overlapped field of view of two adjacent cameras so as to obtain image data of the multi-face calibration target. The data acquisition control module stores, according to camera numbers and the position number of the multi-face calibration target, the image data obtained by means of photography, and detects corner points of the multi-face calibration target to obtain relative poses between the cameras. By means of the present invention, the automation of the whole calibration process can be achieved, the problem of there being no overlapped field of view between cameras can be solved, the operation is simple and convenient, and a relatively high calibration precision is also achieved.

Description

An automatic calibration system based on vision guidance

technical field

The invention relates to the technical field of graphics and image processing, and more particularly, to an automatic calibration system based on vision guidance.

Background technique

Facing the trend of more flexible industrial production lines and more complex and changeable processing tasks, multi-robot collaborative systems are developing rapidly. In the multi-robot collaboration problem, the multi-camera joint calibration technique is used to solve the relative pose relationship (R, T) between the multi-robots.

At present, camera calibration technology can be roughly divided into two categories:

The first category: use a specially made calibration board to determine the camera parameters. Commonly used traditional camera calibration methods include: Faugeras calibration method, Tsai two-step method, and Zhang Zhengyou plane calibration method. The linear model camera calibration of the Faugeras calibration method is based on the least squares problem of a system of linear equations. The Tsai calibration method needs to obtain some parameter values in advance, first solve some parameters by linear method, and then solve the remaining camera parameters by nonlinear optimization. The Zhang Zhengyou calibration method utilizes multiple images of the plane calibration plate at different viewing angles, and calibrates the camera parameters according to the designed homography matrix.

The second category: self-calibration method, that is, calibration is performed according to the corresponding relationship between the two images generated during the camera movement. Such as self-calibration method based on infinite plane, absolute quadratic surface, self-calibration method based on Kruppa equation, etc. The self-calibration method does not depend on the calibration reference, and has high flexibility, but the constraints are strong, and the calibration accuracy is low and the robustness is insufficient.

After analysis, although the traditional camera calibration method has high accuracy and simple operation, it has high requirements on the accuracy of the calibration object. Furthermore, for multi-camera calibration, current schemes generally require overlapping fields of view between cameras, or in a scene with a known structure. This does not meet the complex working conditions, where the calibration scene is complex and there is no overlapping field of view between cameras.

For example, patent application CN110689585A discloses a joint calibration method, device, device and medium for multi-camera external parameters. The specific implementation scheme is as follows: determine the common viewing area of each camera, and obtain a 2D verification point set in the image of the common viewing area; perform external parameter calibration for each camera respectively; The 3D coordinates of the points are verified, and the loss function is calculated according to the 3D coordinates; the joint calibration is performed according to the loss function to obtain the final external parameters of each camera.

For another example, the patent application CN110766759A discloses a multi-camera calibration method and device without overlapping fields of view. The specific implementation scheme is: place the calibration board in the field of view of each camera; calculate the pose relationship between each camera coordinate system and the calibration board; use the dual theodolite three-dimensional coordinate measurement system to measure any n points on each calibration board respectively The three-dimensional coordinates under the theodolite coordinates; then use the 3D-3D pose estimation iterative closest point method to solve the pose relationship between the two calibration boards; use the camera to measure the pose relationship between the camera and the calibration board, and the theodolite calibration The pose relationship between the two calibration plates is calculated to calculate the relationship between the two cameras.

In summary, multi-camera joint calibration is a technology that uses computer vision technology to obtain camera internal parameters and relative poses between multiple cameras. This technology has been widely used in multi-robot collaborative systems, but the existing solutions are often limited to specific scenarios, complex calibration processes, time-consuming and labor-intensive; and can not well meet the situation where there is no overlapping field of view between cameras.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the above-mentioned defects of the prior art, and to provide an automatic calibration system based on vision guidance. For the joint calibration of multiple cameras, there is no need for overlapping fields of view between all cameras, and the automation of the calibration process is realized.

According to a first aspect of the present invention, an automated calibration system based on vision guidance is provided. The system includes: a calibration vehicle, a rotary control pan-tilt, a multi-faceted calibration board, multiple cameras, and a data acquisition control module, wherein the multi-sided calibration board is connected to the rotary control pan-tilt, and the data acquisition control module is used to control the calibration vehicle to be loaded with the multi-sided calibration board Move, and control the corresponding camera to shoot the multi-faceted calibration plate when moving to the overlapping field of view of the adjacent two cameras, and obtain the image data of the multi-faceted calibration plate; the data acquisition control module will take the obtained image data according to the camera number and multi-faceted calibration. The board position number is stored, and the corner points of the multi-faceted calibration board are detected to obtain the relative pose between cameras.

According to a second aspect of the present invention, an automated calibration method based on vision guidance is provided. The method includes the following steps:

Control the calibration vehicle to move with the multi-faceted calibration plate, and control the corresponding camera to shoot the multi-faceted calibration plate when moving to the overlapping field of view of two adjacent cameras, and obtain the image data of the multi-faceted calibration plate;

The image data obtained by shooting is stored according to the camera number and the position number of the multi-faceted calibration plate, and the corner points of the multi-faceted calibration plate are detected to obtain the relative posture between the cameras.

Compared with the prior art, the present invention has the advantages that it can solve the problem of no overlapping field of view between cameras, realize the automation of the entire calibration process, and obtain high calibration accuracy while the calibration process is convenient and fast.

Other features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

1 is a schematic diagram of an automated calibration system based on vision guidance according to an embodiment of the present invention;

2 is a schematic diagram of a calibration trolley according to an embodiment of the present invention;

3 is a schematic diagram of a five-sided three-dimensional checkerboard calibration plate according to an embodiment of the present invention;

4 is a flowchart of data collection and acquisition according to an embodiment of the present invention;

5 is a schematic diagram of solving relative poses between cameras with overlapping fields of view according to an embodiment of the present invention;

6 is a schematic diagram of solving the relative pose between cameras without overlapping fields of view according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of an image partition according to an embodiment of the present invention.

Detailed ways

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

In all examples shown and discussed herein, any specific values should be construed as illustrative only and not limiting. Accordingly, other instances of the exemplary embodiment may have different values.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

In the present invention, an automated calibration system based on vision guidance is proposed, which realizes the automation of multi-camera joint calibration. Referring to FIG. 1 , the system as a whole includes: a calibration vehicle 10 (taking a four-wheeled trolley as an example), Rotation control pan/tilt 20, multi-faceted calibration plate 30, multiple cameras (shown as 3 cameras, high-resolution industrial cameras can be used) and data acquisition control module (not shown), wherein the multi-faceted calibration plate 30 is connected to the rotation control cloud The stage 20, the data acquisition control module is used to control the calibration vehicle 10 to move with the multi-faceted calibration plate 30, and control the corresponding camera to photograph the multi-faceted calibration plate when moving to the overlapping field of view of the adjacent two cameras, and obtain the multi-faceted calibration plate. Image data; the data acquisition control module stores the image data obtained by shooting according to the camera number and the position number of the multi-faceted calibration plate, and detects the corners of the multi-faceted calibration plate to obtain the relative posture between the cameras.

In short, the calibration process of the provided automatic calibration system mainly includes: step S1, collecting and obtaining a calibration image; step S2, solving the initial relative posture between cameras; Attitude calculation re-projection error; step S4, the attitude error is established on the global data and optimized, and the final relative attitude between cameras is obtained. Each link of the calibration process will be described in detail below.

Step S1, collecting and acquiring calibration images

With reference to Figure 1, first use the intelligent car and automatic data acquisition software (or data acquisition control module) to collect image data, and then use Zhang Zhengyou's calibration method to calculate the initialized relative attitude.

In one embodiment, as shown in FIG. 2 , the calibration vehicle is a four-wheel smart car, and the four-wheel smart car carries a multi-faceted calibration board and a cloud control platform, wherein the multi-faceted calibration board is connected to the rotary control pan-tilt, and the signal transceiver device is also set. Wireless WiFi, which can send commands and control the PTZ to the automatic trolley through wireless devices, so that it can perform multi-angle and multi-face rotation on the calibration board.

For example, as shown in FIG. 3 , the calibration plate is a five-sided three-dimensional checkerboard calibration plate. Each side is a 7×10 checkerboard calibration plate, and the angle between the four

calibration plates

②, ③, ④, ⑤ and the calibration plate ① is 45°, so that cameras in different orientations can detect more corners information. The calibration board is followed by an intelligent pan-tilt with a changeable angle, which is fixed on the intelligent trolley for movement.

When the calibration plate moves to the overlapping field of view between two adjacent cameras, the camera captures the corner data of five faces at a time, and then uses the common corner data of the two cameras to calculate the relative pose. Thus, more data can be obtained in less time for later optimization.

The data acquisition software can integrate multiple programs such as smart car control, camera shooting control, and data storage and visualization.

The process of collecting and obtaining the calibration image is shown in Figure 4, which includes: the data acquisition software controls the intelligent car to move with the calibration plate loaded. When the calibration plate moves to the overlapping field of view of two adjacent cameras (such as camera 1 and camera 2) Control the smart car to stop moving and send a shooting command to the camera; the camera shoots the calibration board to obtain the image data of the calibration board; the data acquisition software stores the captured pictures according to the camera number and the position number of the calibration board, and uses The visualization program detects and visualizes the corners of the calibration board; controls the trolley to continue moving with the calibration board (for example, the calibration board moves to the overlapping field of view of camera 2 and camera 3), and repeats the above steps until the data collection is completed.

Step S2, solve the initial relative pose between cameras

For two cameras in multiple cameras, there are two cases of overlapping field of view and no overlapping field of view.

(a) Solving the relative pose between cameras with overlapping fields of view

According to the calibration plate data captured by the camera in step S1, for example, the internal parameter matrix K of each camera and the single-camera rotation and translation matrix (R, t) between each camera and the calibration plate at different positions are obtained by Zhang Zhengyou's calibration method.

For two adjacent cameras with overlapping fields of view, as shown in Figure 5, the rotation and translation matrix of a single camera is used to calculate the relative poses between cameras with overlapping fields of view, which are expressed as:

R ₁₂ =R ₁ ^-1 ·R ₂ (1)

In formula (1), R ₁₂ represents two cameras with overlapping fields of view, that is, the rotation and translation matrix between camera 1 and camera 2, and R ₁ and R ₂ respectively represent the calibration of camera 1 and camera 2 in their overlapping fields of view Rotation translation matrix between plates.

(b) Solving the relative pose between cameras without overlapping fields of view

As shown in Figure 6, for cameras with no overlapping fields of view, the relative poses between the two sets of cameras with overlapping fields of view obtained above are used to transform, and then the rotation and translation matrix is solved to obtain cameras with no overlapping fields of view. The relative attitude between them is expressed as:

R ₁₃ =R ₁₂ ^-1 ·R ₂₃ (2)

In formula (2), R ₁₃ represents two cameras with no overlapping field of view, that is, the rotation and translation matrix between camera 3 and camera 1; R ₁₂ and R ₂₃ are the overlapping field of view cameras calculated by formula (1). The rotation-translation matrix between .

In step S3, re-acquisition control is performed on the image and the re-projection error is calculated by using the initial relative posture between the cameras.

In order to effectively traverse all the shooting areas inside the camera, the shooting areas of the camera pictures are divided. As shown in Figure 7, the image is divided into 9 regions. Due to the existence of the perspective imaging theorem, the visual camera always has high calibration accuracy in its calibration acquisition area, but low calibration accuracy in the non-acquisition area. The core idea of this embodiment is to allow the calibrated position of the trolley to traverse these nine areas. And in each area, it is guaranteed that the car can go to the designated area. Then the camera gets to take the picture.

As shown in FIG. 7 , C cameras can be set, and each camera contains p positions. At least C×p position picture data needs to be collected. In FIG. 7 , the number of p is set to 9 as an example.

The re-acquisition algorithm process includes:

Step S11, construct a corpus location queue T according to the region and the label in sequence.

T={t|t∈C×p}

In step S12, the existing images are checked, and a queue K of the acquired positions is constructed.

K={k|k∈C×p}

In step S13, the difference set L of the set T and the set K is obtained.

L={l|l∈C×p}

Step S14, traverse the set L, search for camera pictures, obtain the current position of the car, and change the car to move forward or backward at the current position.

In the above steps, the labels of all image blocks represented in T, for example, there are nine blocks in one image, and ten pictures are ninety blocks. Each time a picture is collected, an element is added to the queue K, and the uncollected set L is obtained. At this time, send a command to the calibration trolley, instructing to collect pictures in the next area. Repeat the above process until all data is collected. Through the above operations, new calibration plate image data D _add is obtained.

Obtain the three-dimensional coordinates of the corner points in the calibration image data D _add , and convert the three-dimensional coordinates into two-dimensional coordinates through the calculated initial inter-camera RT matrix and the calculated single-camera internal parameter matrix K:

The two-dimensional coordinates of the generated point obtained by formula (3) are compared with the two-dimensional coordinates of the target point, and the initial reprojection error is calculated:

error=||P ^2D -P ^3D_2D || ₂ (4)

In the above formula, P ^3D is the 3-dimensional coordinate of the target point obtained from the calibration image

P ^2D is the 2-dimensional coordinates of the target point obtained from the single camera calibration

P ^3D_2D is the two-dimensional coordinate of the estimated point generated by using the initial inter-camera RT matrix and the camera internal parameter matrix K, R represents the rotation matrix, t is the offset matrix, and P ^3D_2D is generated by using the initial inter-camera RT matrix and the camera internal parameter matrix K. The two-dimensional coordinates of the estimated point, _u , v represent the pixel coordinates on the image, and xw, _yw and _zw represent the three-dimensional coordinates.

Step S4, establish attitude error and optimize the global data

When re-projecting is performed using the initial relative pose between cameras obtained in step S3, the calculated projection error is relatively large. In order to reduce the reprojection error and improve the calibration accuracy, preferably, the obtained relative poses between multiple cameras are globally modeled, and then the Levenberg-Marquardt algorithm is used to iteratively optimize the reprojection error to improve the accuracy of the calibration. The calibration accuracy is used to obtain the final relative pose between cameras.

Global modeling:

min _R,t F(x)=‖f(x)-y‖ ₂ (5)

s.t.0≤f(x)≤800

In formula (5), f(x) is the two-dimensional coordinate of the generated estimated point P ^3D_2D : P ^3D_2D =K[R,t]P ^3D , and y is the two-dimensional coordinate of the target point P ^2D obtained by single camera calibration. Since the present invention has a picture resolution of 600×800 when collecting an image, 0≤f(x)≤800, and the specific situation is determined according to the image data format during the actual calibration.

Further, the Levenberg-Marquardt algorithm is used to optimize the projection error. The execution process of the algorithm is as follows:

In step S21, an initial value x ₀ and an initial optimized radius μ are given.

Step S22, for the k-th iteration process, solve:

where μ is the radius of the confidence region and D is the coefficient matrix.

Step S23, calculate

Step S24, if

Then set μ=2μ.

Step S25, if

Then set μ=0.5μ.

Step S26, if ρ is greater than a certain threshold, it is considered that the approximation is feasible, and let x _k+1 =x _k +Δx _k .

Step S27, it is judged whether the algorithm has converged. If not converged, return to step S22, otherwise end.

For clarity, the above process is described as follows:

x _k represents the initial relative attitude data after k optimizations, x _k+1 represents the initial relative attitude data after k+1 optimizations, Δx _k represents the k+1 optimization process, the obtained pair x _k The correction amount of , f(x _k ) represents the two-dimensional coordinates of the estimated point after the kth optimization; J[x _k ] represents the first derivative of f(x _k ) with respect to x, and D is the coefficient matrix.

The given initial value x ₀ is the initial relative pose data R ₁₂ between the first camera and the second camera. The set initial optimization radius μ can be specifically set according to the actual situation.

After k iterations of optimizing the reprojection error f(x), compute

The value of , where ρ is an index set in the Levenberg-Marquardt algorithm to describe the quality of the approximate second-order Taylor expansion used in the Gauss-Newton method, the denominator J(x) ^T Δx _k is the value at which the actual function falls, and the numerator f(x+Δx _k )-f(x) is the value at which the approximate model falls.

For the process of detecting the size of ρ, if the calculated value of ρ is small, the optimization radius μ should be reduced (in this embodiment, set

When , reduce the optimization radius μ, let μ=0.5μ); if the ρ value obtained by calculation is large, the optimization radius μ should be enlarged (in this embodiment, set

, enlarge the optimized radius μ, let μ=2μ).

When it is detected that ρ is greater than the preset threshold, it is determined that the approximation used in this iterative process is feasible, and x _k+1 = x _k +Δx _k ; at this time, the reprojection error after iterative optimization can be compared with the preset re-projection error. The size of the projection error threshold is used to judge whether the algorithm converges; if the reprojection error after iterative optimization is less than or equal to the preset reprojection error threshold, it is determined that the iterative optimization is complete, and the reprojection error after iterative optimization is used as the second reprojection error. , x _k+1 is the relative pose data between the first camera and the second camera. If the reprojection error after the iterative optimization is greater than the preset reprojection error threshold, continue to perform the k+2 th iterative optimization.

After the optimization algorithm is executed, the finally obtained x _k+1 is the required multi-camera (R, t) matrix. The present invention uses the optimized relative posture between cameras to calculate the reprojection error again. It has been verified that compared with the initial relative posture between cameras obtained in step S2, the optimized error is reduced to about 1/10 of the initial error, and the calibration accuracy Huge improvements.

To sum up, the present invention utilizes the five-sided stereo calibration plate and the intelligent trolley, which greatly simplifies the calibration process, saves time and effort; utilizes the movement of the trolley load calibration plate and the relationship between multiple cameras to solve the problem of no overlapping fields of view between cameras; Using global modeling and optimization, the calibration accuracy is significantly improved. It can solve the situation that there is no overlapping field of view between cameras; realize the automation of the whole calibration process, and obtain high calibration accuracy while being easy to operate.

Through the simulation experiment of the actual scene, the present invention has good results. The proposed multi-camera joint calibration system calibrates multiple image sensors by using the image sensors installed on the robot, and then obtains the relative pose between the robots. relation.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code, written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are all equivalent.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

An automated calibration system based on vision guidance, comprising: a calibration vehicle, a rotary control pan-tilt, a multi-faceted calibration board, a plurality of cameras, and a data acquisition control module, wherein the multi-sided calibration board is connected to the rotary control pan-tilt, and the data acquisition control module is used for Control the calibration vehicle to move with the multi-faceted calibration plate, and control the corresponding camera to shoot the multi-faceted calibration plate when moving to the overlapping field of view of the two adjacent cameras to obtain the image data of the multi-faceted calibration plate; the data acquisition control module will capture the obtained image data. The image data is stored according to the camera number and the position number of the multi-faceted calibration plate, and the corners of the multi-faceted calibration plate are detected to obtain the relative posture between the cameras.
The system according to claim 1, wherein the multi-faceted calibration plate is a five-sided three-dimensional checkerboard calibration plate, each of which is a 7×10 checkerboard calibration plate, wherein the four-sided calibration plate is arranged on the other side of the calibration plate and the angle between it and the other calibration plate is set to 45°.
The system according to claim 1, wherein the data acquisition control module obtains the relative pose between cameras according to the following steps:

According to the calibration plate data obtained by shooting, Zhang Zhengyou's calibration method is used to obtain the internal parameter matrix of each camera and the single-camera rotation and translation matrix between each camera and the calibration plate at different positions;

For two adjacent cameras with overlapping fields of view, use the rotation and translation matrix of a single camera to calculate the relative poses between cameras with overlapping fields of view respectively;

For cameras with no overlapping fields of view, use the obtained relative poses between the two sets of cameras with overlapping fields of view to solve the rotation-translation matrix;

Divide the shooting area of the camera picture, control the calibration position of the calibration vehicle to traverse the divided area, and calculate the initial re-projection error by re-collecting the shooting data;

The reprojection error is iteratively optimized to obtain the final relative pose between cameras.
The system of claim 3, wherein said calculating an initial projection error comprises:

Set C cameras, each of which contains p positions, at least collect image data of C×p positions;

According to the region and label in turn, construct the corpus position queue T, which is expressed as:

T={t|t∈C×p}

Existing images are examined and a queue K of acquired locations is constructed, expressed as:

K={k|k∈C×p}

Calculate the difference set L of set T and set K, expressed as:

L={l|l∈C×p}

Traverse the set L, search for camera pictures, obtain the current position of the calibration car, and change the calibration car forward and backward under the current position;

Obtain the three-dimensional coordinates of the corner points in the calibration image, and convert the three-dimensional coordinates into two-dimensional coordinates through the calculated rotation and translation matrix between the initial cameras and the calculated single-camera internal parameter matrix:

Calculate the initial reprojection error by comparing the two-dimensional coordinates of the generated point with the two-dimensional coordinates of the target point:

error=||P 2D -P 3D_2D || 2

Among them, P 3D is the 3-dimensional coordinates of the target point obtained from the calibration image
P 2D is the 2-dimensional coordinates of the target point obtained from the single camera calibration
P 3D_2D is the two-dimensional coordinate of the estimated point generated by using the rotation and translation matrix between the initial cameras and the camera intrinsic parameter matrix.
The system of claim 1, wherein the iteratively optimizing the reprojection error comprises:

Do global modeling:

s.t.0≤f(x)≤m

The reprojection error is iteratively optimized by the Levenberg-Marquardt algorithm to obtain the final relative pose between cameras;

Among them, f(x) is the two-dimensional coordinate of the generated estimated point P 3D_2D : P 3D_2D =K[R,t]P 3D , y is the two-dimensional coordinate of the target point P 2D obtained by single-camera calibration, m is determined according to the picture rate determined.
The system of claim 5, wherein the iteratively optimizing the reprojection error using the Levenberg-Marquardt algorithm comprises:

Step S61, the initial value x 0 and the initial optimization radius μ are given;

Step S62, for the k-th iteration process, solve:

Step S63, calculate
like
Then set μ=2μ, if
Then set μ=0.5μ;

Step S64, if ρ is greater than a certain threshold, set x k+1 =x k +Δx k ;

Among them, μ is the radius of the trust region, D is the coefficient matrix, x k+1 represents the initial relative attitude data after k+1 optimization, Δx k represents the k+1 optimization process, the obtained pair x k The correction amount of , f(x k ) represents the two-dimensional coordinates of the estimated point after the kth optimization; J[x k ] represents the first derivative of f(x k ) with respect to x, and D is the coefficient matrix.
The system according to claim 3, wherein, for cameras with no overlapping fields of view, the rotation-translation matrix is solved by using the obtained relative poses between the two groups of cameras with overlapping fields of view:

R 13 =R 12 -1 ·R 23

Among them, R 13 represents the rotation and translation matrix between the third camera and the first camera without overlapping fields of view, R 12 represents the rotation and translation matrix between the first camera and the second camera with overlapping fields of view, and R 23 represents the existence of Rotation-translation matrix between the second camera and the third camera with overlapping fields of view.
An automated calibration method based on vision guidance, comprising the following steps:

Control the calibration vehicle to move with the multi-faceted calibration plate, and control the corresponding camera to shoot the multi-faceted calibration plate when moving to the overlapping field of view of two adjacent cameras to obtain the image data of the multi-faceted calibration plate;

The image data obtained by shooting is stored according to the camera number and the position number of the multi-faceted calibration plate, and the corner points of the multi-faceted calibration plate are detected to obtain the relative posture between the cameras.
A computer-readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the steps of the method according to claim 8 .
A computer device, comprising a memory and a processor, a computer program that can be run on the processor is stored in the memory, and characterized in that, when the processor executes the program, the method of claim 8 is implemented. step.