WO2021184218A1

WO2021184218A1 - Relative pose calibration method and related apparatus

Info

Publication number: WO2021184218A1
Application number: PCT/CN2020/079780
Authority: WO
Inventors: 刘会; 陈亦伦; 陈同庆
Original assignee: 华为技术有限公司
Priority date: 2020-03-17
Filing date: 2020-03-17
Publication date: 2021-09-23
Also published as: CN112639883A; CN112639883B

Abstract

A relative pose calibration method and a related apparatus applied to cameras on a self-driving car. The relative pose calibration method comprises: acquiring images by means of a first camera and a second camera; performing feature point matching on the image acquired by the first camera and the image acquired by the second camera, so as to obtain a first matching feature point set; determining a first relative pose between the first camera and the second camera according to the first matching feature point set; and updating a second relative pose to the first relative pose in cases where the difference between the first relative pose and the second relative pose is not greater than a pose change threshold, the second relative pose being a relative pose between the first camera and the second camera currently stored in the self-driving apparatus. The relative pose calibration method provided in the present application has high calibration accuracy and high reliability.

Description

A relative pose calibration method and related device

Technical field

This application relates to the field of autonomous driving, and more particularly to a method and related devices for calibrating relative poses between cameras applied to autonomous vehicles.

Background technique

Self-driving cars need to have a high perception of the 360° environment of the body, and they need to have stable and reliable perception results for different scenes, different lighting conditions, and obstacles at different distances. A single camera cannot achieve full coverage of 360° due to factors such as camera parameters and installation location. Therefore, self-driving vehicles are generally equipped with multiple cameras to improve visual perception. Different cameras generally have different camera parameters (such as focal length, resolution, dynamic range, etc.), and are installed in different positions of the vehicle body to obtain a more comprehensive perception result. For example, self-driving vehicles are equipped with at least one camera in the front, side, and rear directions to cover a 360° environment around the vehicle body. At the same time, different cameras have different fields of view, and different cameras have partially overlapping fields of view.

The relative pose between the cameras on the self-driving car will change as the use time increases. At this time, the calibration parameters of each camera (ie the camera's external parameters) need to be corrected. The process of correcting the camera's external parameters is called For recalibration. With the increasing number of vehicles with automatic driving functions, the demand for recalibration of the camera's calibration parameters has also greatly increased. Auto-driving vehicles that cannot recalibrate the external parameters of each camera in a timely manner pose great safety hazards. The currently adopted solution for online recalibration of the external parameters of the camera is as follows: the autonomous vehicle obtains prior position information that characterizes the relative position between known urban construction landmarks currently distributed near the road, and communicates with the sensor in real time. The location of urban construction markers is compared, and then online calibration is performed. However, this solution has the following disadvantages: it needs to obtain a priori location information of the surrounding environment, which is easy to fail when the prior is not satisfied; it needs to collect a large amount of data to obtain the a priori location information, and the workload is large. Therefore, it is necessary to study a new multi-camera calibration scheme.

Summary of the invention

The embodiment of the present application provides a relative pose calibration method, which can accurately determine the relative pose between cameras and has high reliability.

In the first aspect, an embodiment of the present application provides a method for calibrating a relative pose. The method includes: acquiring images by a first camera and a second camera; the field of view of the first camera and the field of view of the second camera overlap , The first camera and the second camera are installed at different positions of the automatic driving device; feature point matching is performed on the image collected by the first camera and the image collected by the second camera to obtain the first matching feature Point set; the first matching feature point set includes H groups of feature point pairs, each set of feature point pairs includes two matching feature points, one of which is a feature extracted from the image collected by the first camera Point, the other feature point is a feature point extracted from an image collected by the second camera, and the H is an integer not less than 8; according to the first matching feature point set, the first camera and the The first relative pose between the second cameras; when the difference between the first relative pose and the second relative pose is not greater than the pose change threshold, the second relative pose Update to the first relative pose; the second relative pose is the relative pose between the first camera and the second camera currently stored by the automatic driving device.

Since the field of view of the first camera and the field of view of the second camera overlap, at least part of the feature points in the image collected by the first camera matches at least part of the feature points in the image collected by the second camera. The first matching feature point set is obtained by performing feature point matching on the feature points extracted from the image collected by the first camera and the feature points extracted from the image collected by the first camera. The autopilot device may be an autopilot car (also called an unmanned car), a drone, or other devices that need to calibrate the relative pose between cameras. In the embodiment of the present application, the first relative pose between the first camera and the second camera is determined according to the first matching feature point set, which does not depend on a specific venue or landmark, and does not have a priori assumptions about the surrounding environment. Compared with the online calibration solutions commonly used in the industry, the relative pose calibration method provided in the embodiments of the present application does not require a priori assumptions and is more adaptable to the environment. In addition, the relative pose calibration method provided by the embodiment of the present application can achieve a repetition accuracy of less than 0.1°, effectively reduces the number of recalibration of the relative pose, and has high reliability.

In an optional implementation manner, the determining the first relative pose between the first camera and the second camera according to the first matching feature point set includes: determining the first camera and the The essential matrix between the second cameras; according to the singular value decomposition result of the essential matrix, calculate the relative pose of the 5-DOF DOF between the first camera and the second camera; compare the first distance and the second The ratio between the distances is used as a scale factor; the first distance and the second distance are the results of measuring the same distance by a non-visual sensor and a visual sensor on the automatic driving device, and the visual sensor includes the The first camera and/or the second camera; combining the 5-degree-of-freedom relative pose and the scale factor to obtain the first relative pose.

In this implementation, the 5-DOF relative pose between the two cameras is obtained by decomposing the essential matrix, and then the 5-DOF relative pose and the scale factor are used to obtain the 6-DOF relative pose without prior assumptions about the surrounding environment , There is no need to collect 3D information of surrounding scenes, and the workload is small.

In an optional implementation manner, the calculating the 5-degree-of-freedom relative pose between the first camera and the second camera according to the singular value decomposition result of the essential matrix includes: The essential matrix performs singular value decomposition to obtain the singular value decomposition result; according to the singular value decomposition result, at least two relative poses between the first camera and the second camera are obtained; respectively using the at least Two relative poses are used to calculate the three-dimensional coordinate positions of the feature points in the first matching feature point set; the three-dimensional coordinate positions of the feature points in the first matching feature point set are equalized in the at least two relative poses The relative pose located in front of the first camera and the second camera is used as the 5-degree-of-freedom relative pose.

In this implementation, the 5-degree-of-freedom relative pose between the first camera and the second camera can be determined accurately and quickly.

In an optional implementation manner, the determining the first relative pose between the first camera and the second camera according to the first matching feature point set includes: iteratively solving the target equation to obtain the first A relative pose; in the target equation, the parameters included in the first relative pose are unknowns, the pair of feature points in the first matching feature point set, the internal parameters of the first camera, and the second The internal parameters of the camera are known numbers.

In this implementation manner, by iteratively solving the target equation to obtain the first relative pose, the first relative pose can be calculated quickly and accurately, without the need to calculate the essential matrix.

In an optional implementation manner, the automatic driving device is equipped with M cameras, and the M cameras include the first camera, the second camera, and the third camera, and the field of view of the third camera is the same as The field of view of the first camera and the field of view of the second camera overlap, and the M is an integer greater than 2. The method further includes: obtaining M first rotation matrices, M second rotation matrices, M First translation matrices and M second translation matrices; the M first rotation matrices are the rotation matrices between the first camera and the second camera, and at least two of the M first rotation matrices Different rotation matrices, the M second rotation matrices are the rotation matrices between the second camera and the third camera, and at least two of the M second rotation matrices are different, the M The first translation matrices are the translation matrices between the first camera and the second camera, and at least two of the M first translation matrices are different, and the M second translation matrices are the The translation matrix between the second camera and the third camera and at least two translation matrices in the M second translation matrices are different, and the M first rotation matrices and the M first translation matrices are one-to-one Correspondingly, the M second rotation matrices have a one-to-one correspondence with the M second translation matrices, and the M is an integer greater than 1; the first equation set is solved to obtain the third rotation matrix; the first equation set Includes M first equations, the M first equations have a one-to-one correspondence with the M first rotation matrices, and the M second rotation matrices have a one-to-one correspondence; in each first equation, The first rotation matrix and the second rotation matrix are known numbers, and the third rotation matrix is an unknown number; the third rotation matrix is the rotation matrix between the first camera and the third camera; Equation set to obtain a third translation matrix; the second equation set includes M second equations, and the M second equations correspond to the M first rotation matrices one-to-one and correspond to the M first rotation matrices. The two rotation matrices have a one-to-one correspondence; in each second equation, the first rotation matrix, the second rotation matrix, the first translation matrix, and the second translation matrix are known numbers, and the third translation matrix is an unknown number; The third translation matrix is the translation matrix between the first camera and the third camera; the poses including the third rotation matrix and the third translation matrix are used as the first camera and the second camera. The relative pose between the three cameras.

In this implementation, the relative pose matrices between the multiple cameras are formed into a closed-loop equation, and the closed-loop equation is used to achieve the pose calibration between cameras without a common field of view and perform overall optimization, so as to obtain a more accurate The relative pose of the camera.

In an optional implementation manner, the first matching feature point set includes feature points extracted by the first camera from at least two frames of images and features extracted by the second camera from at least two frames of images Points; said determining the first relative pose between the first camera and the second camera according to the first matching feature point set includes: determining the first relative pose between the first camera and the second camera according to the feature point pair in the first matching feature point set The relative pose between the first camera and the second camera to obtain a first intermediate pose; substituting each feature point pair in the first matching feature point set into a first formula to obtain the first matching feature The residuals corresponding to each feature point pair in the point set, the first intermediate pose is included in the first formula representation; the interference feature point pair in the first matching feature point set is eliminated to obtain a second matching feature point set; The interference feature point pair is a feature point pair whose corresponding residual error in the first matching feature point set is greater than a residual threshold; and the first camera and all the points are determined according to the feature point pair in the second matching feature point set. The relative pose between the second cameras to obtain a second intermediate pose; take the target intermediate pose as the first relative pose between the first camera and the second camera; the target intermediate The pose is the relative pose between the first camera and the second camera determined according to the feature point pairs in the target matching feature point set, and the number of the feature point pairs in the target matching feature point set is less than the number threshold Or the ratio of the number of feature point pairs in the target matching feature point set to the number of feature point pairs in the first matching feature point set is less than a ratio threshold.

Optionally, the feature points included in the first matching feature point set are feature points that match in the first image collected by the first camera and the second image collected by the second camera. When the number of matching feature point pairs in the first image and the second image is not less than 8, an estimate of the relative pose of the two cameras can be obtained. When the number of matching feature points in a frame of image collected by the first camera and a frame of image collected by the second camera is small, the estimation is unstable, and the influence of errors can be reduced by multi-frame accumulation. In some embodiments, the automatic driving device can extract the feature points in the two images of the i-th frame, denoted as x′ _i and x _i , respectively, and add them to the feature point sets X ₁ and X ₂ ; after F frames, the feature points The point sets X ₁ and X ₂ store the matching feature points in the two images in the F frame (that is, the first matching feature point set), and the feature point sets X ₁ and X _{2 are} used to estimate the relative pose of the two cameras; due to the features The number of feature points in the point sets X ₁ and X ₂ is far more than the number of feature points in a single frame, so iterative optimization can be used: first _{, all the points in the feature point sets X 1} and X ₂ are used for pose estimation to obtain the first The intermediate pose, and then substitute the first intermediate pose into formula (14) to calculate the residual, and remove the feature points whose residuals are greater than the residual threshold, and use the remaining feature points to re-evaluate the pose to obtain the second intermediate position Pose; repeat the above process until the number of feature points to be eliminated is less than the number threshold (such as 5% of the total). At this time, the optimal solution under a set of F frame data can be obtained. It should be understood that the two images in a frame refer to an image captured by the first camera and an image captured by the second camera, and the time interval between the two images in this frame is less than a time threshold, such as 0.5 ms, 1 ms, and so on. According to the central limit theorem, it is assumed that in a long-term observation, the R and R calculated between different frames

Are independent and identically distributed, then as F increases, R and

Will stabilize. Choose an appropriate F such that R and

The variance of satisfies the demand.

In this implementation, the multi-frame accumulation method can be used to avoid the jitter problem caused by the calculation of a single frame and stabilize the calibration result.

In an optional implementation manner, the method further includes: outputting a reminder if the difference between the first relative pose and the second relative pose is not greater than the pose change threshold Information; the reminder information is used to prompt that the relative pose between the first camera and the second camera is abnormal.

In this implementation, passengers can be reminded in time that the calibration parameters of the camera are abnormal.

In the second aspect, an embodiment of the present application provides another relative pose calibration method. The method includes: a server receives recalibration reference information from an automatic driving device, and the recalibration reference information is used by the server to determine the automatic The relative pose between the first camera and the second camera of the driving device, the field of view of the first camera overlaps the field of view of the second camera, and the first camera and the second camera are installed in the Different locations of the automatic driving device; the server obtains a first matching feature point set according to the recalibration reference information, the first matching feature point set includes H groups of feature point pairs, and each group of feature point pairs includes two phases Matching feature points, one of the feature points is a feature point extracted from the image collected by the first camera, and the other feature point is a feature point extracted from the image collected by the second camera, and the H is not An integer less than 8; the server determines the first relative pose between the first camera and the second camera according to the first matching feature point set; the server converts the first relative pose Send to the automatic driving device.

Optionally, the recalibration reference information includes an image synchronously collected by the first camera and the second camera or the first matching feature point set. The server may store the internal parameters of the first camera and the internal parameters of the second camera, or receive the internal parameters of the first camera and the internal parameters of the second camera from the automatic driving device, and use the internal parameters according to the first camera. Matching a set of feature points to determine a first relative pose between the first camera and the second camera.

In the embodiment of the present application, the automatic driving device does not need to calibrate the relative pose between the cameras by itself, and only needs to send the data required to calibrate the relative pose between the cameras to the server, and the server compares the camera on the automatic driving device. Re-calibration is performed on the relative pose between the two, and the efficiency is high.

In an optional implementation manner, before the determining the first relative pose between the first camera and the second camera according to the first matching feature point set, the method further includes: the server Receiving a pose calibration request from the automatic driving device, the pose calibration request being used to request recalibration of the relative pose between the first camera and the second camera of the automatic driving device, the The pose calibration request carries the internal parameters of the first camera and the internal parameters of the second camera; said determining the first relative relationship between the first camera and the second camera according to the first matching feature point set The pose includes: determining the first relative pose between the first camera and the second camera according to a first matching feature point set, internal parameters of the first camera, and internal parameters of the second camera .

In the third aspect, the embodiments of the present application provide another relative pose calibration method. The method includes: the automatic driving device sends recalibration reference information to the server, and the recalibration reference information is used by the server to determine the status of the automatic driving device. The relative pose between the first camera and the second camera, the field of view of the first camera overlaps the field of view of the second camera, and the difference between the first camera and the second camera installed in the automatic driving device Position; the automatic driving device receives the first relative pose from the server; the difference between the first relative pose and the second relative pose of the automatic driving device is not greater than the pose change In the case of a threshold value, the second relative pose is updated to the first relative pose; the second relative pose is the first camera and the second camera currently stored by the automatic driving device The relative pose between. Optionally, the recalibration reference information includes a first matching feature point set, the first matching feature point set includes H groups of feature point pairs, and each group of feature point pairs includes two matching feature points, one of which is A point is a feature point extracted from an image collected by the first camera, another feature point is a feature point extracted from an image collected by the second camera, and the H is an integer not less than 8. Optionally, the recalibration reference information includes images acquired by the first camera and the second camera synchronously.

In the embodiment of the present application, the automatic driving device does not need to calibrate the relative pose between the cameras by itself, and only needs to send the data required to calibrate the relative pose between the cameras to the server, which requires less workload.

In an optional implementation manner, before the automatic driving device sends the recalibration reference information to the server, the method further includes: the automatic driving device sends a pose calibration request to the server, the pose calibration request Used to request recalibration of the relative pose between the first camera and the second camera of the automatic driving device, the pose calibration request carries the internal parameters of the first camera and the second camera The internal reference.

In a fourth aspect, an embodiment of the present application provides an automatic driving device, including: an image acquisition unit for acquiring images through a first camera and a second camera respectively; the field of view of the first camera and the second camera There is overlap in the field of view, the first camera and the second camera are installed at different positions of the automatic driving device; the image feature point extraction unit is used for the image collected by the first camera and the image collected by the second camera Perform feature point matching to obtain a first matching feature point set; the first matching feature point set includes H groups of feature point pairs, and each group of feature point pairs includes two matching feature points, one of which is from the The feature point extracted from the image collected by the first camera, the other feature point is the feature point extracted from the image collected by the second camera, and the H is an integer not less than 8; the pose calculation unit is used for According to the first matching feature point set, the first relative pose between the first camera and the second camera is determined; the first matching feature point set includes H groups of feature point pairs, each group of feature points The pair includes two matching feature points, one of which is a feature point extracted from the image collected by the first camera, and the other feature point is a feature point extracted from the image collected by the second camera, The field of view of the first camera and the field of view of the second camera overlap, the first camera and the second camera are installed at different positions of the automatic driving device, and the H is an integer not less than 8;

The calibration parameter update unit is configured to update the second relative pose to the first when the difference between the first relative pose and the second relative pose is not greater than the pose change threshold. Relative pose; the second relative pose is the relative pose between the first camera and the second camera currently stored by the automatic driving device.

In an optional implementation manner, the pose calculation unit is specifically configured to determine the essential matrix between the first camera and the second camera; and calculate the essential matrix according to the singular value decomposition result of the essential matrix. The 5-degree-of-freedom DOF relative pose between the first camera and the second camera; the device further includes: a scale calculation unit configured to use the ratio between the first distance and the second distance as a scale factor; The first distance and the second distance are the results obtained by measuring the same distance by a non-visual sensor and a visual sensor on the automatic driving device, and the visual sensor includes the first camera and/or the second distance. Camera; the pose calculation unit is also used to combine the 5-degree-of-freedom relative pose and the scale factor to obtain the first relative pose.

In an optional implementation manner, the pose calculation unit is specifically configured to perform singular value decomposition of the essential matrix to obtain the singular value decomposition result; according to the singular value decomposition result, the first The at least two relative poses between the camera and the second camera; the three-dimensional coordinate positions of the feature points in the first matching feature point set are calculated by using the at least two relative poses respectively; and the at least two In the relative pose, the relative pose in which the three-dimensional coordinate positions of each feature point in the first matching feature point set are located in front of the first camera and the second camera is used as the 5-degree-of-freedom relative pose.

In an optional implementation manner, the pose calculation unit is specifically configured to iteratively solve a target equation to obtain the first relative pose; in the target equation, the parameters included in the first relative pose Is an unknown number, and the feature point pairs in the first matching feature point set, the internal parameters of the first camera, and the internal parameters of the second camera are known numbers.

In an optional implementation manner, the automatic driving device is equipped with M cameras, and the M cameras include the first camera, the second camera, and the third camera, and the field of view of the third camera is the same as There is overlap between the field of view of the first camera and the field of view of the second camera, and the M is an integer greater than 2. The device further includes: a closed loop optimization unit for obtaining M first rotation matrices, M A second rotation matrix, M first translation matrices, and M second translation matrices; the M first rotation matrices are the rotation matrices between the first camera and the second camera, and the M At least two rotation matrices in a rotation matrix are different, the M second rotation matrices are the rotation matrices between the second camera and the third camera, and at least two of the M second rotation matrices rotate The matrices are different, the M first translation matrices are the translation matrices between the first camera and the second camera, and at least two of the M first translation matrices are different, the M first translation matrices are different. The second translation matrix is the translation matrix between the second camera and the third camera, and at least two of the M second translation matrices are different, and the M first rotation matrices are different from the M first rotation matrices. The first translation matrix has a one-to-one correspondence, the M second rotation matrices have a one-to-one correspondence with the M second translation matrices, and the M is an integer greater than 1; the first equation system is solved to obtain the third rotation matrix; The first equation group includes M first equations, and the M first equations have a one-to-one correspondence with the M first rotation matrices, and have a one-to-one correspondence with the M second rotation matrices; In the first equation, the first rotation matrix and the second rotation matrix are known numbers, and the third rotation matrix is an unknown number; the third rotation matrix is the difference between the first camera and the third camera A rotation matrix; solve a second equation system to obtain a third translation matrix; the second equation system includes M second equations, and the M second equations have a one-to-one correspondence with the M first rotation matrices, and Corresponds to the M second rotation matrices one-to-one; in each second equation, the first rotation matrix, the second rotation matrix, the first translation matrix, and the second translation matrix are known numbers, and the third translation The matrix is an unknown number; the third translation matrix is the translation matrix between the first camera and the third camera; the pose including the third rotation matrix and the third translation matrix is taken as the first The relative pose between a camera and the third camera.

In an optional implementation manner, the first matching feature point set includes feature points extracted by the first camera from at least two frames of images and features extracted by the second camera from at least two frames of images Point; the pose calculation unit is specifically configured to determine the relative pose between the first camera and the second camera according to the feature point pair in the first matching feature point set to obtain the first intermediate position Pose; Substituting each feature point pair in the first matching feature point set into a first formula to obtain a residual error corresponding to each feature point pair in the first matching feature point set, the first formula representation includes the first formula An intermediate pose; eliminate the interference feature point pair in the first matching feature point set to obtain a second matching feature point set; the interference feature point pair is that the corresponding residual in the first matching feature point set is greater than the residual Threshold feature point pairs; according to the feature point pairs in the second matching feature point set, determine the relative pose between the first camera and the second camera to obtain a second intermediate pose; set the target intermediate position The pose is the first relative pose between the first camera and the second camera; the intermediate pose of the target is the first camera and the total pose determined according to the feature point pair in the target matching feature point set. The relative poses between the second cameras, the number of feature point pairs in the target matching feature point set is less than a number threshold or the number of feature point pairs in the target matching feature point set is the same as the first matching feature The ratio of the number of feature point pairs in the point set is less than the ratio threshold.

In an optional implementation manner, the device further includes: a reminding unit, configured to determine when the difference between the first relative pose and the second relative pose is not greater than the pose change threshold In this case, output reminding information; the reminding information is used to prompt that the relative pose between the first camera and the second camera is abnormal.

In a fifth aspect, an embodiment of the present application provides a server, including: an obtaining unit, configured to obtain a first matching feature point set, the first matching feature point set includes H groups of feature point pairs, and each group of feature point pairs includes Two matching feature points, one feature point is a feature point extracted from the image collected by the first camera, and the other feature point is a feature point extracted from the image collected by the second camera. The field of view and the field of view of the second camera overlap, the first camera and the second camera are installed at different positions of the automatic driving device, and the H is an integer not less than 8; the pose calculation unit is used to calculate The first matching feature point set determines a first relative pose between the first camera and the second camera; a sending unit is configured to send the first relative pose to the automatic driving device.

In an optional implementation manner, the server further includes: a receiving unit configured to receive a pose calibration request from the automatic driving device, the pose calibration request being used to request recalibration of the automatic driving device The relative pose between the first camera and the second camera, the pose calibration request carries the internal parameters of the first camera and the internal parameters of the second camera; the pose calculation unit specifically It is used to determine the first relative pose between the first camera and the second camera according to the first matching feature point set, the internal parameters of the first camera, and the internal parameters of the second camera.

In a sixth aspect, an embodiment of the present application provides an automatic driving device, including: a sending unit, configured to send recalibration reference information to a server, where the recalibration reference information is used by the server to determine the first camera of the automatic driving device The relative pose between the first camera and the second camera, the field of view of the first camera overlaps the field of view of the second camera, and the first camera and the second camera are installed in different positions of the automatic driving device; receiving; Unit for receiving the first relative pose from the server; a calibration parameter update unit for the difference between the first relative pose and the second relative pose is not greater than the pose change threshold , The second relative pose is updated to the first relative pose; the second relative pose is one of the first camera and the second camera currently stored by the automatic driving device The relative pose between.

In an optional implementation manner, the sending unit is further configured to send a pose calibration request to the server, and the pose calibration request is used to request recalibration of the first camera and the first camera of the automatic driving device. For the relative poses between the second cameras, the pose calibration request carries the internal parameters of the first camera and the internal parameters of the second camera.

In a seventh aspect, an embodiment of the present application provides a car, the car includes a memory and a processor, the memory is used to store code; the processor is used to execute the program stored in the memory, when the program is executed At this time, the processor is configured to execute the methods of the foregoing first aspect or the foregoing third aspect and optional implementation manners.

In an eighth aspect, an embodiment of the present application provides a server that includes a memory and a processor, the memory is used to store code; the processor is used to execute the program stored in the memory, and when the program is executed At this time, the processor is configured to execute the above-mentioned second aspect and optional implementation methods.

In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium that stores a computer program, and the computer program includes program instructions that, when executed by a processor, cause the processor to execute the above-mentioned first From one aspect to the third aspect and optional implementation methods.

In a tenth aspect, an embodiment of the present application provides a chip that includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface, and executes the above-mentioned first to third aspects and any An alternative implementation method.

In an eleventh aspect, an embodiment of the present application provides a computer program product. The computer program product includes program instructions that, when executed by a processor, cause the processor to execute the first to third aspects and any one of An alternative implementation method.

Description of the drawings

Fig. 1 is a functional block diagram of an automatic driving device provided by an embodiment of the present application;

2 is a schematic structural diagram of an automatic driving system provided by an embodiment of the application;

FIG. 3 is a flowchart of a relative pose calibration method provided by an embodiment of this application;

FIG. 4 is a schematic diagram of a public field of view between cameras provided by an embodiment of this application;

FIG. 5 is a schematic structural diagram of an automatic driving device provided by an embodiment of the application;

6 is a schematic structural diagram of another automatic driving device provided by an embodiment of the application;

FIG. 7 is a schematic structural diagram of another automatic driving device provided by an embodiment of the application;

FIG. 8 is a flowchart of another relative pose calibration method provided by an embodiment of the application;

FIG. 9 is a flowchart of another relative pose calibration method provided by an embodiment of the application;

FIG. 10 is a schematic structural diagram of a server provided by an embodiment of this application.

Detailed ways

The terms "first", "second", and "third" in the specification embodiments and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or Priority. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device need not be limited to those clearly listed steps or units, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices. "And/or" is used to indicate one or all of the two connected objects. For example, "A and/or B" means A, B, or A+B.

As described in the background art, the relative poses between cameras on an autonomous vehicle will change with the increase of use time. At this time, it is necessary to correct the calibration parameters of each camera (that is, the external parameters of the camera). With the increasing number of vehicles with automatic driving functions, the demand for recalibration of the camera's calibration parameters has also greatly increased. Auto-driving vehicles that cannot recalibrate the external parameters of each camera in a timely manner pose great safety hazards. The relative pose calibration method provided in the embodiments of the present application can be applied to an autonomous driving scenario. The following is a brief introduction to the driving scene.

Driving scenario 1: The automatic driving device determines the relative pose between the two cameras according to the matching feature points in the images synchronously collected by the two cameras with a common field of view; the relative pose between the two cameras is determined again If the difference between the pose and the relative pose between the two cameras currently calibrated by the automatic driving device is not greater than the pose change threshold, the currently calibrated relative pose is updated.

Driving scenario 2: The automatic driving device sends a pose calibration request to the server and information needed to determine the relative pose between at least two cameras on the automatic driving device. The above pose calibration request is used to request automatic recalibration. The relative pose between at least two cameras on the driving device; the automatic driving device updates its currently stored relative pose between the at least two cameras according to the relative pose from the server

Fig. 1 is a functional block diagram of an automatic driving device provided by an embodiment of the present application. In one embodiment, the automatic driving device 100 is configured in a fully or partially automatic driving mode. For example, the automatic driving device 100 can control itself while in the automatic driving mode, and can determine the current state of the automatic driving device 100 and its surrounding environment through human operation, and determine the possible behavior of at least one other vehicle in the surrounding environment, The confidence level corresponding to the possibility of the other vehicle performing possible behavior is determined, and the automatic driving device 100 is controlled based on the determined information. When the automatic driving device 100 is in the automatic driving mode, the automatic driving device 100 can be set to operate without human interaction.

The automatic driving apparatus 100 may include various subsystems, such as a traveling system 102, a sensor system 104, a control system 106, one or more peripheral devices 108 and a power supply 110, a computer system 112, and a user interface 116. Optionally, the automatic driving device 100 may include more or fewer sub-systems, and each sub-system may include multiple elements. In addition, each subsystem and element of the automatic driving device 100 may be interconnected by wire or wirelessly.

The traveling system 102 may include components that provide power movement for the autonomous driving device 100. In one embodiment, the propulsion system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels/tires 121. The engine 118 may be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines, such as a hybrid engine composed of a gasoline engine and an electric motor, or a hybrid engine composed of an internal combustion engine and an air compression engine. The engine 118 converts the energy source 119 into mechanical energy.

Examples of energy sources 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. The energy source 119 may also provide energy for other systems of the automatic driving device 100.

The transmission device 120 can transmit mechanical power from the engine 118 to the wheels 121. The transmission device 120 may include a gearbox, a differential, and a drive shaft. In an embodiment, the transmission device 120 may also include other devices, such as a clutch. Among them, the drive shaft may include one or more shafts that can be coupled to one or more wheels 121.

The sensor system 104 may include several sensors that sense information about the environment around the automatic driving device 100. For example, the sensor system 104 may include a positioning system 122 (the positioning system may be a global positioning system (GPS) system, a Beidou system or other positioning systems), an inertial measurement unit (IMU) 124, and a radar 126, a laser rangefinder 128, and a camera 130. The sensor system 104 may also include sensors of the internal system of the automatic driving device 100 to be monitored (for example, an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). Such detection and recognition are key functions for the safe operation of the autonomous automatic driving device 100.

The positioning system 122 may be used to estimate the geographic location of the automatic driving device 100. The IMU 124 is used to sense the position and orientation changes of the automatic driving device 100 based on inertial acceleration. In one embodiment, the IMU 124 may be a combination of an accelerometer and a gyroscope.

The radar 126 may use radio signals to sense objects in the surrounding environment of the automatic driving device 100.

The laser rangefinder 128 can use laser light to sense objects in the environment where the automatic driving device 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, laser scanners, and one or more detectors, as well as other system components.

The camera 130 may be used to capture multiple images of the surrounding environment of the automatic driving device 100. The camera 130 may be a still camera or a video camera. The camera 130 may capture multiple images of the surrounding environment of the automatic driving device 100 in real time or periodically. The camera 130 includes at least two cameras with overlapping fields of view, that is, at least two cameras have a common field of view.

The control system 106 controls the operation of the automatic driving device 100 and its components. The control system 106 may include various components, including a steering system 132, a throttle 134, a braking unit 136, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

The steering system 132 is operable to adjust the forward direction of the automatic driving device 100. For example, in one embodiment, it may be a steering wheel system.

The throttle 134 is used to control the operating speed of the engine 118 and thereby control the speed of the automatic driving device 100.

The braking unit 136 is used to control the automatic driving device 100 to decelerate. The braking unit 136 may use friction to slow down the wheels 121. In other embodiments, the braking unit 136 may convert the kinetic energy of the wheels 121 into electric current. The braking unit 136 may also take other forms to slow down the rotation speed of the wheels 121 to control the speed of the automatic driving device 100.

The computer vision system 140 may be operated to process and analyze the images captured by the camera 130 in order to recognize objects and/or features in the surrounding environment of the autonomous driving device 100. The aforementioned objects and/or features may include traffic signals, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, automatic driving methods, Structure from Motion (SFM) algorithms, video tracking, and other computer vision technologies. In some embodiments, the computer vision system 140 may be used to map the environment, track objects, estimate the speed of objects, and so on. The computer vision system 140 may use the point cloud obtained by the lidar and the image of the surrounding environment obtained by the camera to locate the position of the obstacle.

The route control system 142 is used to determine the driving route of the automatic driving device 100. In some embodiments, the route control system 142 may combine data from the sensor 138, the GPS 122, and one or more predetermined maps to determine the driving route for the automatic driving device 100.

The obstacle avoidance system 144 is used to identify, evaluate and avoid or otherwise cross over potential obstacles in the environment of the automatic driving device 100.

Of course, in one example, the control system 106 may add or alternatively include components other than those shown and described. Alternatively, a part of the components shown above may be reduced.

The automatic driving device 100 interacts with external sensors, other vehicles, other computer systems, or users through peripheral devices 108. The peripheral device 108 may include a wireless communication system 146, an onboard computer 148, a microphone 150, and/or a speaker 152.

In some embodiments, the peripheral device 108 provides a means for the user of the autonomous driving apparatus 100 to interact with the user interface 116. For example, the onboard computer 148 may provide information to the user of the automatic driving device 100. The user interface 116 can also operate the on-board computer 148 to receive user input. The on-board computer 148 can be operated through a touch screen. In other cases, the peripheral device 108 may provide a means for the autonomous driving device 100 to communicate with other devices located in the vehicle. For example, the microphone 150 may receive audio (eg, voice commands or other audio input) from the user of the autonomous driving device 100. Similarly, the speaker 152 may output audio to the user of the automatic driving device 100.

The wireless communication system 146 may wirelessly communicate with one or more devices directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 146 may use WiFi to communicate with a wireless local area network (WLAN). In some embodiments, the wireless communication system 146 may directly communicate with the device using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems. For example, the wireless communication system 146 may include one or more dedicated short-range communications (DSRC) devices. These devices may include vehicles and/or roadside stations. Public and/or private data communications.

The power supply 110 may provide power to various components of the automatic driving device 100. In one embodiment, the power source 110 may be a rechargeable lithium ion or lead-acid battery. One or more battery packs of such batteries may be configured as a power source to provide power to various components of the automatic driving device 100. In some embodiments, the power source 110 and the energy source 119 may be implemented together, such as in some all-electric vehicles.

Part or all of the functions of the automatic driving device 100 are controlled by the computer system 112. The computer system 112 may include at least one processor 113 that executes instructions 115 stored in a non-transitory computer readable medium such as a data storage device 114. The computer system 112 may also be multiple computing devices that control individual components or subsystems of the automatic driving apparatus 100 in a distributed manner.

The processor 113 may be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, the processor may be a dedicated device such as an ASIC or other hardware-based processor. Although FIG. 1 functionally illustrates the processor, memory, and other elements of the computer system 112 in the same block, those of ordinary skill in the art should understand that the processor, computer, or memory may actually include Multiple processors, computers, or memories stored in the same physical enclosure. For example, the memory may be a hard disk drive or other storage medium located in a housing other than the computer system 112. Therefore, a reference to a processor or computer will be understood to include a reference to a collection of processors or computers or memories that may or may not operate in parallel. Rather than using a single processor to perform the steps described herein, some components such as the steering component and the deceleration component may each have its own processor, and the above-mentioned processors only perform calculations related to component-specific functions.

In the various aspects described herein, the processor may be located far from the automatic driving device and wirelessly communicate with the automatic driving device. In other aspects, some operations in the process described herein are performed on a processor arranged in the automatic driving device while others are performed by a remote processor, including taking the necessary steps to perform a single manipulation.

In some embodiments, the data storage device 114 may include instructions 115 (eg, program logic), which may be executed by the processor 113 to perform various functions of the automatic driving device 100, including those described above. The data storage device 114 may also contain additional instructions, including sending data to, receiving data from, interacting with, and/or performing data on one or more of the propulsion system 102, the sensor system 104, the control system 106, and the peripheral device 108. Control instructions.

In addition to the instructions 115, the data storage device 114 may also store data, such as road maps, route information, the location, direction, speed, and other information of the vehicle. This information may be used by the automatic driving device 100 and the computer system 112 during the operation of the automatic driving device 100 in autonomous, semi-autonomous, and/or manual modes.

The user interface 116 is used to provide information to or receive information from the user of the automatic driving device 100. Optionally, the user interface 116 may include one or more input/output devices in the set of peripheral devices 108, such as a wireless communication system 146, a car computer 148, a microphone 150, and a speaker 152.

The computer system 112 may control the functions of the automatic driving device 100 based on inputs received from various subsystems (for example, the traveling system 102, the sensor system 104, and the control system 106) and from the user interface 116. For example, the computer system 112 may utilize input from the control system 106 in order to control the steering unit 132 to avoid obstacles detected by the sensor system 104 and the obstacle avoidance system 144. In some embodiments, the computer system 112 is operable to provide control of many aspects of the autonomous driving device 100 and its subsystems.

Optionally, one or more of the aforementioned components may be installed or associated with the automatic driving device 100 separately. For example, the data storage device 114 may exist partially or completely separately from the automatic driving device 100. The above-mentioned components may be communicatively coupled together in a wired and/or wireless manner.

Optionally, the above-mentioned components are only an example. In practical applications, the components in the above-mentioned modules may be added or deleted according to actual needs. FIG. 1 should not be construed as a limitation to the embodiments of the present application.

A self-driving car traveling on a road, such as the self-driving device 100 above, can recognize objects in its surrounding environment to determine the adjustment to the current speed. The aforementioned objects may be other vehicles, traffic control equipment, or other types of objects. In some examples, each recognized object can be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, distance from the vehicle, etc., can be used to determine the speed to be adjusted by the self-driving car.

Optionally, the automatic driving device 100 or a computing device associated with the automatic driving device 100 (such as the computer system 112, the computer vision system 140, and the data storage device 114 in FIG. 1) may be based on the characteristics of the identified object and the surrounding environment. The state (for example, traffic, rain, ice on the road, etc.) predicts the behavior of the above-identified object. Optionally, each recognized object depends on each other's behavior, so all recognized objects can also be considered together to predict the behavior of a single recognized object. The automatic driving device 100 can adjust its speed based on the predicted behavior of the aforementioned recognized object. In other words, an autonomous vehicle can determine what stable state the vehicle will need to adjust to (for example, accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, other factors may also be considered to determine the speed of the automatic driving device 100, such as the lateral position of the automatic driving device 100 on the traveling road, the curvature of the road, the proximity of static and dynamic objects, and so on.

In addition to providing instructions to adjust the speed of the self-driving car, the computing device can also provide instructions to modify the steering angle of the self-driving device 100, so that the self-driving car follows a given trajectory and/or maintains objects near the self-driving car. (For example, a car in an adjacent lane on a road) The safe horizontal and vertical distance.

The above-mentioned automatic driving device 100 may be a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a lawn mower, a recreational vehicle, a playground vehicle, construction equipment, a tram, a golf cart, a train, a trolley, etc. , The embodiment of the present invention does not make any special limitation.

The automatic driving device 100 can collect images in real time or periodically through the camera 130. Two or more cameras included in the camera 130 can collect images synchronously. The simultaneous acquisition of images by two cameras means that the shortest time for the two cameras to acquire images is less than a time threshold, for example, 10 ms. Optionally, the automatic driving device 100 sends the image collected by the camera 130 and the information needed to determine the relative pose between at least two cameras included in the camera 130 to the server; receives the relative pose from the server, and The relative pose between at least two cameras included in the camera 130 is updated. Optionally, the automatic driving device 100 determines the relative pose between at least two cameras included in the camera 130 according to the images collected by the camera 130.

Fig. 1 shows a functional block diagram of an automatic driving device 100, and an automatic driving system 101 is introduced below. Fig. 2 is a schematic structural diagram of an automatic driving system provided by an embodiment of the application. Fig. 1 and Fig. 2 describe the automatic driving device 100 from different perspectives. As shown in FIG. 2, the computer system 101 includes a processor 103, and the processor 103 is coupled to a system bus 105. The processor 103 may be one or more processors, where each processor may include one or more processor cores. A display adapter (video adapter) 107, the display adapter can drive the display 109, and the display 109 is coupled to the system bus 105. The system bus 105 is coupled with an input/output (I/O) bus 113 through a bus bridge 111. The I/O interface 115 is coupled to the I/O bus. The I/O interface 115 communicates with a variety of I/O devices, such as an input device 117 (such as a keyboard, a mouse, a touch screen, etc.), a media tray 121, and a multimedia interface. Transceiver 123 (can send and/or receive radio communication signals), camera 155 (can capture scene and dynamic digital video images) and external USB interface 125. Optionally, the interface connected to the I/O interface 115 may be a USB interface.

The processor 103 may be any conventional processor, including a reduced instruction set computing ("RISC") processor, a complex instruction set computing ("CISC") processor, or a combination of the foregoing. Alternatively, the processor may be a dedicated device such as an application specific integrated circuit ("ASIC"). Optionally, the processor 103 may be a neural network processor (Neural-network Processing Unit, NPU) or a combination of a neural network processor and the foregoing traditional processors. Optionally, the processor 103 is mounted with a neural network processor.

The computer system 101 can communicate with the software deployment server 149 through the network interface 129. The network interface 129 is a hardware network interface, such as a network card. The network 127 may be an external network, such as the Internet, or an internal network, such as an Ethernet or a virtual private network. Optionally, the network 127 may also be a wireless network, such as a WiFi network, a cellular network, and so on.

The hard disk drive interface is coupled to the system bus 105. The hardware drive interface is connected with the hard drive. The system memory 135 is coupled to the system bus 105. The data running in the system memory 135 may include the operating system 137 and application programs 143 of the computer system 101.

The operating system includes a shell (Shell) 139 and a kernel (kernel) 141. The shell 139 is an interface between the user and the kernel of the operating system. The shell 139 is the outermost layer of the operating system. The shell 139 manages the interaction between the user and the operating system: waiting for the user's input, interpreting the user's input to the operating system, and processing the output results of various operating systems.

The kernel 141 is composed of those parts of the operating system that are used to manage memory, files, peripherals, and system resources. Directly interact with the hardware. The operating system kernel usually runs processes and provides inter-process communication, providing CPU time slice management, interrupts, memory management, IO management, and so on.

The application program 141 includes programs related to automatic driving, such as programs that manage the interaction between the automatic driving device and obstacles on the road, programs that control the driving route or speed of the automatic driving device, and programs that control the interaction between the automatic driving device 100 and other automatic driving devices on the road. The application program 141 also exists on the system of a software deployment server (deploying server) 149. In one embodiment, when the application program 141 needs to be executed, the computer system 101 may download the application program 141 from the software deployment server 149.

The sensor 153 is associated with the computer system 101. The sensor 153 is used to detect the environment around the computer system 101. For example, the sensor 153 can detect animals, cars, obstacles, and crosswalks. Further, the sensor can also detect the environment around objects such as animals, cars, obstacles, and crosswalks. Other animals, weather conditions, the brightness of the surrounding environment, etc. Optionally, if the computer system 101 is located on the automatic driving device, the sensor may be a camera (ie, a camera), a laser radar, an infrared sensor, a chemical detector, a microphone, etc. The sensor 153 senses information at preset intervals when activated and provides the sensed information to the computer system 101 in real time or near real time. Optionally, the sensor may include a lidar, which can provide the acquired point cloud to the computer system 101 in real time or near real-time, and provide a series of acquired point clouds to the computer system 101, each time the acquired point cloud Corresponds to a timestamp. Optionally, the camera provides the acquired images to the computer system 101 in real time or near real time, and each frame of image corresponds to a time stamp. It should be understood that the computer system 101 can obtain an image sequence from a camera.

Optionally, in the various embodiments described above, the computer system 101 may be located far away from the automatic driving device, and may perform wireless communication with the automatic driving device. The transceiver 123 can send automatic driving tasks, sensor data collected by the sensor 153, and other data to the computer system 101; and can also receive control instructions sent by the computer system 101. The automatic driving device can execute the control instructions from the computer system 101 received by the transceiver, and perform corresponding driving operations. In other respects, some of the processes described in this article are executed on a processor installed in an autonomous vehicle, and others are executed by a remote processor, including taking actions required to perform a single manipulation.

The autopilot device updates the relative pose between the cameras deployed in the autopilot process in time. The follow-up will detail how the autopilot device updates the relative pose between the cameras deployed. In addition, the relative pose calibration method provided by the embodiments of the present application can be applied not only to automatic driving devices, but also to vehicles that are not deployed with automatic driving systems. The following describes the relative pose calibration method provided by the embodiment of the present application.

Fig. 3 is a flowchart of a relative pose calibration method provided by an embodiment of the application. As shown in Figure 3, the method includes:

301. The automatic driving device collects images through a first camera and a second camera.

The field of view of the first camera and the field of view of the second camera overlap, and the first camera and the second camera are installed at different positions of the automatic driving device. The self-driving device can be a self-driving car; it can also be a drone, or other devices that need to calibrate the relative pose between cameras.

302. The automatic driving device performs feature point matching on the image collected by the first camera and the image collected by the second camera to obtain a first matching feature point set.

The first matching feature point set includes H groups of feature point pairs, and each set of feature point pairs includes two matching feature points. One feature point is a feature point extracted from an image collected by the first camera, and the other feature point is The point is a feature point extracted from the image collected by the second camera, and the H is an integer not less than 8. The automatic driving device performs feature point matching on the image collected by the first camera and the image collected by the second camera to obtain the first matching feature point set. The feature point is matched to obtain the above-mentioned first matching feature point set. For example, the first camera collects image 1 at the first time point, and the second camera collects image 2 at the first time point; the automatic driving device extracts the feature points in image 1 to obtain the first image feature point set 1, and extracts The feature points in the image 2 obtain the image feature point set 2; then, the feature points in the image feature point set 1 and the image feature point set 2 are matched to obtain the matching feature point set (corresponding to the first matching feature point set).

In some embodiments, one way for the automatic driving device to implement step 302 is as follows: the automatic driving device uses the first camera and the second camera to synchronously acquire images to obtain the first image sequence and the second image sequence, wherein the first image The multiple images in the sequence are in one-to-one correspondence with the images in the second image sequence; feature point matching is performed on the one-to-one corresponding images in the first image sequence and the second image sequence to obtain the above-mentioned first matching feature point set . For example, image 1 to image 5 in the first image sequence corresponds to image 6 to image 10 in the second image sequence, and the automatic driving device can compare the feature points extracted from image 1 with those extracted from image 6. Perform feature matching on the feature points obtained, and add the obtained feature point pairs to the first matching feature point set; perform feature matching on the feature points extracted from the image 2 and the feature points extracted from the image 7, and the obtained features The point pair is added to the above-mentioned first matching feature point set; and so on. It should be understood that image 1 and image 6 are images acquired by the first camera and the second camera synchronously (that is, corresponding to the same frame of image), and image 2 and image 7 are images acquired by the first camera and the second camera synchronously (that is, corresponding to The same frame image), and so on. In practical applications, the automatic driving device can perform feature point matching on multiple frames of images synchronously collected by the first camera and the second camera, and store the obtained feature point pairs in the first matching feature point set. One frame of image may include one image collected by the first camera and one image collected by the second camera.

303. The automatic driving device determines the first relative pose between the first camera and the second camera according to the first matching feature point set.

The first matching feature point set includes H groups of feature point pairs, and each set of feature point pairs includes two matching feature points. One feature point is a feature point extracted from an image collected by the first camera, and the other feature point is The point is the feature point extracted from the image collected by the second camera, the field of view of the first camera overlaps the field of view of the second camera, and the first camera and the second camera are installed at different positions of the automatic driving device, The above H is an integer not less than 8.

304. When the difference between the first relative pose and the second relative pose is not greater than the pose change threshold, the automatic driving device updates the second relative pose to the first relative pose.

The second relative pose is the relative pose between the first camera and the second camera currently stored by the automatic driving device. The pose change threshold may be the maximum deviation threshold allowed by the normal operation of the multi-camera system, for example, the rotation angle deviation between the first relative pose and the second relative pose does not exceed 0.5 degrees. The method flow in FIG. 3 describes the manner of determining the relative pose between the first camera and the second camera with a common field of view. It should be understood that the automatic driving device may use a method similar to that in FIG. 3 to determine the relative pose between any two cameras with a common field of view. In the multi-camera vision system of the automatic driving device (corresponding to the camera 130 in Figure 1), two cameras at different installation positions generally have a common field of view. According to the distribution of the common field of view between the cameras, the common field of view can be The two cameras of the field of view form a pair (by default, the left camera is in the front and the right camera is in the back). FIG. 4 is a schematic diagram of a public field of view between cameras provided by an embodiment of the application. As shown in Figure 4, cameras with a common field of view include: camera 3 and camera 1, camera 3 and camera 2, and camera 1 and camera 2. Figure 4 only shows the common field of view between camera 1 and camera 2. In some embodiments, the cameras in the multi-camera system can be grouped in the above manner to obtain multiple pairs of cameras, that is, two cameras with a common field of view are grouped into one group; then, the method flow in FIG. 3 is used to obtain The relative pose between each pair of cameras.

The relative pose calibration method provided by the embodiments of the present application can achieve a repetition accuracy of less than 0.1°, effectively reduces the number of times of recalibrating the relative pose, and has high reliability.

FIG. 3 does not detail how to determine the first relative pose between the first camera and the second camera according to the first matching feature point set. Some optional implementations of step 303 are described below.

method one

According to the first matching feature point set, the automatic driving device determines the first relative pose between the first camera and the second camera in the following manner: determining the essential matrix between the first camera and the second camera; Based on the singular value decomposition result of the above essential matrix, calculate the 5-degree-of-freedom (DOF) relative pose between the above-mentioned first camera and the above-mentioned second camera; take the ratio between the first distance and the second distance as Scale factor; the first distance and the second distance are the results obtained by measuring the same distance by a non-visual sensor (such as lidar) and a visual sensor (such as the camera 130) on the automatic driving device, and the visual sensor includes the first The camera and/or the above-mentioned second camera; the above-mentioned 5-degree-of-freedom relative pose and the above-mentioned scale factor are combined to obtain the above-mentioned first relative pose.

Exemplarily, the automatic driving device may adopt the following steps to determine the essential matrix between the first camera and the second camera:

1) Acquire the feature point x′ extracted from the image collected by the first camera and the feature point x extracted from the image collected by the second camera from the first matching feature point set. For example, x=[x y 1] ^T , x′=[x′ y′ 1] ^T.

According to the nature of the basic matrix between the two cameras, the following formula holds:

x′Fx=0 (1);

Where F represents the basic matrix between the first camera and the second camera,

Expanding the above formula (1), we have:

x′xf ₁₁ +x′yf ₁₂ +x′f ₁₃ +y′xf ₂₁ +y′yf ₂₂ +y′f ₂₃ +xf ₃₁ +yf ₃₂ +f ₃₃ =0 (2);

Write the above formula in the form of a matrix:

(x′x,x′y,x′,y′x,y′y,y′,x,y,1) f=0 (3);

Among them, f=[f ₁₁ f ₁₂ f ₁₃ f ₂₁ f ₂₂ f ₂₃ f ₃₁ f ₃₂ f ₃₃ ] ^T.

The following matrix equations can be obtained for the characteristic point pairs of group g:

2) Solve the above matrix equation (4) to obtain the linear solution of the fundamental matrix F.

3) Use the following formula to perform singular value decomposition of the fundamental matrix F:

F=UDV ^T (5);

Where D is a diagonal matrix, except for diagonal elements, all are 0. The diagonal elements are represented by r, s, t, then: D = diag (r, s, t), and satisfies r ≥ s ≥ t . Let t=0 to obtain: F′=Udiag(r,s,0)V ^T , F′ is an optimal estimation of the fundamental matrix F that satisfies the singularity constraint.

4) Use the conversion relationship between the fundamental matrix and the essential matrix to calculate the essential matrix E:

E=K′ ^T FK (6);

Among them, K is the internal parameter matrix of the first camera, K′ is the internal parameter matrix of the second camera, and both K and K′ are known quantities. According to the above formula (6), the essential matrix E can be obtained.

Optionally, according to the singular value decomposition result of the essential matrix, the automatic driving device calculates the relative pose of the 5-degree-of-freedom DOF between the first camera and the second camera as follows:

(1) Use the following formula to perform singular value decomposition of the essential matrix to obtain the singular value decomposition result:

(2) According to the singular value decomposition result, at least two relative poses between the first camera and the second camera are obtained.

Assume that the pose of the first camera is P=Rt[I 0], where Rt is the transformation matrix of the first camera relative to the vehicle body coordinate system, and I is a 3*3 unit matrix. The four possible forms of the pose P′ of the second camera are as follows:

(3) Use the triangulation method to measure the distance of each feature point in the first matching feature point set, and make the three-dimensional coordinate positions of all the feature points located in front of the first camera and the second camera at the same time as the second camera. The pose.

Using the triangulation method to measure the distance of each feature point in the first matching feature point set may be: using a triangulation formula to determine a three-dimensional space coordinate according to each group of feature point pairs in the first matching feature point set. A three-dimensional space coordinate calculated from a set of feature point pairs is the space coordinates corresponding to the two feature points included in the set of feature point pairs. Triangulation was first proposed by Gauss and used in surveying. Simply put: Observe the same three-dimensional point P(x,y,z) at different positions, and know the two-dimensional projection points X1(x1,y1), X2(x2, y2), using the triangle relationship to recover the depth information of the three-dimensional point, that is, the three-dimensional space coordinates. Triangulation is mainly to calculate the three-dimensional coordinates of the feature points in the camera coordinate system through the matched feature points (ie, pixel points). After determining the expression form of P', the translation part (the last column) of P'is determined by

Ok, and

Is a normalized unit vector, so the translation part of P'differs from the true value by a scale factor. At this time, the 5-degree-of-freedom pose between the two cameras is determined. In this way, the 5-DOF relative pose between the first camera and the second camera can be accurately and quickly determined.

The relative pose between the two cameras has 6 degrees of freedom. The 5DOF relative pose between the two cameras can be obtained by decomposing the essential matrix, namely

Among them, the relative rotation angle between the two cameras can be accurately obtained by R, and the relative position between the two cameras can only obtain a normalized result, that is, a scale factor that differs from the true value. Determining the scale factor depends on the output of other sensors. Optionally, remember that a non-visual sensor obtains the measurement distance s, using the obtained relative pose of the camera, the same measurement distance s′ can be obtained through the multi-camera 3D reconstruction technology, then the scale factor is

The 6DOF relative pose between the two cameras is

A feasible solution for determining the scale factor is as follows: use lidar or millimeter wave radar to measure the distance s between a target and the autopilot device, and use the camera to use binocular (or multi-eye) vision measurement to get the same target to the automatic The distance s′ of the driving device to obtain the scale factor.

Another feasible solution for determining the scale factor is as follows: use an Inertial Measurement Unit (IMU) or wheel speed meter to measure the moving distance s of the automatic driving device, and use binocular (or multi-eye) visual measurement to obtain a certain static The distance difference s′ between the first distance between the target and the automatic driving device at the first moment and the second distance between the stationary target and the automatic driving device at the second moment can also obtain a scale factor. Wherein, the first time is the starting time of the moving distance s of the automatic driving device, and the second time is the ending time of the moving distance s of the automatic driving device.

Way two

The automatic driving device determines the first relative pose between the first camera and the second camera according to the first matching feature point set as follows: iteratively solves the target equation to obtain the first relative pose; in the target equation Wherein, the parameters included in the first relative pose are unknown numbers, and the feature point pairs in the first matching feature point set, the internal parameters of the first camera, and the internal parameters of the second camera are known numbers.

Assuming that the first matching feature points are concentrated, the feature point extracted from the image collected by the first camera is x′, and the feature point extracted from the image collected by the second camera is x. For example, x=[x y 1] ^T , x′=[x′ y′ 1] ^T.

The essential matrix E between the first camera and the second camera has the following equation:

Both sides of the equal sign of the above equation are the expressions of E, where K is the internal parameter matrix of the first camera, and K′ is the internal parameter matrix of the second camera,

R represents the rotation matrix between the first camera and the second camera, t represents the translation vector between the first camera and the second camera, and t=[t ₁ t ₂ t ₃ ] ^T. The basic matrix F between the first camera and the second camera has the following equation:

x′Fx=0 (13);

Using formula (12) to eliminate F in formula (13), the following target equation can be obtained:

Among them, K is the internal parameter matrix of the first camera, K′ is the internal parameter matrix of the second camera, K and K′ are both known quantities, and there are a total of 6 parameters in the target equation (that is, the 6 parameters that constitute the relative pose), Denoted as J, and its gradient as

Optionally, the gradient descent method is used to iteratively solve the target equation using the following formula to obtain the above-mentioned first relative pose:

Among them, α is the step length parameter, which can be set according to experience. Substitute x′ and x into formula (15), optimize and update

This step does not depend on the number of x'and x points. Even if the number of feature points in the first matching feature point set is small (such as in special scenes such as tunnels), it can still be

Optimize until a stable result is obtained.

In some embodiments, after the autonomous driving device obtains the relative pose between two cameras with a common field of view (for example, by executing the method flow in FIG. 1), it can use the difference between each pair of cameras with a common field of view. The closed loop composed of relative poses is used to construct a system of equations for overall optimization. The following uses camera 1 (ie, the first camera), camera 2 (ie, the second camera), and camera 3 (ie, the third camera) as examples to introduce how to optimize the obtained relative poses between the two cameras.

Suppose the pose between camera i and camera j is

Rt _ij represents the conversion matrix from camera i to camera j coordinate system, R _ij represents the rotation matrix between camera i and camera j,

Represents the translation matrix between camera i and camera j. Exemplarily, the pose relationship among camera 1, camera 2, and camera 3 is as follows:

Rt ₃₁ Rt ₂₃ = Rt ₂₁ (16);

Among them, three Rt form a closed loop, and better results can be obtained through overall optimization. The three conversion matrices can all be calculated separately through the public view relationship (for example, using the method flow in Figure 1). Therefore, formula (16) is a closed-loop equation. Better results can be obtained by optimizing the above equations. For a system composed of more cameras, as long as the two cameras have a common field of view and can form a closed loop, the above methods can be used for optimization. For example, camera a, camera b, camera c,..., camera z, and two adjacent cameras have a common field of view, then there is a closed-loop equation: Rt _ab Rt _bc Rt _cd …Rt _yz Rt _za =0. The following describes how to use the closed-loop equation formed by the transformation matrix between the cameras to optimize the relative pose between the cameras.

Taking camera 1 (ie the first camera), camera 2 (ie the second camera) and camera 3 (ie the third camera) as examples, the closed-loop equation composed of these three cameras is shown in formula (16), and formula (16) is expanded The following two equations can be obtained:

R ₃₁ R ₂₃ ＝R ₂₁ (17);

It should be understood that if R ₂₃ and R ₂₁ _{are known, R 31} can be obtained, and then can be used

R ₃₁ finds

In some embodiments, Rt ₃₁ can be regarded as an unknown quantity, Rt ₂₃ is the relative pose obtained by using matching feature points in a set of images (corresponding to a frame) synchronously captured by camera 2 and camera 3, Rt ₂₁ is the relative pose obtained by using matching feature points in a group of images taken by camera 2 and camera 1 simultaneously. Using formula (17) and formula (18), the following equations can be obtained:

Exemplarily, R ₂₁ is the first rotation matrix, R ₂₃ is the second rotation matrix,

Is the first translation matrix,

Is the second translation matrix. Assuming that the automatic driving device uses different sets of images to obtain M Rt ₂₃ and M Rt ₂₁ , the following equations can be obtained from formula (19) and formula (20):

It can be seen from the equation system (21) that R ₂₁ and R ₂₃ are known numbers,

Only related to R ₃₁ , can be solved by Newton's method or gradient descent method

Partially obtain the optimal solution of _{R 31.} Among them, the equation set (21) is the first equation set, and R ₃₁ is the third rotation matrix. It can be seen from the equation system (22) that

with

Is a known number, when R _{31 is} known,

Only with

Related to

get

The optimal solution. Among them, the equation system (22) is the second equation system,

Is the third translation matrix. _{In some embodiments, the automatic driving device may determine an Rt 21} using two images (corresponding to one frame) simultaneously collected by the camera 1 and the camera 2. _{It should be understood that the automatic driving device can determine M Rt 21} by using M groups of images (corresponding to M frames) collected synchronously. In the same way, the automatic driving device can determine M Rt ₂₃ .

Optionally, treat Rt ₃₁ as a known quantity, substitute the results obtained in the previous step into formulas (17) and (18), use a similar method to construct equations and treat Rt ₂₁ as the quantity to be optimized, you can get The optimal solution of Rt ₂₁ when Rt _{31 is known.}

Optionally, considering both Rt ₃₁ and Rt ₂₁ as known quantities, using a similar method to construct an equation set and Rt ₂₃ as the quantity to be optimized, the optimal solution of _{Rt 23 can be obtained.} At this point, a set of closed-loop optimization is completed. It should be understood that the main principle of this method is: first treat the rotation matrix between a pair of two cameras with a common field of view as an unknown number, and treat the rotation matrix and translation matrix between the other pairs of two cameras with a common field of view as an unknown number. Knowing the number, solve the unknown by equation (21); then the translation matrix between the pair of cameras with a common field of view is regarded as an unknown number, and the rotation matrix and translation matrix between the other pairs of cameras with a common field of view are regarded as A known number, the rotation matrix between the pair of cameras with a common field of view is regarded as a known number, and the unknown number is solved by equation (22).

It should be understood that the numbers between the cameras are arbitrarily designated, so the order of the cameras can be changed and the above steps can be repeated to obtain different solutions of poses between multiple sets of cameras, and a set of solutions with the smallest error is selected as the output. Take 4 cameras as an example, denoted as A, B, C, D. Assuming that each of the 4 cameras has a common field of view, the closed-loop equation has many forms, such as Rt _AD Rt _DC Rt _CB Rt _BA =0 , Rt _AD Rt _DB Rt _BC Rt _CA = 0, etc. Different closed-loop equations can be obtained respectively to obtain different solutions, and the optimal one can be selected. The error is the residual error of the equation.

In practical applications, after the automatic driving device obtains the relative poses between the two cameras, it can optimize the relative poses between the two cameras in a similar way, so as to obtain a more accurate relative pose. posture.

The foregoing embodiment describes the manner of determining the relative pose between the first camera and the second camera by using two images synchronously collected by the first camera and the second camera. When the number of feature point pairs that match the two images simultaneously collected by the first camera and the second camera is not less than 8, a relatively accurate relative pose can be obtained. However, when the number of matching feature point pairs in the two images simultaneously collected by the first camera and the second camera is less than 8, an accurate relative pose cannot be obtained, that is, the pose estimation is unstable. The following introduces a way to determine the relative pose between two cameras through the accumulation of multiple frames. In this way, a more accurate relative pose can be obtained. A way to determine the relative pose between two cameras through multi-frame accumulation is as follows:

1) Extract the feature points in the two images of the i-th frame, denoted as x′ _i and x _i , respectively, and add them to the feature point set X ₁ and the feature point set X ₂ .

The two images of the i-th frame include an image collected by the first camera in the i-th frame and an image collected by the second camera in the i-th frame. Extracting the feature points in the two images of the i-th frame can be extracting the feature points in an image collected by the first camera in the i-th frame to obtain x′ _i ; extracting the feature points in an image collected by the second camera in the i-th frame The feature points of, get x _i . i is an integer greater than zero. It should be understood that the two images in one frame include one image collected by the first camera and one image collected by the second camera, and the time interval between the two images in this frame is less than a time threshold, such as 0.5 ms, 1 ms, and so on.

2) Extract the feature points in the two images of the j-th frame, denoted as x′ _j and x _j , respectively, and add them to the feature point set X ₁ and the feature point set X ₂ .

j is an integer greater than 1 and not equal to i. It should be understood that after the feature points in the two images of N frames are continuously extracted, the feature point set X ₁ and the feature point set X ₂ store the matching feature points in the two images of the N frame.

3) Use the feature point set X ₁ and the feature point set X ₂ to estimate the relative pose of the two cameras.

Optionally, an implementation manner of step 3) may be to _{perform feature point matching on the feature points in the feature point set X 1} and the feature points in the feature point set X ₂ to obtain multiple sets of feature point pairs; Set the feature point pairs to determine the relative pose between the two cameras. Using the multiple sets of feature point pairs, the implementation manner of determining the relative pose between the two cameras may be similar to the implementation manner of step 303.

Since the number of feature points in the feature point sets X ₁ and X ₂ is far more than the number of feature points in a single frame, the iterative optimization method can be used to estimate the relative pose of the two cameras: first use the feature point sets X ₁ and X ₂ Perform pose estimation for all points to obtain the first intermediate pose; then, substitute the first intermediate pose into formula (14) to calculate the residual, and remove the feature points whose residuals are greater than the residual threshold; then use the remaining features Point pose estimation is performed again to obtain the second intermediate pose; the above process is repeated until the number of feature points to be eliminated is less than the number threshold (such as 5% of the total). The unit of formula (14) is pixel, and the residual threshold can be 0.8, 1, 1.2, etc., which is not limited in this application. Finally, a more accurate relative pose under multiple frames of images can be obtained. The implementation of the pose estimation here can be the same as that of step 303.

The central limit theorem shows that the mean of a large number of mutually independent random variables converges to a normal distribution. Suppose that in a long-term observation, the R and calculated between different frames

Are independent and identically distributed. Record the 5DOF relative pose obtained by solving the i-th frame as

According to the central limit theorem, remember

Then there are:

Where N(0,1) represents the standard normal distribution. remember

As you can see, the variance of X is

Inversely proportional. n is the size of the time window (corresponding to the n frames of images collected). As the time window increases, the variance of X can continue to decrease, and it will stabilize at a certain value in practical applications. According to the central limit theorem, it is assumed that in a long-term observation, the R and R calculated between different frames

Are independent and identically distributed, then as n increases, R and

Will stabilize. Choose an appropriate n such that R and

The variance of satisfies the demand.

Using the method of multi-frame accumulation to determine the relative pose between the two cameras can avoid the jitter problem caused by single-frame calculation and stabilize the calibration result.

The following describes how to realize the recalibration of the relative pose between the two cameras in conjunction with the structure of the automatic driving device. FIG. 5 is a schematic structural diagram of an automatic driving device provided by an embodiment of the application. As shown in Figure 5, the automatic driving device includes: an image acquisition unit (corresponding to the first camera and the second camera) 501, an image feature point extraction unit 502, a pose calculation unit 503, a closed loop optimization unit 504, and a multi-frame accumulation optimization unit 505, a scale calculation unit 506, and a calibration parameter update unit 507. Among them, the closed-loop optimization unit 504 and the multi-frame accumulation optimization unit 505 are optional, but not necessary. In some embodiments, the functions of the image feature point extraction unit 502, the pose calculation unit 503, the closed-loop optimization unit 504, the multi-frame accumulation optimization unit 505, the scale calculation unit 506, and the calibration parameter update unit 507 are all performed by the processor 112 (corresponding to On-board processor). In some embodiments, the functions of the pose calculation unit 503, the closed loop optimization unit 504, the multi-frame accumulation optimization unit 505, the scale calculation unit 506, and the calibration parameter update unit 507 are all implemented by the processor 112 (corresponding to the on-board processor); The function of the image feature point extraction unit 502 is implemented by a graphics processor. The image feature point extraction unit 502, the pose calculation unit 503, the closed loop optimization unit 504, the multi-frame accumulation optimization unit 505, the scale calculation unit 506, and the calibration parameter update unit 507 are used in the automatic driving device to determine the relative relationship between two cameras. The unit of the pose, that is, the online calibration part (the part used to realize the relative pose recalibration). The functions of each unit are introduced below.

The image acquisition unit 501 is used to acquire images collected by multiple cameras. Exemplarily, the image acquisition unit is used to acquire images synchronously acquired by each camera. The multiple cameras correspond to the camera 130 in FIG. 1.

The image feature point extraction unit 502 is configured to extract feature points in images collected by multiple cameras, and match the extracted feature points to obtain a first matching feature point set. For example, the image feature point extraction unit is configured to extract feature points in a first image collected by a first camera to obtain a first feature point set, and extract feature points in a second image collected by a second camera to obtain a first feature point set. Two feature point sets: the feature points in the first feature point set and the feature points in the second feature point set are matched to obtain multiple sets of feature point pairs (corresponding to the first matching feature point set). For another example, the image feature point extraction unit is used to extract feature points in multiple sets of images synchronously collected by the first camera and the second camera, and perform feature point matching to obtain multiple sets of feature point pairs (corresponding to the first Matching feature point set), each group of images (corresponding to a frame) includes an image collected by the first camera and an image collected by the second camera.

The pose calculation unit 503 is configured to determine the basic matrix between the two cameras according to the first matching feature point set (ie, the matched feature points), and decompose the basic matrix to obtain a 5DOF relative pose. It should be understood that the pose calculation unit may perform pose calculation on all cameras with a common field of view in the multi-camera system, that is, determine the relative poses between all two cameras with a common field of view in the multi-camera system. In the case that the automatic driving device does not include the closed-loop optimization unit 505 and the multi-frame accumulation optimization unit 506, the pose calculation unit 503 is also used to combine the scale factor and the 5DOF relative pose to obtain a complete 6DOF relative pose. Exemplarily, the pose calculation unit 503 is specifically configured to determine the essential matrix between the first camera and the second camera; according to the singular value decomposition result of the essential matrix, calculate the difference between the first camera and the second camera. Relative pose between 5 degrees of freedom DOF. Optionally, the pose calculation unit 503 is specifically configured to perform singular value decomposition on the essential matrix to obtain the singular value decomposition result; according to the singular value decomposition result, obtain at least the difference between the first camera and the second camera. Two relative poses; each of the above at least two relative poses is used to calculate the three-dimensional coordinate position of the feature point in the first matching feature point set; each of the at least two relative poses that makes the first matching feature point set is The relative poses in which the three-dimensional coordinate positions of the feature points are located in front of the first camera and the second camera are regarded as the 5-degree-of-freedom relative pose.

The scale calculation unit 504 is configured to use the ratio between the first distance and the second distance as a scale factor; the first distance and the second distance are respectively the same distance measured by the non-visual sensor and the visual sensor on the automatic driving device As a result, the visual sensor includes the first camera and/or the second camera. Exemplarily, the scale calculation unit 504 is specifically configured to use lidar or millimeter wave radar to measure the distance s of a certain target from the automatic driving device, while using a camera to use binocular (or multi-eye) vision measurement to obtain the same target to the automatic driving The distance s'of the device to obtain the scale factor. It should be understood that combining the scale factor and the 5DOF relative pose can obtain a complete 6DOF relative pose.

The closed-loop optimization unit 505 is used to construct a system of equations based on the closed-loop composed of the relative poses of the two cameras in the multi-camera system to perform overall optimization. Exemplarily, the closed-loop optimization unit 505 optimizes the relative pose obtained by executing the method flow in FIG. 3 by solving the equation system (21) and the equation system (21) successively. In the case that the automatic driving device includes the closed-loop optimization unit 505 and does not include the multi-frame accumulation optimization unit 506, the closed-loop optimization unit 505 is also used to combine the scale factor and the 5DOF relative pose to obtain a complete 6DOF relative pose.

The multi-frame accumulation optimization unit 506 is configured to accumulate, filter and optimize the multi-frame calculation results to obtain stable calibration parameters (corresponding to the relative pose). Exemplarily, the multi-frame accumulation optimization unit 406 is used to implement a manner of determining the relative pose between two cameras through multi-frame accumulation. In the case that the automatic driving device includes the multi-frame accumulation optimization unit 505, the multi-frame accumulation optimization unit 505 may combine the scale factor and the 5DOF relative pose to obtain a complete 6DOF relative pose.

The calibration parameter update unit 507 is used to change the relative pose currently used by the automatic driving device when the difference between the calculated relative pose and the relative pose currently used by the automatic driving device is not greater than the pose change threshold. Update to the calculated relative pose. In other words, the calibration parameter update unit 507 is used to compare the calculated relative pose (corresponding to the calibration parameter) with the relative pose currently used by the automatic driving device, and if the difference is too large, dynamically update the currently used Relative pose (or reminder at the same time).

The automatic driving device in FIG. 5 can use method one to determine the relative pose between two cameras. The following introduces the structure of an automatic driving device that can adopt the second method to determine the relative pose between two cameras. Fig. 6 is a schematic structural diagram of another automatic driving device provided by an embodiment of the application. As shown in Figure 6, the automatic driving device includes:

The image acquisition unit 601 is used to acquire images through the first camera and the second camera respectively; the field of view of the first camera and the field of view of the second camera overlap, and the first camera and the second camera are installed in the automatic driving device different positions;

The image feature point extraction unit 602 is configured to perform feature point matching on the image collected by the first camera and the image collected by the second camera to obtain a first matching feature point set; the first matching feature point set includes H groups of features Point pairs, each set of feature point pairs includes two matching feature points, one of the feature points is a feature point extracted from the image collected by the above-mentioned first camera, and the other feature point is a feature point extracted from the image collected by the above-mentioned second camera For the extracted feature points, the above H is an integer not less than 8;

The pose calculation unit 603 is configured to determine the first relative pose between the first camera and the second camera according to the first matching feature point set; the first matching feature point set includes H groups of feature point pairs, each The set of feature point pairs includes two matching feature points, one of the feature points is the feature point extracted from the image captured by the first camera, and the other feature point is the feature point extracted from the image captured by the second camera. , The field of view of the first camera and the field of view of the second camera overlap, the first camera and the second camera are installed at different positions of the automatic driving device, and the above H is an integer not less than 8;

The calibration parameter update unit 604 (corresponding to the calibration parameter update unit 507) is configured to: when the difference between the first relative pose and the second relative pose is not greater than the pose change threshold, the second relative The pose is updated to the first relative pose; the second relative pose is the relative pose between the first camera and the second camera currently stored by the automatic driving device.

In some embodiments, the functions of the pose calculation unit 603 and the calibration parameter update unit 604 are both implemented by the processor 112 (corresponding to the on-board processor).

Optionally, the automatic driving device in FIG. 6 further includes: a closed-loop optimization unit 605 and a multi-frame accumulation optimization unit 606. Among them, the closed-loop optimization unit 605 and the multi-frame accumulation optimization unit 606 are optional, but not necessary. The function of the image acquisition unit 601 can be the same as that of the image acquisition unit 501, the function of the image feature point extraction unit 602 can be the same as the function of the image feature point extraction unit 502, and the function of the closed loop optimization unit 605 can be the same as that of the closed loop optimization unit 505. Similarly, the function of the multi-frame accumulation optimization unit 606 may be the same as the function of the multi-frame accumulation optimization unit 506.

Exemplarily, the pose calculation unit 603 is specifically configured to iteratively solve the target equation to obtain the first relative pose; in the target equation, the parameter included in the first relative pose is an unknown number, and the first matching feature point is The concentrated feature point pairs, the internal parameters of the first camera, and the internal parameters of the second camera are known numbers. It should be understood that the pose calculation unit 603 may adopt the second method to determine the relative pose between two cameras with a common field of view.

It should be understood that the division of each unit in the above automatic driving device is only a division of logical functions, and may be fully or partially integrated into a physical entity in actual implementation, or may be physically separated. For example, the above units can be separately set up processing elements, or they can be integrated into a certain chip of the automatic driving device for implementation. In addition, they can also be stored in the storage element of the controller in the form of program code, which is determined by a certain part of the processor. A processing element calls and executes the functions of each of the above units. In addition, each unit can be integrated together or implemented independently. The processing element here can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software. The processing element may be a general-purpose processor, such as a central processing unit (English: central processing unit, CPU for short), or one or more integrated circuits configured to implement the above methods, for example: one or more specific integrated circuits Circuit (English: application-specific integrated circuit, abbreviation: ASIC), or, one or more microprocessors (English: digital signal processor, abbreviation: DSP), or, one or more field programmable gate arrays (English: field-programmable gate array, referred to as FPGA), etc.

FIG. 7 is a schematic structural diagram of another automatic driving device provided by an embodiment of the application. As shown in FIG. 7, the automatic driving device includes: a multi-camera vision system 701, an image processor 702, an image feature extractor 703, an on-board processor 704, a memory 705, a calibration parameter update device 706, and a parameter abnormality reminder 707.

The multi-camera vision system 701 is used to synchronize images collected by multiple cameras. The multi-camera vision system 701 corresponds to the camera 130 in FIG. 1. The multi-camera vision system consists of multiple cameras, which can collect image data synchronously.

The image processor 702 is used to perform preprocessing such as scaling on the image data collected by the multi-camera vision system 701.

The image feature extractor 703 is configured to perform feature point extraction on the image preprocessed by the image processor 702.

In order to increase the speed, the preprocessed image can enter the special image feature extractor 703 for feature point extraction. The image feature extractor 703 may be a graphics processor with general computing capabilities or a specially designed ISP or other hardware.

The on-board processor 704 is configured to determine the relative pose between the two cameras according to the feature points extracted by the image feature extractor 703.

The on-board processor (corresponding to the processor 113 in FIG. 1) may be a hardware platform with general computing capabilities. Optionally, the on-board processor includes a pose estimation part and an optimization processing part. The pose estimation part (corresponding to the pose calculation unit 601 in FIG. 6 or the pose calculation unit 503 and the scale calculation unit 504 in FIG. 5) is used to determine the relative pose between the two cameras, that is, the relative pose estimation . The optimization processing part is used to implement closed-loop constrained optimization and/or multi-frame accumulation optimization. Optionally, the on-board processor includes a pose estimation part and does not include an optimization processing part. The on-board processor is also used to read calibration parameters (that is, the currently stored relative pose) from the memory 705 (corresponding to the memory 114 in FIG. 1), and to compare the read calibration parameters (that is, the relative pose) and Calculated calibration parameters (ie relative pose). After that, the on-board processor sends the comparison results to the calibration parameter update device 706 and the parameter abnormality reminder 707 respectively. The calibration parameter update device 706 (corresponding to the calibration parameter update unit 507 in FIG. 5 and the calibration parameter update unit 602 in FIG. 6) can update the calibration parameters in the memory as required. The parameter abnormality reminding device 707 can remind the driver of the abnormality of the calibration parameters of the camera by sound or image. For example, when the in-vehicle processor determines that the relative pose between the first camera and the second camera differs greatly from the relative pose between the first camera and the second camera currently used by the automatic driving device, it reminds the driver of the first camera and the second camera. The relative pose between the camera and the second camera is abnormal. Take the on-board application scenario as an example. If the calibration parameters calculated by the on-board processor are too far from the parameters calibrated when the car leaves the factory, if the angle deviation exceeds 0.3 degrees, it can be considered that the sensor is loosely installed, which may cause the performance of the automatic driving system to decrease. Need to remind the owner to go to the 4s shop to re-calibrate (such as the pop-up warning window of the central control). If the vehicle owner has not recalibrated for a long time and the angle deviation is not large, such as greater than 0.3 degrees but less than 0.5 degrees, the autopilot system can be operated with online calibration parameters instead of factory calibration parameters with the owner's permission.

The automatic driving device in FIG. 7 can use method one to determine the relative pose between two cameras, or method two to determine the relative pose between two cameras.

FIG. 8 is a flowchart of another relative pose calibration method provided by an embodiment of the application. The method in Figure 8 is a further refinement and improvement of the method in Figure 3. As shown in Figure 8, the method includes:

801. The automatic driving device determines a pair of cameras with a common field of view.

802. Extract feature points from an image synchronously collected by a pair of cameras, and perform feature point matching.

For example, the first camera and the second camera have a common field of view, and the first image and the second image are the images (corresponding to one frame) collected by the first camera and the second camera synchronously, and the first image is extracted Feature points to obtain a third feature point set, and extract feature points in the second image to obtain a fourth feature point set; match the feature points in the third feature point set with the feature points in the fourth feature point set To get multiple sets of feature point pairs. It should be understood that multiple sets of feature point pairs corresponding to the pair of cameras can be obtained by using the images synchronously collected by each pair of cameras.

803. Calculate an essential matrix between the pair of cameras through multiple sets of feature point pairs corresponding to the pair of cameras.

804. Determine the 5DOF relative pose between the pair of cameras through the essential matrix between the pair of cameras.

The implementation manner of step 803 and step 804 may be the same as the manner of determining the 5DOF relative pose between a pair of cameras with a common field of view in the first manner, and will not be described in detail here. Optionally, step 802 to step 804 may be replaced with the method of multi-frame accumulation in the foregoing embodiment to determine the relative pose between the two cameras. It should be understood that each time the automatic driving device executes steps 801 to 804, a 5DOF relative pose between a pair of cameras with a common field of view can be determined.

805. Construct a closed-loop equation for a multi-camera system.

Exemplarily, equation (19) and equation (20) are a set of closed-loop equations of a multi-camera system constructed.

806. Solve a solution conforming to the closed-loop equation of the multi-camera system.

Steps

805 and 806 are to construct an equation set using a closed loop composed of the relative poses between the pairs of cameras with a common field of view to optimize the calculated 5DOF relative poses between the two cameras. The foregoing embodiment describes the method of constructing a system of equations using a closed loop composed of the relative poses between pairs of cameras with a common field of view to perform overall optimization.

807. Calculate the scale factor, and combine the 5DOF relative pose and the scale factor to obtain a 6DOF relative pose.

808. Determine whether to update the currently used calibration parameter (ie, relative pose).

If yes, go to step 809; if not, go to step 810.

809. Update the currently used calibration parameters.

Updating the currently used calibration parameters may be updating the currently used calibration parameters to the calculated relative pose (ie, new calibration parameters).

810. Output information used to remind the driver of abnormalities in the calibration parameters of the camera.

811. End this process.

In the embodiment of this application, the public field of view between the two cameras is used to construct the pose constraint equation between the cameras, such as formula (4). The 5DOF relative pose of the two cameras is obtained by decomposing the essential matrix, making full use of The public field of view information between the cameras. In addition, using feature point matching technology to obtain 5DOF pose information and optimize it through closed-loop equations, a higher recalibration accuracy is achieved.

The foregoing embodiments describe the manner in which the automatic driving device determines the relative pose between the two cameras. In some embodiments, the automatic driving device may send the information needed to determine the relative pose between the two cameras to the server, and the server determines the pair of cameras on the automatic driving device according to the information sent by the automatic driving device. The relative pose between. The following introduces a solution for determining the relative pose between the two cameras on the automatic driving device by the server.

FIG. 9 is a flowchart of another relative pose calibration method provided by an embodiment of the application. As shown in Figure 9, the method includes:

901. The automatic driving device sends a pose calibration request to the server.

Optionally, the pose calibration request is used to request recalibration of the relative pose between the first camera and the second camera of the automatic driving device, and the pose calibration request carries the internal parameters of the first camera and the The internal parameters of the second camera. Optionally, the pose calibration request is used to request recalibration of the relative poses between the cameras on the automatic driving device, and the pose calibration request carries internal parameters of the cameras on the automatic driving device.

902. The server sends confirmation information for the pose calibration request to the automatic driving device.

When the server determines to accept the pose calibration request from the automatic driving device, it sends confirmation information for the pose calibration request to the automatic driving device. Optionally, if the server determines that the autonomous driving device is an authorized device according to the pose calibration request, it accepts the pose calibration request from the automatic driving device, and sends confirmation information for the pose calibration request to the automatic driving device. The authorization device refers to an automatic driving device to which the server needs to provide a pose calibration service (ie, recalibrate the relative pose).

903. The automatic driving device sends recalibration reference information to the server.

Optionally, the recalibration reference information is used by the server to determine the relative pose between the first camera and the second camera of the automatic driving device, the field of view of the first camera and the field of view of the second camera overlap, and the first camera The first camera and the aforementioned second camera are installed at different positions of the automatic driving device. Exemplarily, the recalibration reference information includes images acquired synchronously by the first camera and the second camera, or multiple pairs of characteristic point pairs obtained by matching the characteristic points of the automatic driving device according to the synchronously acquired images of the first camera and the second camera. (Corresponding to the first matching feature point set). Optionally, the above-mentioned recalibration reference information is used by the server to determine the relative poses between the cameras on the above-mentioned automatic driving device, and the above-mentioned cameras are installed in different positions of the automatic driving device. Exemplarily, the recalibration reference information includes images synchronously collected by each camera or multiple pairs of feature points obtained by matching feature points of the automatic driving device according to the images synchronously collected by each pair of cameras.

904. The server obtains a first matching feature point set according to the recalibration reference information.

Optionally, the aforementioned first matching feature point set includes H groups of feature point pairs, and each group of feature point pairs includes two matching feature points, one of which is a feature point extracted from an image collected by the first camera, The other feature point is the feature point extracted from the image collected by the second camera, and the above H is an integer not less than 8. Optionally, the aforementioned recalibration reference information includes a first matching feature point set. Optionally, the above-mentioned recalibration reference information includes images synchronously collected by the first camera and the second camera, and the server may perform feature point matching on the images synchronously collected by the first camera and the second camera to obtain the first matching feature point set. .

905. The server determines the first relative pose between the first camera and the second camera according to the first matching feature point set.

The implementation of step 905 may be the same as the implementation of step 303. The server can determine the first relative position between the first camera and the second camera in either way one or two, or it can use other ways to determine the first relative position between the first camera and the second camera. The posture is not limited in the embodiment of this application.

906. The server sends the first relative pose to the automatic driving device.

907. The automatic driving device updates the second relative pose to the first relative pose when the difference between the first relative pose and the second relative pose is not greater than the pose change threshold.

The second relative pose is the relative pose between the first camera and the second camera currently stored by the automatic driving device. It should be understood that the server may determine the relative pose between the two cameras on the automatic driving device according to the information sent by the automatic driving device. That is to say, the manner in which the automatic driving device in the foregoing embodiment determines the relative pose between the two cameras can be implemented by a server, for example, using a closed loop composed of the relative poses between the two cameras with a common field of view. Construct a set of equations, optimize the calculated 5DOF relative pose between the two cameras, and use the multi-frame accumulation method to determine the relative pose between the two cameras.

Fig. 10 is a schematic structural diagram of a server provided by an embodiment of the application. As shown in Fig. 10, the server includes: a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004; The interface 1003 realizes the communication connection between each other through the bus 1004. The communication interface 1003 is used for data interaction with the automatic driving device.

The processor 1003 reads the code stored in the memory to perform the following operations: receiving recalibration reference information from the automatic driving device, and the recalibration reference information is used by the server to determine the first camera and the first camera of the automatic driving device. The relative pose between the second cameras, the field of view of the first camera overlaps the field of view of the second camera, and the first camera and the second camera are installed in different positions of the automatic driving device; according to the recalibration reference Information, the first matching feature point set is obtained. The first matching feature point set includes H groups of feature point pairs, and each set of feature point pairs includes two matching feature points, one of which is an image collected from the first camera The other feature point is the feature point extracted from the image collected by the second camera, and the above H is an integer not less than 8; according to the first matching feature point set, the first camera and the first camera are determined The first relative pose between the two cameras; the first relative pose is sent to the automatic driving device.

The embodiments of the present application also provide a computer-readable storage medium. The above-mentioned computer-readable storage medium stores instructions, which when run on a computer, cause the computer to execute the relative pose calibration method provided in the foregoing embodiments.

Optionally, the above instructions can be implemented when running on a computer: determine the first relative pose between the first camera and the second camera according to the first matching feature point set; the first matching feature point set includes H Set of feature point pairs, each set of feature point pairs includes two matching feature points, one of the feature points is a feature point extracted from the image collected by the above-mentioned first camera, and the other feature point is a feature point collected from the above-mentioned second camera Feature points extracted from the image, the field of view of the first camera and the field of view of the second camera overlap, the first camera and the second camera are installed at different positions of the automatic driving device, and the above H is an integer not less than 8; In the case that the difference between the first relative pose and the second relative pose is not greater than the pose change threshold, the second relative pose is updated to the first relative pose; the second relative pose Is the relative pose between the first camera and the second camera currently stored by the automatic driving device.

Optionally, the above instructions can be implemented when running on a computer: receiving recalibration reference information from the automatic driving device, and the above recalibration reference information is used by the server to determine the relationship between the first camera and the second camera of the automatic driving device. With respect to the pose, the field of view of the first camera overlaps the field of view of the second camera, and the first camera and the second camera are installed in different positions of the automatic driving device; according to the recalibration reference information, a first match is obtained Feature point set, the above-mentioned first matching feature point set includes H groups of feature point pairs, each set of feature point pairs includes two matching feature points, one of which is a feature point extracted from an image collected by a first camera, The other feature point is the feature point extracted from the image collected by the second camera, and the above H is an integer not less than 8; according to the first matching feature point set, the first camera and the second camera are determined. A relative pose; sending the first relative pose to the automatic driving device.

The embodiments of the present application provide a computer program product containing instructions, which when run on a computer, cause the computer to execute the relative pose calibration method provided in the foregoing embodiments.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited to this. Anyone familiar with the technical field can easily think of various equivalents within the technical scope disclosed in the present invention. Modifications or replacements, these modifications or replacements should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

A relative pose calibration method, which is characterized in that it includes:

Images are collected by a first camera and a second camera; the field of view of the first camera and the field of view of the second camera overlap, and the first camera and the second camera are installed in different positions of the automatic driving device;

Perform feature point matching on the image collected by the first camera and the image collected by the second camera to obtain a first matching feature point set; the first matching feature point set includes H groups of feature point pairs, each group of features The point pair includes two matching feature points. One feature point is a feature point extracted from the image collected by the first camera, and the other feature point is a feature point extracted from the image collected by the second camera. , The H is an integer not less than 8;

Determine a first relative pose between the first camera and the second camera according to the first matching feature point set;

In the case that the difference between the first relative pose and the second relative pose is not greater than the pose change threshold, the second relative pose is updated to the first relative pose; the first The second relative pose is the relative pose between the first camera and the second camera currently stored by the automatic driving device.
The method according to claim 1, wherein the determining the first relative pose between the first camera and the second camera according to a first matching feature point set comprises:

Determining the essential matrix between the first camera and the second camera;

According to the singular value decomposition result of the essential matrix, calculate the relative pose of the 5-DOF DOF between the first camera and the second camera;

The ratio between the first distance and the second distance is used as a scale factor; the first distance and the second distance are the results obtained by measuring the same distance by the non-visual sensor and the visual sensor on the automatic driving device, respectively, The vision sensor includes the first camera and/or the second camera;

Combine the 5-degree-of-freedom relative pose and the scale factor to obtain the first relative pose.
The method according to claim 2, wherein the singular value decomposition result based on the essential matrix is calculated to obtain the relative pose of the 5-degree-of-freedom DOF between the first camera and the second camera include:

Performing singular value decomposition on the essential matrix to obtain the singular value decomposition result;

Obtaining at least two relative poses between the first camera and the second camera according to the singular value decomposition result;

Respectively calculating the three-dimensional coordinate positions of the feature points in the first matching feature point set by using the at least two relative poses;

In the at least two relative poses, the three-dimensional coordinate positions of the feature points in the first matching feature point set are all located in front of the first camera and the second camera as the 5 free poses. Degree relative pose.
The method according to claim 1, wherein the determining the first relative pose between the first camera and the second camera according to a first matching feature point set comprises:

The target equation is solved iteratively to obtain the first relative pose; in the target equation, the parameters included in the first relative pose are unknowns, and the pair of feature points in the first matching feature point set, the first The internal parameters of a camera and the internal parameters of the second camera are known numbers.
The method according to any one of claims 1 to 4, wherein the automatic driving device is equipped with M cameras, and the M cameras include the first camera, the second camera, and the third camera The field of view of the third camera overlaps the field of view of the first camera and the field of view of the second camera, and the M is an integer greater than 2; the method further includes:

Obtain M first rotation matrices, M second rotation matrices, M first translation matrices, and M second translation matrices; the M first rotation matrices are one of the first camera and the second camera And at least two of the M first rotation matrices are different, the M second rotation matrices are the rotation matrices between the second camera and the third camera, and the M At least two of the second rotation matrices are different, the M first translation matrices are the translation matrices between the first camera and the second camera, and at least two of the M first translation matrices are Different translation matrices, the M second translation matrices are the translation matrices between the second camera and the third camera, and at least two of the M second translation matrices are different, the M One first rotation matrix corresponds to the M first translation matrices, the M second rotation matrix corresponds to the M second translation matrix one to one, and the M is an integer greater than 1;

Solve the first equation set to obtain the third rotation matrix; the first equation set includes M first equations, and the M first equations correspond to the M first rotation matrices one-to-one, and correspond to the M first rotation matrices. The M second rotation matrices have a one-to-one correspondence; in each first equation, the first rotation matrix and the second rotation matrix are known numbers, and the third rotation matrix is an unknown number; the third rotation matrix is the A rotation matrix between the first camera and the third camera;

Solve the second equation set to obtain the third translation matrix; the second equation set includes M second equations, and the M second equations correspond to the M first rotation matrices one-to-one, and correspond to the M first rotation matrices. The M second rotation matrices have a one-to-one correspondence; in each second equation, the first rotation matrix, the second rotation matrix, the first translation matrix, and the second translation matrix are known numbers, and the third translation matrix is an unknown number ; The third translation matrix is the translation matrix between the first camera and the third camera;

The pose including the third rotation matrix and the third translation matrix is taken as the relative pose between the first camera and the third camera.
The method according to claim 1, wherein the first set of matching feature points includes feature points extracted by the first camera from at least two frames of images, and the second camera from at least two frames of images. The extracted feature points; the determining the first relative pose between the first camera and the second camera according to the first matching feature point set includes:

Determining the relative pose between the first camera and the second camera according to the feature point pairs in the first matching feature point set to obtain a first intermediate pose;

Substituting each feature point pair in the first matching feature point set into a first formula to obtain the residual error corresponding to each feature point pair in the first matching feature point set, and the first formula representation includes the first intermediate Posture

The interference feature point pair in the first matching feature point set is removed to obtain a second matching feature point set; the interference feature point pair is the feature point pair whose corresponding residual in the first matching feature point set is greater than the residual threshold ；

Determining the relative pose between the first camera and the second camera according to the feature point pairs in the second matching feature point set to obtain a second intermediate pose;

The target intermediate pose is taken as the first relative pose between the first camera and the second camera; the target intermediate pose is the first determined according to the feature point pair in the target matching feature point set The relative pose between a camera and the second camera, the number of feature point pairs in the target matching feature point set is less than a number threshold or the number of feature point pairs in the target matching feature point set is equal to the The ratio of the number of feature point pairs in the first matching feature point set is less than the ratio threshold.
The method according to any one of claims 1 to 6, wherein the method further comprises:

When the difference between the first relative pose and the second relative pose is not greater than the pose change threshold, output reminder information; the reminder information is used to remind the first camera and The relative pose between the second cameras is abnormal.
An automatic driving device, characterized in that it comprises:

The image acquisition unit is used to acquire images through a first camera and a second camera respectively; the field of view of the first camera and the field of view of the second camera overlap, and the first camera and the second camera are installed in an automatic Different positions of the driving gear;

The image feature point extraction unit is configured to perform feature point matching on the image collected by the first camera and the image collected by the second camera to obtain a first matching feature point set; the first matching feature point set includes H Group of feature point pairs, each group of feature point pairs includes two matching feature points, one of the feature points is a feature point extracted from the image collected by the first camera, and the other feature point is from the second camera For feature points extracted from the collected image, the H is an integer not less than 8;

A pose calculation unit, configured to determine a first relative pose between the first camera and the second camera according to the first matching feature point set; the first matching feature point set includes H groups of features Point pairs, each set of feature point pairs includes two matching feature points, one of the feature points is a feature point extracted from the image collected by the first camera, and the other feature point is a feature point collected from the second camera Feature points extracted from the image, the field of view of the first camera and the field of view of the second camera overlap, the first camera and the second camera are installed in different positions of the automatic driving device, and the H is not Integer less than 8;

The calibration parameter update unit is configured to update the second relative pose to the first when the difference between the first relative pose and the second relative pose is not greater than the pose change threshold. Relative pose; the second relative pose is the relative pose between the first camera and the second camera currently stored by the automatic driving device.
The device according to claim 8, wherein:

The pose calculation unit is specifically configured to determine the essential matrix between the first camera and the second camera; calculate the first camera and the second camera according to the singular value decomposition result of the essential matrix 5 DOF relative poses between the cameras; the device further includes:

The scale calculation unit is configured to use the ratio between the first distance and the second distance as a scale factor; the first distance and the second distance are measured by the non-visual sensor and the visual sensor on the automatic driving device, respectively As a result obtained from the same distance, the vision sensor includes the first camera and/or the second camera;

The pose calculation unit is further configured to combine the 5-degree-of-freedom relative pose and the scale factor to obtain the first relative pose.
The device according to claim 9, wherein:

The pose calculation unit is specifically configured to perform singular value decomposition on the essential matrix to obtain the singular value decomposition result; according to the singular value decomposition result, obtain the difference between the first camera and the second camera Calculate the three-dimensional coordinate positions of the feature points in the first matching feature point set by using the at least two relative poses; respectively use the at least two relative poses to calculate the three-dimensional coordinate positions of the feature points in the first matching feature point set; The relative pose where the three-dimensional coordinate positions of each feature point in the matching feature point set are located in front of the first camera and the second camera is used as the 5-degree-of-freedom relative pose.
The device according to claim 8, wherein:

The pose calculation unit is specifically configured to iteratively solve a target equation to obtain the first relative pose; in the target equation, the parameter included in the first relative pose is an unknown number, and the first matching feature The characteristic point pairs in the point set, the internal parameters of the first camera, and the internal parameters of the second camera are known numbers.
The device according to any one of claims 8 to 11, wherein the automatic driving device is equipped with M cameras, and the M cameras include the first camera, the second camera, and the third camera The field of view of the third camera overlaps the field of view of the first camera and the field of view of the second camera, and the M is an integer greater than 2; the device further includes:

A closed-loop optimization unit for obtaining M first rotation matrices, M second rotation matrices, M first translation matrices, and M second translation matrices; the M first rotation matrices are the first camera and The rotation matrices between the second cameras and at least two of the M first rotation matrices are different, and the M second rotation matrices are the difference between the second camera and the third camera Rotation matrix and at least two rotation matrices in the M second rotation matrices are different, the M first translation matrices are the translation matrices between the first camera and the second camera, and the M th At least two translation matrices in a translation matrix are different, the M second translation matrices are the translation matrices between the second camera and the third camera, and at least two of the M second translation matrices are translated The matrices are different, the M first rotation matrices correspond to the M first translation matrices one-to-one, the M second rotation matrices correspond to the M second translation matrices one-to-one, and the M is greater than An integer of 1;

Solve the first equation set to obtain the third rotation matrix; the first equation set includes M first equations, and the M first equations correspond to the M first rotation matrices one-to-one, and correspond to the M first rotation matrices. The M second rotation matrices have a one-to-one correspondence; in each first equation, the first rotation matrix and the second rotation matrix are known numbers, and the third rotation matrix is an unknown number; the third rotation matrix is the A rotation matrix between the first camera and the third camera;

Solve the second equation set to obtain the third translation matrix; the second equation set includes M second equations, and the M second equations correspond to the M first rotation matrices one-to-one, and correspond to the M first rotation matrices. The M second rotation matrices have a one-to-one correspondence; in each second equation, the first rotation matrix, the second rotation matrix, the first translation matrix, and the second translation matrix are known numbers, and the third translation matrix is an unknown number ; The third translation matrix is the translation matrix between the first camera and the third camera;

The pose including the third rotation matrix and the third translation matrix is taken as the relative pose between the first camera and the third camera.
The device according to claim 8, wherein the first matching feature point set includes feature points extracted by the first camera from at least two frames of images, and the second camera from at least two frames of images Feature points extracted;

The pose calculation unit is specifically configured to determine the relative pose between the first camera and the second camera according to the feature point pair in the first matching feature point set to obtain the first intermediate pose;

Substituting each feature point pair in the first matching feature point set into a first formula to obtain the residual error corresponding to each feature point pair in the first matching feature point set, and the first formula representation includes the first intermediate Posture

The interference feature point pair in the first matching feature point set is removed to obtain a second matching feature point set; the interference feature point pair is the feature point pair whose corresponding residual in the first matching feature point set is greater than the residual threshold ；

Determining the relative pose between the first camera and the second camera according to the feature point pairs in the second matching feature point set to obtain a second intermediate pose;

The target intermediate pose is taken as the first relative pose between the first camera and the second camera; the target intermediate pose is the first determined according to the feature point pair in the target matching feature point set The relative pose between a camera and the second camera, the number of feature point pairs in the target matching feature point set is less than a number threshold or the number of feature point pairs in the target matching feature point set is equal to the The ratio of the number of feature point pairs in the first matching feature point set is less than the ratio threshold.
The device according to any one of claims 8 to 13, wherein the device further comprises:

The reminder unit is configured to output reminder information when the difference between the first relative pose and the second relative pose is not greater than the pose change threshold; the reminder information is used to remind all The relative pose between the first camera and the second camera is abnormal.
A car characterized by comprising:

Memory, used to store programs;

The processor is configured to execute the program stored in the memory, and when the program is executed, the processor is configured to execute the method according to any one of claims 1 to 7.
A computer-readable storage medium, wherein the computer storage medium stores a computer program, the computer program includes program instructions, and when executed by a processor, the program instructions cause the processor to execute as claimed in claim 1. To the method of any one of 7.