CN112639883B

CN112639883B - Relative attitude calibration method and related device

Info

Publication number: CN112639883B
Application number: CN202080004815.0A
Authority: CN
Inventors: 刘会; 陈亦伦; 陈同庆
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-03-17
Filing date: 2020-03-17
Publication date: 2021-11-19
Anticipated expiration: 2040-03-17
Also published as: CN112639883A; WO2021184218A1

Abstract

A relative pose calibration method and a related device applied between cameras on an automatic driving automobile are provided, wherein the relative pose calibration method comprises the following steps: capturing images by a first camera and a second camera; performing feature point matching on the image acquired by the first camera and the image acquired by the second camera to obtain a first matched feature point set; determining a first relative pose between the first camera and the second camera according to the first matching feature point set; under the condition that the difference value between the first relative pose and the second relative pose is not larger than a pose change threshold value, updating the second relative pose to the first relative pose; the second relative pose is the relative pose between the first camera and the second camera currently stored by the autopilot device. The relative pose calibration method provided by the application is high in calibration precision and high in reliability.

Description

Relative attitude calibration method and related device

Technical Field

The application relates to the field of automatic driving, in particular to a relative pose calibration method and a relative device applied to cameras on an automatic driving automobile.

Background

The automatic driving automobile needs to have higher sensing capability for 360-degree environments of an automobile body, and needs to have stable and reliable sensing results for obstacles with different scenes, different illumination conditions and different distances. A single camera cannot achieve full coverage in a 360-degree range due to the influence of factors such as camera parameters and installation positions. As a result, autonomous vehicles are generally equipped with multiple cameras to improve visual perception. Different cameras generally have different camera parameters (such as focal length, resolution, dynamic range, etc.) and are installed at different positions of the vehicle body to obtain more comprehensive perception results. For example, autonomous vehicles are equipped with at least one camera, all forward, sideways, and backward, to cover a 360 ° environment around the body. Meanwhile, the different cameras have different fields of view, and the different cameras have partially overlapped fields of view.

The relative pose between cameras on the automatic driving automobile changes along with the increase of the using time, calibration parameters (namely external parameters of the cameras) of each camera need to be corrected at the moment, and the process of correcting the external parameters is called recalibration. Along with more and more vehicles with the automatic driving function, the requirement for recalibrating the calibration parameters of the cameras is greatly increased, and the automatic driving vehicle which cannot recalibrate the external parameters of the cameras in time has great potential safety hazards. The scheme of online recalibrating external parameters of the camera adopted at present is as follows: the automatic driving vehicle acquires prior position information representing the relative positions of known urban construction markers distributed nearby a road at present, and compares the prior position information with the positions of the urban construction markers sensed by the sensor in real time, so that online calibration is performed. However, this solution has several disadvantages: prior position information of the surrounding environment needs to be obtained, and the condition that the prior is not met is easy to fail; a large amount of data needs to be acquired to obtain the prior position information, and the workload is large. Therefore, new multi-camera calibration schemes need to be investigated.

Disclosure of Invention

The embodiment of the application provides a relative pose calibration method, which can accurately determine the relative pose between cameras and is high in reliability.

In a first aspect, an embodiment of the present application provides a relative attitude calibration method, including: capturing images by a first camera and a second camera; the field of view of the first camera and the field of view of the second camera overlap, the first camera and the second camera being mounted at different locations on an autopilot device; performing feature point matching on the image acquired by the first camera and the image acquired by the second camera to obtain a first matched feature point set; the first matching feature point set comprises H groups of feature point pairs, each group of feature point pairs comprises two matched feature points, one feature point is a feature point extracted from the image acquired by the first camera, the other feature point is a feature point extracted from the image acquired by the second camera, and H is an integer not less than 8; determining a first relative pose between the first camera and the second camera according to the first set of matched feature points; updating the second relative pose to the first relative pose if the difference between the first relative pose and the second relative pose is not greater than a pose change threshold; the second relative pose is a relative pose between the first camera and the second camera currently stored by the autopilot device.

Because the field of view of the first camera and the field of view of the second camera overlap, at least some of the feature points in the image captured by the first camera match at least some of the feature points in the image captured by the second camera. The first matching feature point set is obtained by matching feature points extracted from the image collected by the first camera with feature points extracted from the image collected by the first camera. The automatic driving device can be an automatic driving automobile (also called an unmanned automobile), an unmanned aerial vehicle and other devices needing to calibrate the relative pose between cameras. In the embodiment of the application, the first relative pose between the first camera and the second camera is determined according to the first matching feature point set, the first relative pose is independent of a specific field or a specific marker, and no prior assumption is made on the surrounding environment. Compared with an online calibration scheme commonly used in the industry, the relative pose calibration method provided by the embodiment of the application does not need prior hypothesis, and has stronger adaptability to the environment. In addition, the relative pose calibration method provided by the embodiment of the application can achieve the repetition precision of less than 0.1 degree, effectively reduces the times of re-calibrating the relative pose, and is high in reliability.

In an optional implementation, the determining, from the first set of matched feature points, a first relative pose between the first camera and the second camera includes: determining an essential matrix between the first camera and the second camera; calculating a 5-degree-of-freedom (DOF) relative pose between the first camera and the second camera according to a singular value decomposition result of the intrinsic matrix; taking the ratio of the first distance to the second distance as a scale factor; the first distance and the second distance are respectively the result of measuring the same distance by a non-vision sensor and a vision sensor on the automatic driving device, and the vision sensor comprises the first camera and/or the second camera; merging the 5-degree-of-freedom relative pose and the scale factor to obtain the first relative pose.

In the implementation mode, 5-degree-of-freedom relative pose between two cameras is obtained by decomposing the essential matrix, and then 6-degree-of-freedom relative pose is obtained by the 5-degree-of-freedom relative pose and the scale factor, so that no prior assumption is made on the surrounding environment, no 3D information of surrounding scenes needs to be acquired, and the workload is low.

In an optional implementation manner, the calculating the relative pose with 5 degrees of freedom between the first camera and the second camera according to the singular value decomposition result of the essential matrix includes: performing singular value decomposition on the intrinsic matrix to obtain a singular value decomposition result; obtaining at least two relative poses between the first camera and the second camera according to the singular value decomposition result; calculating the three-dimensional coordinate positions of the feature points in the first matching feature point set by using the at least two relative poses respectively; taking, as the 5-degree-of-freedom relative pose, one of the at least two relative poses at which the three-dimensional coordinate position of each feature point in the first set of matched feature points is located in front of the first camera and the second camera.

In this implementation, the 5-degree-of-freedom relative pose between the first camera and the second camera can be accurately and quickly determined.

In an optional implementation, the determining, from the first set of matched feature points, a first relative pose between the first camera and the second camera includes: iteratively solving an objective equation to obtain the first relative pose; in the object equation, the first relative pose includes parameters that are unknowns, and the pairs of feature points in the first matching feature point set, the internal parameters of the first camera, and the internal parameters of the second camera are known numbers.

In the implementation mode, the first relative pose is obtained by iteratively solving the objective equation, so that the first relative pose can be rapidly and accurately calculated without calculating an essential matrix.

In an optional implementation manner, the automatic driving device is provided with M cameras, the M cameras include the first camera, the second camera and a third camera, a visual field of the third camera overlaps with a visual field of the first camera and a visual field of the second camera, and M is an integer greater than 2; the method further comprises the following steps: obtaining M first rotation matrixes, M second rotation matrixes, M first translation matrixes and M second translation matrixes; the M first rotation matrices are rotation matrices between the first camera and the second camera and at least two rotation matrices among the M first rotation matrices are different, the M second rotation matrices are rotation matrices between the second camera and the third camera and at least two rotation matrices among the M second rotation matrices are different, the M first translation matrices are translation matrices between the first camera and the second camera and at least two translation matrices among the M first translation matrices are different, the M second translation matrices are translation matrices between the second camera and the third camera and at least two translation matrices among the M second translation matrices are different, the M first rotation matrices are in one-to-one correspondence with the M first translation matrices, and the M second rotation matrices are in one-to-one correspondence with the M second translation matrices, m is an integer greater than 1; solving the first equation group to obtain a third rotation matrix; the first equation set comprises M first equations, and the M first equations are in one-to-one correspondence with the M first rotation matrixes and in one-to-one correspondence with the M second rotation matrixes; in each first equation, the first rotation matrix and the second rotation matrix are known numbers, and the third rotation matrix is an unknown number; the third rotation matrix is a rotation matrix between the first camera and the third camera; solving the second equation set to obtain a third translation matrix; the second equation set comprises M second equations, and the M second equations are in one-to-one correspondence with the M first rotation matrixes and in one-to-one correspondence with the M second rotation matrixes; in each second equation, the first rotation matrix, the second rotation matrix, the first translation matrix and the second translation matrix are known numbers, and the third translation matrix is an unknown number; the third translation matrix is a translation matrix between the first camera and the third camera; setting a pose including the third rotation matrix and the third translation matrix as a relative pose between the first camera and the third camera.

In the implementation mode, a closed loop equation is formed by relative pose matrixes between every two multiple cameras, and the closed loop equation is used for realizing pose calibration between the cameras without public vision and carrying out overall optimization, so that more accurate relative pose of the cameras is obtained.

In an optional implementation manner, the first matching feature point set includes feature points extracted from at least two frames of images by the first camera and feature points extracted from at least two frames of images by the second camera; the determining, from the first set of matched feature points, a first relative pose between the first camera and the second camera comprises: determining a relative pose between the first camera and the second camera according to the feature point pairs in the first matching feature point set to obtain a first intermediate pose; substituting each characteristic point pair in the first matching characteristic point set into a first formula to obtain a residual error corresponding to each characteristic point pair in the first matching characteristic point set, wherein the first formula representation comprises the first intermediate pose; rejecting interference characteristic point pairs in the first matching characteristic point set to obtain a second matching characteristic point set; the interference characteristic point pair is a characteristic point pair of which the residual error corresponding to the first matching characteristic point set is greater than a residual error threshold value; determining a relative pose between the first camera and the second camera according to the feature point pairs in the second matching feature point set to obtain a second intermediate pose; taking an object intermediate pose as the first relative pose between the first camera and the second camera; the target intermediate pose is a relative pose between the first camera and the second camera determined according to the feature point pairs in the target matching feature point set, and the number of the feature point pairs in the target matching feature point set is smaller than a quantity threshold or the ratio of the number of the feature point pairs in the target matching feature point set to the number of the feature point pairs in the first matching feature point set is smaller than a proportion threshold.

Optionally, the feature points included in the first matching feature point set are matched feature points in a first image acquired by the first camera and a second image acquired by the second camera. When the number of the matched feature point pairs in the first image and the second image is not less than 8, an estimation of the relative pose of the two cameras can be obtained. When the number of matched feature points in one frame of image collected by the first camera and one frame of image collected by the second camera is small, the estimation is unstable, and the error influence can be reduced through multi-frame accumulation. In some embodiments, the autopilot device may extract feature points in the two maps of the ith frame, denoted x 'respectively'_iAnd x_iAdding it to the feature point set X₁And X₂(ii) a After F frames, the feature point set X₁And X₂The matched feature points (namely the first matched feature point set) in the two images in the F frame are stored, and the feature point set X is used₁And X₂Estimating the relative pose of the two cameras; due to the feature point set X₁And X₂The number of feature points in (2) is far more than that of feature points of a single frame, so an iterative optimization mode can be adopted: first, a feature point set X is used₁And X₂Performing pose estimation on all the points to obtain a first intermediate pose, substituting the first intermediate pose into a formula (14) to calculate a residual error, eliminating the feature points of which the residual errors are greater than a residual error threshold value, and performing pose estimation again by using the remaining feature points to obtain a second intermediate pose; the above processes are repeated until the number of the feature points to be eliminated is less than the number threshold (e.g. 5% of the total number). The optimal solution for a set of F frames of data is now available. It should be understood that two images in a frame refer to one image captured by the first camera and one image captured by the second camera,and the time interval between two images in this frame is less than a time threshold, e.g., 0.5ms, 1ms, etc. According to the central limit theorem, the sum of R calculated between different frames in long-time observation is assumed

Are independently and identically distributed, then R and F are increased

Will tend to level off. Selecting appropriate F such that R and

the variance of (a) can meet the requirement.

In the implementation mode, a multi-frame accumulation mode is utilized, so that the problem of jitter caused by single-frame calculation can be avoided, and the calibration result tends to be stable.

In an optional implementation, the method further includes: outputting reminding information under the condition that the difference value between the first relative pose and the second relative pose is not greater than the pose change threshold; the reminding information is used for reminding that the relative pose between the first camera and the second camera is abnormal.

In the implementation mode, the passenger can be reminded in time that the calibration parameters of the camera are abnormal.

In a second aspect, an embodiment of the present application provides another relative pose calibration method, where the method includes: the method comprises the steps that a server receives recalibration reference information from an automatic driving device, the recalibration reference information is used for determining the relative pose between a first camera and a second camera of the automatic driving device through the server, the visual field of the first camera and the visual field of the second camera are overlapped, and the first camera and the second camera are installed at different positions of the automatic driving device; the server obtains a first matching feature point set according to the recalibration reference information, wherein the first matching feature point set comprises H groups of feature point pairs, each group of feature point pairs comprises two matched feature points, one feature point is a feature point extracted from an image acquired by the first camera, the other feature point is a feature point extracted from an image acquired by the second camera, and H is an integer not less than 8; the server determines a first relative pose between the first camera and the second camera according to the first matching feature point set; the server sends the first relative pose to the autopilot device.

Optionally, the recalibration reference information includes an image acquired by the first camera and the second camera synchronously or the first matching feature point set. The server may store or receive from the autopilot an internal reference of the first camera and an internal reference of the second camera and determine a first relative pose between the first camera and the second camera from a first set of matching feature points.

In the embodiment of the application, the automatic driving device does not need to calibrate the relative pose between the cameras, only data needed for calibrating the relative pose between the cameras are sent to the server, and the server recalibrates the relative pose between the cameras on the automatic driving device, so that the efficiency is high.

In an optional implementation manner, before determining the first relative pose between the first camera and the second camera according to the first matching feature point set, the method further includes: the server receives a pose calibration request from the automatic driving device, wherein the pose calibration request is used for requesting to recalibrate a relative pose between the first camera and the second camera of the automatic driving device, and the pose calibration request carries internal parameters of the first camera and internal parameters of the second camera; the determining, from the first set of matched feature points, a first relative pose between the first camera and the second camera comprises: determining the first relative pose between the first camera and the second camera from a first set of matched feature points, the internal reference of the first camera, and the internal reference of the second camera.

In a third aspect, an embodiment of the present application provides another method for calibrating a relative pose, where the method includes: the method comprises the steps that an automatic driving device sends recalibration reference information to a server, the recalibration reference information is used for the server to determine the relative pose between a first camera and a second camera of the automatic driving device, the visual field of the first camera and the visual field of the second camera are overlapped, and the first camera and the second camera are installed at different positions of the automatic driving device; the autopilot device receiving the first relative pose from the server; the autopilot device updates a second relative pose to the first relative pose if a difference between the first relative pose and the second relative pose is not greater than a pose change threshold; the second relative pose is a relative pose between the first camera and the second camera currently stored by the autopilot device. Optionally, the recalibration reference information includes a first matching feature point set, the first matching feature point set includes H groups of feature point pairs, each group of feature point pairs includes two matched feature points, one feature point is a feature point extracted from the image acquired by the first camera, the other feature point is a feature point extracted from the image acquired by the second camera, and H is an integer not less than 8. Optionally, the recalibration reference information includes images acquired by the first camera and the second camera synchronously.

In the embodiment of the application, the automatic driving device does not need to calibrate the relative pose between the cameras, only needs to send data required for calibrating the relative pose between the cameras to the server, and is low in workload.

In an optional implementation manner, before the automatic driving device sends the recalibration reference information to the server, the method further includes: the automatic driving device sends a pose calibration request to the server, the pose calibration request is used for requesting to recalibrate the relative pose between the first camera and the second camera of the automatic driving device, and the pose calibration request carries the internal parameters of the first camera and the internal parameters of the second camera.

In a fourth aspect, an embodiment of the present application provides an automatic driving device, including: the image acquisition unit is used for acquiring images through the first camera and the second camera respectively; the field of view of the first camera and the field of view of the second camera overlap, the first camera and the second camera being mounted at different locations on an autopilot device; the image feature point extraction unit is used for carrying out feature point matching on the image acquired by the first camera and the image acquired by the second camera to obtain a first matched feature point set; the first matching feature point set comprises H groups of feature point pairs, each group of feature point pairs comprises two matched feature points, one feature point is a feature point extracted from the image acquired by the first camera, the other feature point is a feature point extracted from the image acquired by the second camera, and H is an integer not less than 8; a pose calculation unit for determining a first relative pose between the first camera and the second camera according to the first set of matched feature points; the first matching feature point set comprises H groups of feature point pairs, each group of feature point pairs comprises two matching feature points, one feature point is a feature point extracted from an image acquired by the first camera, the other feature point is a feature point extracted from an image acquired by the second camera, the visual fields of the first camera and the second camera are overlapped, the first camera and the second camera are installed at different positions of the automatic driving device, and H is an integer not less than 8;

a calibration parameter updating unit, configured to update the second relative pose to the first relative pose when a difference between the first relative pose and the second relative pose is not greater than a pose change threshold; the second relative pose is a relative pose between the first camera and the second camera currently stored by the autopilot device.

In an optional implementation manner, the pose calculation unit is specifically configured to determine an essential matrix between the first camera and the second camera; calculating a 5-degree-of-freedom (DOF) relative pose between the first camera and the second camera according to a singular value decomposition result of the intrinsic matrix; the device further comprises: a scale calculation unit for taking a ratio between the first distance and the second distance as a scale factor; the first distance and the second distance are respectively the result of measuring the same distance by a non-vision sensor and a vision sensor on the automatic driving device, and the vision sensor comprises the first camera and/or the second camera; the pose calculation unit is further configured to combine the 5-degree-of-freedom relative pose and the scale factor to obtain the first relative pose.

In an optional implementation manner, the pose calculation unit is specifically configured to perform singular value decomposition on the intrinsic matrix to obtain a singular value decomposition result; obtaining at least two relative poses between the first camera and the second camera according to the singular value decomposition result; calculating the three-dimensional coordinate positions of the feature points in the first matching feature point set by using the at least two relative poses respectively; taking, as the 5-degree-of-freedom relative pose, one of the at least two relative poses at which the three-dimensional coordinate position of each feature point in the first set of matched feature points is located in front of the first camera and the second camera.

In an optional implementation manner, the pose calculation unit is specifically configured to iteratively solve an objective equation to obtain the first relative pose; in the object equation, the first relative pose includes parameters that are unknowns, and the pairs of feature points in the first matching feature point set, the internal parameters of the first camera, and the internal parameters of the second camera are known numbers.

In an optional implementation manner, the automatic driving device is provided with M cameras, the M cameras include the first camera, the second camera and a third camera, a visual field of the third camera overlaps with a visual field of the first camera and a visual field of the second camera, and M is an integer greater than 2; the device further comprises: the closed-loop optimization unit is used for obtaining M first rotation matrixes, M second rotation matrixes, M first translation matrixes and M second translation matrixes; the M first rotation matrices are rotation matrices between the first camera and the second camera and at least two rotation matrices among the M first rotation matrices are different, the M second rotation matrices are rotation matrices between the second camera and the third camera and at least two rotation matrices among the M second rotation matrices are different, the M first translation matrices are translation matrices between the first camera and the second camera and at least two translation matrices among the M first translation matrices are different, the M second translation matrices are translation matrices between the second camera and the third camera and at least two translation matrices among the M second translation matrices are different, the M first rotation matrices are in one-to-one correspondence with the M first translation matrices, and the M second rotation matrices are in one-to-one correspondence with the M second translation matrices, m is an integer greater than 1; solving the first equation group to obtain a third rotation matrix; the first equation set comprises M first equations, and the M first equations are in one-to-one correspondence with the M first rotation matrixes and in one-to-one correspondence with the M second rotation matrixes; in each first equation, the first rotation matrix and the second rotation matrix are known numbers, and the third rotation matrix is an unknown number; the third rotation matrix is a rotation matrix between the first camera and the third camera; solving the second equation set to obtain a third translation matrix; the second equation set comprises M second equations, and the M second equations are in one-to-one correspondence with the M first rotation matrixes and in one-to-one correspondence with the M second rotation matrixes; in each second equation, the first rotation matrix, the second rotation matrix, the first translation matrix and the second translation matrix are known numbers, and the third translation matrix is an unknown number; the third translation matrix is a translation matrix between the first camera and the third camera; setting a pose including the third rotation matrix and the third translation matrix as a relative pose between the first camera and the third camera.

In an optional implementation manner, the first matching feature point set includes feature points extracted from at least two frames of images by the first camera and feature points extracted from at least two frames of images by the second camera; the pose calculation unit is specifically configured to determine, according to the feature point pairs in the first matching feature point set, a relative pose between the first camera and the second camera to obtain a first intermediate pose; substituting each characteristic point pair in the first matching characteristic point set into a first formula to obtain a residual error corresponding to each characteristic point pair in the first matching characteristic point set, wherein the first formula representation comprises the first intermediate pose; rejecting interference characteristic point pairs in the first matching characteristic point set to obtain a second matching characteristic point set; the interference characteristic point pair is a characteristic point pair of which the residual error corresponding to the first matching characteristic point set is greater than a residual error threshold value; determining a relative pose between the first camera and the second camera according to the feature point pairs in the second matching feature point set to obtain a second intermediate pose; taking an object intermediate pose as the first relative pose between the first camera and the second camera; the target intermediate pose is a relative pose between the first camera and the second camera determined according to the feature point pairs in the target matching feature point set, and the number of the feature point pairs in the target matching feature point set is smaller than a quantity threshold or the ratio of the number of the feature point pairs in the target matching feature point set to the number of the feature point pairs in the first matching feature point set is smaller than a proportion threshold.

In an optional implementation, the apparatus further comprises: the reminding unit is used for outputting reminding information under the condition that the difference value between the first relative pose and the second relative pose is not larger than the pose change threshold; the reminding information is used for reminding that the relative pose between the first camera and the second camera is abnormal.

In a fifth aspect, an embodiment of the present application provides a server, including: an acquisition unit, configured to obtain a first matching feature point set, where the first matching feature point set includes H groups of feature point pairs, each group of feature point pairs includes two matched feature points, one of the feature points is a feature point extracted from an image captured by a first camera, the other feature point is a feature point extracted from an image captured by a second camera, a field of view of the first camera and a field of view of the second camera overlap, the first camera and the second camera are installed at different positions of an automatic driving apparatus, and H is an integer not less than 8; a pose calculation unit for determining a first relative pose between the first camera and the second camera according to a first set of matching feature points; and the sending unit is used for sending the first relative pose to the automatic driving device.

In an optional implementation manner, the server further includes: a receiving unit, configured to receive a pose calibration request from the autopilot, where the pose calibration request is used to request to recalibrate a relative pose between the first camera and the second camera of the autopilot, and the pose calibration request carries internal parameters of the first camera and internal parameters of the second camera; the pose calculation unit is specifically configured to determine the first relative pose between the first camera and the second camera according to a first matching feature point set, the internal reference of the first camera, and the internal reference of the second camera.

In a sixth aspect, an embodiment of the present application provides an automatic driving device, including: the device comprises a sending unit, a judging unit and a display unit, wherein the sending unit is used for sending recalibration reference information to a server, the recalibration reference information is used for the server to determine the relative pose between a first camera and a second camera of the automatic driving device, the visual field of the first camera and the visual field of the second camera are overlapped, and the first camera and the second camera are installed at different positions of the automatic driving device; a receiving unit configured to receive the first relative pose from the server; a calibration parameter updating unit, configured to update the second relative pose to the first relative pose when a difference between the first relative pose and the second relative pose is not greater than a pose change threshold; the second relative pose is a relative pose between the first camera and the second camera currently stored by the autopilot device.

In an optional implementation manner, the sending unit is further configured to send a pose calibration request to the server, where the pose calibration request is used to request to recalibrate a relative pose between the first camera and the second camera of the autopilot device, and the pose calibration request carries internal parameters of the first camera and internal parameters of the second camera.

In a seventh aspect, an embodiment of the present application provides an automobile, including a memory and a processor, where the memory is used for storing codes; a processor for executing the program stored in the memory, the processor being configured to perform the method of the first aspect or the third aspect and optional implementations described above when the program is executed.

In an eighth aspect, an embodiment of the present application provides a server, including a memory and a processor, where the memory is used for storing codes; a processor for executing the program stored in the memory, the processor being configured to perform the method of the second aspect and the alternative implementations described above when the program is executed.

In a ninth aspect, the present application provides a computer-readable storage medium, which stores a computer program, where the computer program includes program instructions, and the program instructions, when executed by a processor, cause the processor to execute the method of the first to third aspects and optional implementation manners.

In a tenth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a data interface, and the processor reads instructions stored on a memory through the data interface to perform a method according to the first aspect to the third aspect and any optional implementation manner described above.

In an eleventh aspect, the present application provides a computer program product, which includes program instructions, and when executed by a processor, causes the processor to execute the method of the first aspect to the third aspect and any optional implementation manner.

Drawings

FIG. 1 is a functional block diagram of an autopilot device provided by an embodiment of the present application;

fig. 2 is a schematic structural diagram of an automatic driving system according to an embodiment of the present application;

fig. 3 is a flowchart of a relative attitude calibration method according to an embodiment of the present application;

fig. 4 is a schematic view of a common view between cameras according to an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of an automatic steering apparatus according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of another automatic steering apparatus provided in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of another autopilot device provided by an embodiment of the present application;

fig. 8 is a flowchart of another relative pose calibration method provided in the embodiment of the present application;

fig. 9 is a flowchart of another relative pose calibration method provided in the embodiment of the present application;

fig. 10 is a schematic structural diagram of a server according to an embodiment of the present application.

Detailed Description

The terms "first," "second," and "third," etc. in the description and claims of the present application and the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. Furthermore, the terms "comprises" and "comprising," as well as any variations thereof, are intended to cover a non-exclusive inclusion, such as a list of steps or elements. A method, system, article, or apparatus is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, system, article, or apparatus. "and/or" is used to indicate the selection of one or both between two objects to which it is connected. For example "A and/or B" means A, B or A + B.

As described in the background art, the relative pose between cameras on an autonomous vehicle changes with the increase of the usage time, and at this time, the calibration parameters (i.e., external parameters of the cameras) of each camera need to be corrected. Along with more and more vehicles with the automatic driving function, the requirement for recalibrating the calibration parameters of the cameras is greatly increased, and the automatic driving vehicle which cannot recalibrate the external parameters of the cameras in time has great potential safety hazards. The relative pose calibration method provided by the embodiment of the application can be applied to an automatic driving scene. The driving scenario will be briefly described below.

Driving scene 1: the automatic driving device determines the relative pose between the two cameras according to the matched feature points in the images synchronously acquired by the two cameras with the common visual field; and updating the currently calibrated relative pose in the case that the difference value between the re-determined relative pose between the two cameras and the relative pose between the two cameras currently calibrated by the automatic driving device is not larger than a pose change threshold value.

Driving scene 2: the method comprises the steps that an automatic driving device sends a pose calibration request and information needed for determining the relative pose between at least two cameras on the automatic driving device to a server, wherein the pose calibration request is used for requesting to recalibrate the relative pose between the at least two cameras on the automatic driving device; the automatic driving device updates the currently stored relative pose between the at least two cameras according to the relative pose from the server

Fig. 1 is a functional block diagram of an automatic driving device according to an embodiment of the present application. In one embodiment, the autopilot device 100 is configured in a fully or partially autopilot mode. For example, the autonomous device 100 may control itself while in the autonomous mode, and may determine a current state of the autonomous device 100 and its surroundings by human operation, determine a possible behavior of at least one other vehicle in the surroundings, and determine a confidence level corresponding to a likelihood that the other vehicle performs the possible behavior, controlling the autonomous device 100 based on the determined information. The autonomous device 100 may be placed into operation without human interaction while the autonomous device 100 is in the autonomous mode.

The autopilot device 100 may include various subsystems such as a travel system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, as well as a power supply 110, a computer system 112, and a user interface 116. Alternatively, the autopilot device 100 may include more or fewer subsystems and each subsystem may include multiple elements. In addition, each of the subsystems and components of the autopilot device 100 may be interconnected by wires or wirelessly.

The travel system 102 may include components that provide powered motion to the autopilot device 100. In one embodiment, the propulsion system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels/tires 121. The engine 118 may be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a hybrid engine of a gasoline engine and an electric motor, or a hybrid engine of an internal combustion engine and an air compression engine. The engine 118 converts the energy source 119 into mechanical energy.

Examples of energy sources 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electrical power. The energy source 119 may also provide energy to other systems of the autopilot device 100.

The transmission 120 may transmit mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other devices, such as a clutch. Wherein the drive shaft may comprise one or more shafts that may be coupled to one or more wheels 121.

The sensor system 104 may include a number of sensors that sense information about the environment surrounding the autopilot device 100. For example, the sensor system 104 may include a positioning system 122 (which may be a Global Positioning System (GPS) system, a Beidou system, or other positioning system), an Inertial Measurement Unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. The sensor system 104 may also include sensors that are monitored for internal systems of the autopilot device 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors may be used to detect the object and its corresponding characteristics (position, shape, orientation, velocity, etc.). Such detection and identification is a key function of the safe operation of the autonomous driving apparatus 100.

The positioning system 122 may be used to estimate the geographic location of the autonomous device 100. The IMU 124 is used to sense position and orientation changes of the autopilot device 100 based on inertial acceleration. In one embodiment, IMU 124 may be a combination of an accelerometer and a gyroscope.

The radar 126 may utilize radio signals to sense objects within the surrounding environment of the autopilot device 100.

The laser rangefinder 128 may utilize laser light to sense objects in the environment in which the autopilot device 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, laser scanners, and one or more detectors, among other system components.

The camera 130 may be used to capture multiple images of the surrounding environment of the autonomous device 100. The camera 130 may be a still camera or a video camera. The camera 130 may capture multiple images of the environment surrounding the autopilot device 100 in real time or periodically. The cameras 130 include cameras where at least two fields of view overlap, i.e., at least two cameras have a common field of view.

Control system 106 is provided to control the operation of autopilot device 100 and its components. Control system 106 may include various elements including a steering system 132, a throttle 134, a braking unit 136, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

The steering system 132 is operable to adjust the heading of the autopilot device 100. For example, in one embodiment, a steering wheel system.

The throttle 134 is used to control the speed of operation of the engine 118 and thus the speed of the autopilot device 100.

The brake unit 136 is used to control the deceleration of the autopilot device 100. The brake unit 136 may use friction to slow the wheel 121. In other embodiments, the brake unit 136 may convert the kinetic energy of the wheel 121 into an electric current. The brake unit 136 may take other forms to slow the rotational speed of the wheels 121 to control the speed of the autopilot device 100.

The computer vision system 140 may be operable to process and analyze images captured by the camera 130 to identify objects and/or features in the environment surrounding the autonomous device 100. Such objects and/or features may include traffic signals, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, automated driving methods, Motion from Motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 may be used to map an environment, track objects, estimate the speed of objects, and so forth. The computer vision system 140 may locate the position of the obstacle using the point cloud acquired by the lidar and the image of the surrounding environment acquired by the camera.

The route control system 142 is used to determine a travel route for the autonomous device 100. In some embodiments, the route control system 142 may combine data from the sensors 138, the GPS 122, and one or more predetermined maps to determine a travel route for the autopilot device 100.

The obstacle avoidance system 144 is used to identify, assess, and avoid or otherwise negotiate potential obstacles in the environment of the autonomous device 100.

Of course, in one example, the control system 106 may additionally or alternatively include components other than those shown and described. Or may reduce some of the components shown above.

The autopilot device 100 interacts with external sensors, other vehicles, other computer systems, or users through peripherals 108. The peripheral devices 108 may include a wireless communication system 146, an in-vehicle computer 148, a microphone 150, and/or speakers 152.

In some embodiments, the peripheral device 108 provides a means for a user of the autopilot device 100 to interact with the user interface 116. For example, the onboard computer 148 may provide information to a user of the autopilot device 100. The user interface 116 may also operate the in-vehicle computer 148 to receive user input. The in-vehicle computer 148 may be operated via a touch screen. In other cases, the peripheral devices 108 may provide a means for the autopilot device 100 to communicate with other devices located within the vehicle. For example, the microphone 150 may receive audio (e.g., voice commands or other audio input) from a user of the autopilot device 100. Similarly, the speaker 152 may output audio to a user of the autopilot device 100.

The wireless communication system 146 may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 146 may communicate with a Wireless Local Area Network (WLAN) using WiFi. In some embodiments, the wireless communication system 146 may utilize an infrared link, bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system 146 may include one or more Dedicated Short Range Communications (DSRC) devices that may include public and/or private data communications between vehicles and/or roadside stations.

The power supply 110 may provide power to various components of the autopilot device 100. In one embodiment, power source 110 may be a rechargeable lithium ion or lead acid battery. One or more battery packs of such batteries may be configured as a power source to provide power to the various components of the autopilot device 100. In some embodiments, the power source 110 and the energy source 119 may be implemented together, such as in some all-electric vehicles.

Some or all of the functions of the autopilot device 100 are controlled by the computer system 112. The computer system 112 may include at least one processor 113, the processor 113 executing instructions 115 stored in a non-transitory computer readable medium, such as a data storage device 114. The computer system 112 may also be a plurality of computing devices that control individual components or subsystems of the autopilot device 100 in a distributed manner.

The processor 113 may be any conventional processor, such as a commercially available Central Processing Unit (CPU). Alternatively, the processor may be a dedicated device such as an ASIC or other hardware-based processor. Although fig. 1 functionally illustrates a processor, memory, and other elements of the computer system 112 in the same block, those skilled in the art will appreciate that the processor, computer, or memory may actually comprise multiple processors, computers, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard drive or other storage medium located in a different enclosure than the computer system 112. Thus, references to a processor or computer are to be understood as including references to a collection of processors or computers or memories which may or may not operate in parallel. Rather than using a single processor to perform the steps described herein, some of the components, such as the steering and retarding components, may each have their own processor that performs only computations related to the component-specific functions.

In various aspects described herein, the processor may be located remotely from the autonomous device and in wireless communication with the autonomous device. In other aspects, some of the operations of the processes described herein are performed on a processor disposed within the autopilot device while others are performed by a remote processor, including taking the steps necessary to perform a single maneuver.

In some embodiments, the data storage device 114 may include instructions 115 (e.g., program logic), the instructions 115 being executable by the processor 113 to perform various functions of the autopilot device 100, including those described above. The data storage 114 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and/or control one or more of the propulsion system 102, the sensor system 104, the control system 106, and the peripherals 108.

In addition to instructions 115, data storage device 114 may also store data such as road maps, route information, the location, direction, speed, and other information of the vehicle. Such information may be used by the autonomous device 100 and the computer system 112 during operation of the autonomous device 100 in autonomous, semi-autonomous, and/or manual modes.

A user interface 116 for providing information to or receiving information from a user of the autopilot device 100. Optionally, the user interface 116 may include one or more input/output devices within the collection of peripheral devices 108, such as a wireless communication system 146, an in-vehicle computer 148, a microphone 150, and a speaker 152.

The computer system 112 may control the functions of the autopilot device 100 based on inputs received from various subsystems, such as the travel system 102, the sensor system 104, and the control system 106, as well as from the user interface 116. For example, the computer system 112 may utilize input from the control system 106 in order to control the steering unit 132 to avoid obstacles detected by the sensor system 104 and the obstacle avoidance system 144. In some embodiments, the computer system 112 is operable to provide control over many aspects of the autopilot device 100 and its subsystems.

Alternatively, one or more of these components may be mounted separately from or associated with the autopilot device 100. For example, the data storage device 114 may exist partially or completely separate from the autopilot device 100. The above components may be communicatively coupled together in a wired and/or wireless manner.

Optionally, the above components are only an example, in an actual application, components in the above modules may be added or deleted according to an actual need, and fig. 1 should not be construed as limiting the embodiment of the present application.

An autonomous vehicle traveling on a roadway, such as the autonomous device 100 above, may identify objects within its surrounding environment to determine an adjustment to the current speed. The object may be another vehicle, a traffic control device, or another type of object. In some examples, each identified object may be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, separation from the vehicle, etc., may be used to determine the speed at which the autonomous vehicle is to be adjusted.

Alternatively, the autonomous driving apparatus 100 or a computing device associated with the autonomous driving apparatus 100 (e.g., the computer system 112, the computer vision system 140, the data storage 114 of fig. 1) may predict behavior of the identified objects based on characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on roads, etc.). Optionally, each identified object depends on the behavior of each other, so it is also possible to predict the behavior of a single identified object taking all identified objects together into account. The autopilot device 100 is capable of adjusting its speed based on the predicted behavior of the above identified object. In other words, the autonomous vehicle is able to determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, other factors may also be considered to determine the speed of the autonomous device 100, such as the lateral position of the autonomous device 100 in the road being traveled, the curvature of the road, the proximity of static and dynamic objects, and so forth.

In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device may also provide instructions to modify the steering angle of the autonomous device 100 to cause the autonomous vehicle to follow a given trajectory and/or maintain a safe lateral and longitudinal distance from objects in the vicinity of the autonomous vehicle (e.g., cars in adjacent lanes on a road).

The automatic steering device 100 may be a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a lawn mower, an amusement car, a playground vehicle, construction equipment, an electric car, a golf cart, a train, a trolley, etc., and the embodiment of the present invention is not particularly limited.

The autopilot device 100 may capture images in real time or periodically via the camera 130. The two or more cameras included in the camera 130 may capture images simultaneously. The two cameras acquiring images synchronously means that the shortest time for the two cameras to acquire images is less than a time threshold, for example 10 ms. Optionally, the autopilot device 100 sends the images acquired by the cameras 130 and the information required for determining the relative pose between at least two cameras comprised by the cameras 130 to the server; the relative pose from the server is received and the relative pose between at least two cameras comprised by the camera 130 is updated. Alternatively, the autopilot device 100 determines the relative pose between at least two cameras comprised by the camera 130 from the images captured by the camera 130.

Fig. 1 illustrates a functional block diagram of an autopilot system 100, and an autopilot system 101 is described below. Fig. 2 is a schematic structural diagram of an automatic driving system according to an embodiment of the present application. Fig. 1 and 2 depict the autopilot device 100 from different angles. As shown in FIG. 2, computer system 101 comprises a processor 103, processor 103 coupled to a system bus 105. Processor 103 may be one or more processors, where each processor may include one or more processor cores. A display adapter (video adapter)107, which may drive a display 109, the display 109 coupled with system bus 105. System bus 105 is coupled through a bus bridge 111 and an input/output (I/O) bus 113. The I/O interface 115 is coupled to an I/O bus. The I/O interface 115 communicates with various I/O devices, such as an input device 117 (e.g., keyboard, mouse, touch screen, etc.), a multimedia disk (media tray)121, a multimedia interface, etc. A transceiver 123 (which can send and/or receive radio communication signals), a camera 155 (which can capture scenic and motion digital video images), and an external USB interface 125. And (4) optional. The interface connected to the I/O interface 115 may be a USB interface.

Processor 103 may be any conventional processor, including a reduced instruction set computing ("RISC") processor, a complex instruction set computing ("CISC") processor, or a combination thereof. Alternatively, the processor may be a dedicated device such as an application specific integrated circuit ("ASIC"). Alternatively, the processor 103 may be a Neural-Network Processing Unit (NPU) or a combination of a Neural network processor and the above conventional processors. Optionally, the processor 103 is mounted with a neural network processor.

Computer system 101 may communicate with software deploying server 149 via network interface 129. The network interface 129 is a hardware network interface, such as a network card. The network 127 may be an external network, such as the internet, or an internal network, such as an ethernet or virtual private network. Optionally, the network 127 may also be a wireless network, such as a WiFi network, a cellular network, and the like.

The hard drive interface is coupled to system bus 105. The hardware drive interface is connected with the hard disk drive. System memory 135 is coupled to system bus 105. The data running in system memory 135 may include the operating system 137 and application programs 143 of computer system 101.

The operating system includes a Shell (Shell)139 and a kernel (kernel) 141. The shell 139 is an interface between the user and the kernel of the operating system. The shell 139 is the outermost layer of the operating system. The shell 139 manages the interaction between the user and the operating system: waits for user input, interprets the user input to the operating system, and processes the output results of the various operating systems.

Kernel 141 is comprised of those portions of the operating system that are used to manage memory, files, peripherals, and system resources. Interacting directly with the hardware, the operating system kernel typically runs processes and provides inter-process communication, CPU slot management, interrupts, memory management, IO management, and the like.

The application programs 141 include automatic driving-related programs, such as programs for managing interactions between the automatic driving apparatus and obstacles on the road, programs for controlling the driving route or speed of the automatic driving apparatus, and programs for controlling interactions between the automatic driving apparatus 100 and other automatic driving apparatuses on the road. Application 141 also resides on the system of software deploying server (deploying server) 149. In one embodiment, computer system 101 may download application 141 from software deploying server 149 when execution of application 141 is required.

Sensor 153 is associated with computer system 101. The sensors 153 are used to detect the environment surrounding the computer system 101. For example, the sensor 153 may detect an animal, a car, an obstacle, a crosswalk, and the like, and further, the sensor may detect an environment around the animal, the car, the obstacle, the crosswalk, and the like, such as: the environment surrounding the animal, e.g., other animals present around the animal, weather conditions, brightness of the surrounding environment, etc. Alternatively, if computer system 101 is located on an autopilot, the sensor may be a camera (i.e., a camera), a lidar, an infrared sensor, a chemical detector, a microphone, or the like. Sensor 153, when activated, senses information at preset intervals and provides the sensed information to computer system 101 in real time or near real time. Alternatively, the sensor may comprise a lidar that may provide the acquired point cloud to computer system 101 in real-time or near real-time, i.e., provide a series of acquired point clouds to computer system 101, one timestamp for each acquired point cloud. Optionally, the camera provides the captured images to computer system 101 in real-time or near real-time, with each frame of image corresponding to a timestamp. It should be understood that the computer system 101 may obtain a sequence of images from a camera.

Alternatively, in various embodiments described herein above, computer system 101 may be located remotely from the autopilot device and may communicate wirelessly with the autopilot device. Transceiver 123 may transmit the autopilot task, sensor data collected by sensors 153, and other data to computer system 101; control instructions sent by computer system 101 may also be received. The autopilot may execute control commands received by the transceiver from computer system 101 and perform corresponding driving operations. In other aspects, some of the processes described herein are executed on a processor disposed within an autonomous vehicle, others being executed by a remote processor, including taking the actions necessary to perform a single maneuver.

The automatic driving device updates the relative pose between the deployed cameras in time in the automatic driving process. The manner in which the autopilot device updates the relative pose between its deployed cameras will be detailed later. In addition, the relative pose calibration method provided by the embodiment of the application can be applied to an automatic driving device and a vehicle without an automatic driving system. The relative pose calibration method provided by the embodiment of the application is described below.

Fig. 3 is a flowchart of a relative attitude calibration method according to an embodiment of the present application. As shown in fig. 3, the method includes:

301. the autopilot device captures images with a first camera and a second camera.

The first camera and the second camera are mounted at different positions of an automatic driving device. The autopilot device may be an autopilot vehicle; the device can also be an unmanned aerial vehicle and other devices which need to calibrate the relative pose between cameras.

302. And the automatic driving device performs characteristic point matching on the image acquired by the first camera and the image acquired by the second camera to obtain a first matched characteristic point set.

The first matching feature point set includes H sets of feature point pairs, each set including two matching feature points, one of the feature points being a feature point extracted from the image captured by the first camera, the other feature point being a feature point extracted from the image captured by the second camera, and H being an integer not less than 8. The automatic driving device performs feature point matching on the image acquired by the first camera and the image acquired by the second camera to obtain a first matching feature point set, and the feature point matching may be performed on two frames of images synchronously acquired by the first camera and the second camera to obtain the first matching feature point set. For example, a first camera acquires image 1 at a first point in time, and a second camera acquires image 2 at the first point in time; the automatic driving device extracts the feature points in the image 1 to obtain a first image feature point set 1, and extracts the feature points in the image 2 to obtain an image feature point set 2; then, the feature points in the image feature point set 1 and the image feature point set 2 are matched to obtain a matching feature point set (corresponding to the first matching feature point set).

In some embodiments, one way for the autopilot to implement step 302 is as follows: the automatic driving device synchronously acquires images by utilizing a first camera and a second camera to obtain a first image sequence and a second image sequence, wherein a plurality of images in the first image sequence correspond to images in the second image sequence one by one; and performing feature point matching on the images in the first image sequence and the second image sequence, wherein the images correspond to each other one by one, so as to obtain the first matching feature point set. For example, the images 1 to 5 in the first image sequence correspond to the images 6 to 10 in the second image sequence one by one, and the automatic driving device may perform feature matching on the feature points extracted from the image 1 and the feature points extracted from the image 6, and add the obtained feature point pairs to the first matching feature point set; performing feature matching on the feature points extracted from the image 2 and the feature points extracted from the image 7, and adding the obtained feature point pairs into the first matching feature point set; and so on. It should be understood that image 1 and image 6 are images captured simultaneously by the first and second cameras (i.e., corresponding to the same frame of image), image 2 and image 7 are images captured simultaneously by the first and second cameras (i.e., corresponding to the same frame of image), and so on. In practical application, the automatic driving device can perform characteristic point matching on multi-frame images synchronously acquired by the first camera and the second camera, and store the obtained characteristic point pairs into the first matched characteristic point set. The one frame image may include one image captured by the first camera and one image captured by the second camera.

303. The autopilot device determines a first relative pose between the first camera and the second camera based on the first set of matched feature points.

The first matching feature point set includes H sets of feature point pairs, each set including two matching feature points, one of the feature points being a feature point extracted from an image captured by the first camera and the other of the feature points being a feature point extracted from an image captured by the second camera, a field of view of the first camera and a field of view of the second camera overlap, the first camera and the second camera being mounted at different positions of an automatic driving apparatus, and H being an integer not less than 8.

304. And under the condition that the difference value between the first relative pose and the second relative pose is not greater than a pose change threshold value, the automatic driving device updates the second relative pose to the first relative pose.

The second relative pose is a relative pose between the first camera and the second camera currently stored by the autopilot device. The pose change threshold may be a maximum deviation threshold allowed for the multi-camera system to function properly, for example, the rotational angle deviation between the first relative pose and the second relative pose does not exceed 0.5 degrees. The method flow in fig. 3 describes a manner of determining the relative pose between a first camera and a second camera having a common field of view. It should be appreciated that the autopilot device may employ a method flow similar to that of fig. 3 to determine the relative pose between any two cameras having a common field of view. In a multi-camera vision system (corresponding to the camera 130 in fig. 1) of an autopilot, two cameras in different installation positions will generally have a common field of view therebetween, and the two cameras having the common field of view may be paired (default left camera in front and right camera in back) according to the common field of view distribution between the cameras. Fig. 4 is a schematic view of a common view between cameras according to an embodiment of the present disclosure. As shown in fig. 4, the camera having a common field of view includes: camera 3 and camera 1, camera 3 and camera 2, camera 1 and camera 2. Only the common field of view between camera 1 and camera 2 is shown in fig. 4. In some embodiments, the cameras in the multi-camera system may all be grouped in the above manner, resulting in multiple pairs of cameras, i.e., two cameras each having a common field of view are grouped together; then, the method flow in fig. 3 is adopted to obtain the relative pose between each pair of cameras.

The relative pose calibration method provided by the embodiment of the application can achieve the repetition precision of less than 0.1 degree, effectively reduces the times of re-calibrating the relative pose and has high reliability.

In fig. 3, how to determine the first relative pose between the first camera and the second camera according to the first matching feature point set is not described in detail. Some alternative implementations of step 303 are described below.

In a first mode

The manner in which the autopilot device determines the first relative pose between the first camera and the second camera based on the first set of matching feature points is as follows: determining an essential matrix between the first camera and the second camera; calculating a 5 Degree of Freedom (DOF) relative pose between the first camera and the second camera according to a singular value decomposition result of the intrinsic matrix; taking the ratio of the first distance to the second distance as a scale factor; the first distance and the second distance are the result of measuring the same distance by a non-vision sensor (e.g., laser radar) and a vision sensor (e.g., camera 130) on the automatic driving device, respectively, and the vision sensor includes the first camera and/or the second camera; and combining the 5-degree-of-freedom relative pose and the scale factor to obtain the first relative pose.

For example, the autopilot device may determine the essence matrix between the first camera and the second camera by:

1) feature points x' extracted from an image captured by a first camera and feature points x extracted from an image captured by a second camera are acquired from the first set of matching feature points. For example, x ═ x y1]^T，x′＝[x′ y′ 1]^T。

From the properties of the fundamental matrix between two cameras, the following equation holds:

x′Fx＝0 (1)；

wherein F represents a fundamental matrix between the first camera and the second camera,

when the above formula (1) is developed, the following steps are carried out:

x′xf₁₁+x′yf₁₂+x′f₁₃+y′xf₂₁+y′yf₂₂+y′f₂₃+xf₃₁+yf₃₂+f₃₃＝0 (2)；

writing the above equation in matrix form has:

(x′x，x′y，x′，y′x，y′y，y′，x，y，1)f＝0 (3)；

wherein f ═ f₁₁ f₁₂ f₁₃ f₂₁ f₂₂ f₂₃ f₃₁ f₃₂ f₃₃]^T。

The following matrix equation can be obtained for g sets of pairs of characteristic points:

2) solving the above matrix equation (4) yields a linear solution of the fundamental matrix F.

3) The basis matrix F is singular value decomposed using the following formula:

F＝UDV^T (5)；

where D is a diagonal matrix, all but the diagonal elements are 0, and the diagonal elements are represented by r, s, and t, then: d is diag (r, s, t), and r is more than or equal to s and more than or equal to t. Let t be 0 to give: f' ═ udiag (r, s, 0) V^TAnd F' is an optimal estimation that the fundamental matrix F satisfies the singularity constraint.

4) And (3) calculating by using a conversion relation between the basic matrix and the essential matrix to obtain an essential matrix E:

E＝K′^TFK (6)：

wherein K is the internal reference matrix of the first camera, K 'is the internal reference matrix of the second camera, and K' are both known quantities. The intrinsic matrix E is obtained from equation (6) above.

Optionally, the step of calculating, by the autopilot device, the 5-degree-of-freedom DOF relative pose between the first camera and the second camera according to the singular value decomposition result of the intrinsic matrix is as follows:

(1) and performing singular value decomposition on the intrinsic matrix by adopting the following formula to obtain a singular value decomposition result:

(2) and obtaining at least two relative poses between the first camera and the second camera according to the singular value decomposition result.

Assume that the pose of the first camera is P ═ Rt [ I0 ], where Rt is the transformation matrix of the first camera with respect to the vehicle body coordinate system, and I is the identity matrix of 3 × 3. Four possible forms of the pose P' of the second camera are as follows:

(3) and respectively ranging each feature point in the first matched feature point set by using a trigonometry method, and taking a pose P' which enables the three-dimensional coordinate positions of all the feature points to be positioned in front of the first camera and the second camera at the same time as the pose of the second camera.

The distance measurement of each feature point in the first matching feature point set by using the trigonometry may be: and determining a three-dimensional space coordinate according to each group of characteristic point pairs in the first matching characteristic point set by adopting a triangularization formula. And calculating a three-dimensional space coordinate obtained by a group of characteristic point pairs, namely the space coordinate corresponding to the two characteristic points included in the group of characteristic point pairs. Triangularization was first proposed by gaussians and applied in metrology. The method is as follows simply: the same three-dimensional point P (X, y, z) is observed at different positions, and the two-dimensional projection points X1(X1, y1) and X2(X2, y2) of the three-dimensional point observed at different positions are known, and the depth information, namely three-dimensional space coordinates, of the three-dimensional point is recovered by utilizing a triangular relation. Triangularization is mainly to calculate the three-dimensional coordinates of feature points in a camera coordinate system through matched feature points (namely pixel points). After determining the expression form of P ', the translation part (last column) due to P' is composed of

Is determined by

Is a normalized unit vector, so the translated portion of P' differs from the true value by a scale factor. At which time a 5 degree-of-freedom pose between the two cameras is determined. In this way, the 5-degree-of-freedom relative pose between the first camera and the second camera can be accurately and quickly determined.

The relative pose between the two cameras has 6 degrees of freedom, and the 5DOF relative pose between the two cameras can be obtained through the decomposition of the essential matrix, namely

The relative rotation angle between two cameras can be accurately obtained through R, and the relative position between two cameras can only obtain a normalization result, namely, the normalization result is different from a real value by a scale factor. Determining the scale factor requires relying on the output of other sensors. Optionally, a non-visual sensor is recorded to obtain a measurement distance s, the obtained relative pose of the camera is utilized, the same measurement distance s' can be obtained through a multi-camera three-dimensional reconstruction technology, and the scale factor is

The 6DOF relative pose between the two cameras is

One possible approach to determining the scale factor is as follows: the distance s between a certain target and the automatic driving device is measured by using a laser radar or a millimeter wave radar, and the distance s' between the same target and the automatic driving device is obtained by using a camera through binocular (or multi-view) vision measurement, so that a scale factor is obtained.

Another possible approach to determine the scale factor is as follows: the moving distance s of the automatic driving device is measured by an Inertial Measurement Unit (IMU) or a wheel speed meter, and a distance difference s' between a first distance of a certain stationary target and the automatic driving device at a first time and a second distance of the stationary target and the automatic driving device at a second time is obtained by binocular (or multi-view) vision Measurement. The first time is the starting time of the moving distance s of the automatic driving device, and the second time is the ending time of the moving distance s of the automatic driving device.

Mode two

The manner in which the autopilot device determines the first relative pose between the first camera and the second camera based on the first set of matching feature points is as follows: iteratively solving an objective equation to obtain the first relative pose; in the object equation, the first relative pose includes an unknown parameter, and the pair of feature points in the first matching feature point set, the internal parameter of the first camera, and the internal parameter of the second camera are known parameters.

Assuming that in the first set of matching feature points, the feature point extracted from the image captured by the first camera is x', and the feature point extracted from the image captured by the second camera is x. For example, x ═ x y1]T，x′＝[x′ y′ 1]^T。

The intrinsic matrix E between the first camera and the second camera is formed by the following equation:

the two sides of the equal sign of the above formula are expressions of E, wherein K is an internal reference matrix of the first camera, K' is an internal reference matrix of the second camera,

r denotes a rotation matrix between the first camera and the second camera, t denotes a translation vector between the first camera and the second camera, and t ═ t₁ t₂ t₃]^T. The fundamental matrix F between the first camera and the second camera is formed by the following equation:

x′Fx＝0 (13)；

the elimination of F in equation (13) using equation (12) yields the following objective equation:

wherein K is an internal reference matrix of the first camera, K 'is an internal reference matrix of the second camera, K and K' are both known quantities, 6 parameters (namely 6 parameters forming relative pose) are totally recorded in the target equation, and are recorded as J, and the gradient of the target equation is recorded as J

Optionally, the target equation is iteratively solved by using a gradient descent method by using the following formula to obtain the first relative pose:

where α is a step size parameter, which can be set empirically. Substituting x' and x into equation (15) optimizes the update

The step is independent of the number of x' and x points, and even if the number of feature points in the first matching feature point set is small (such as in a special scene like a tunnel), the step can still be used for matching the feature points

Optimization was performed until stable results were obtained.

In some embodiments, after the autopilot has derived the relative pose between two cameras having a common field of view (e.g., by performing the method flow of fig. 1), the global optimization can be performed using a closed loop set of equations formed by the relative pose between each pair of cameras having a common field of view. The following takes camera 1 (i.e., the first camera), camera 2 (i.e., the second camera), and camera 3 (i.e., the third camera) as an example to describe how to optimize the obtained relative pose between two cameras.

Assume pose between camera i and camera j as

Rt_ijRepresenting a transformation matrix, R, in the coordinate system from camera i to camera j_ijRepresenting a rotation matrix between camera i and camera j,

representing a translation matrix between camera i and camera j. Illustratively, the pose relationships among the cameras 1, 2, and 3 are as follows:

Rt₃₁Rt₂₃＝Rt₂₁ (16)；

three Rts form a closed loop, and a better result can be obtained through integral optimization. All three transformation matrices can be calculated separately from the common view relationship (e.g., using the method flow of fig. 1), and thus equation (16) is a closed-loop equation. Better results can be obtained by optimizing the above equations. For a system composed of more cameras, as long as two cameras have a common view and can form a closed loop, the optimization can be carried out in the above mode. If camera a, camera b, camera c, camera z, two adjacent cameras have a common field of view, then there is a closed loop equation: rt_abRt_bcRt_cd...Rt_yzRt_za0. The manner how the relative pose between cameras is optimized using the closed loop equations formed by the transformation matrices between cameras is described below.

Taking camera 1 (i.e., the first camera), camera 2 (i.e., the second camera), and camera 3 (i.e., the third camera) as an example, the closed loop equation formed by these three cameras is shown as equation (16), and the following two equations can be obtained by expanding equation (16):

R₃₁R₂₃＝R₂₁ (17)；

it is to be understood that if R is known₂₃And R₂₁Can find R₃₁Can then utilize

R₃₁To obtain

In some embodiments, Rt may be₃₁Regarded as unknown quantity, Rt₂₃Relative poses, Rt, obtained for matched feature points in a set of images (corresponding to one frame) taken with camera 2 and camera 3 in synchronization₂₁The relative poses obtained for the matched feature points in a set of images synchronously captured by the camera 2 and the camera 1. Using equation (17) and equation (18), the following equation can be obtained:

exemplary, R₂₁Is a first rotation matrix, R₂₃In order to be the second rotation matrix, the first rotation matrix,

in order to be the first translation matrix,

is a second translation matrix. Assume that the autopilot has obtained M Rts using different sets of images₂₃And M Rts₂₁Then, from the formula (19) and the formula (20), the following equation system can be obtained:

from equation (21), R₂₁And R₂₃In order to be a known number of,

only with R₃₁The correlation can be solved by Newton method or gradient descent method

In part obtain R₃₁The optimal solution of (1). Wherein the equation set (21) is a first equation set, R₃₁Is a third rotation matrix. As can be seen from the system of equations (22),

and

is a known number at R₃₁When the information is known, the information is transmitted to the mobile terminal,

only with

Related to, solving for

To obtain

The optimal solution of (1). Wherein the system of equations (22) is a second system of equations,

is a third translation matrix. In some embodiments, the autopilot may utilize two images (corresponding to one frame) captured simultaneously by camera 1 and camera 2 to determineDetermining an Rt₂₁. It will be appreciated that the autopilot device can determine the M Rt's using M sets of images (corresponding to M frames) acquired in synchronism₂₁. Similarly, the autopilot can determine M Rts₂₃。

Optionally, mixing Rt₃₁Considering the known quantity, the result obtained in the previous step is substituted into equations (17) and (18), a similar method is used to construct the equation set and Rt is used₂₁As the quantity to be optimized, Rt can be determined₃₁Rt in known cases₂₁The optimal solution of (1).

Optionally, mixing Rt₃₁And Rt₂₁All are regarded as known quantities, and a similar method is adopted to construct an equation set and Rt is obtained₂₃As the quantity to be optimized, Rt can be determined₂₃The optimal solution of (1). To this end, a set of closed loop optimizations is completed. It will be appreciated that the main principle of this approach is: firstly, a rotation matrix between a pair of two cameras with a common visual field is regarded as an unknown number, rotation matrixes and translation matrixes between other pairs of two cameras with the common visual field are regarded as known numbers, and the unknown numbers are solved through an equation set (21); then, the translation matrix between the pair of two cameras with the common visual field is regarded as an unknown number, the rotation matrix and the translation matrix between the other pairs of two cameras with the common visual field are regarded as known numbers, the rotation matrix between the pair of two cameras with the common visual field is regarded as a known number, and the unknown number is solved through an equation set (22).

It should be understood that the number between the cameras is arbitrarily specified, so the order of the cameras can be changed and the above steps repeated, different solutions of the poses between multiple groups of cameras are obtained, and a group of solutions with the smallest errors is selected as output. Taking 4 cameras as an example, noted A, B, C, D, assuming that there is a common view between 4 cameras, there are many forms of forming a closed loop equation, such as Rt_ADRt_DCRt_CBRt_BA＝0，Rt_ADRt_DBRt_BCRt_CAAnd (4) respectively solving different closed-loop equations to obtain different solutions, and selecting the optimal solution. The error is the residual magnitude of the equation.

In practical application, after the relative poses of a plurality of pairs of cameras are obtained, the automatic driving device can optimize the relative poses of the cameras in a similar mode so as to obtain the relative poses with higher precision.

The foregoing embodiments describe a manner of determining a relative pose between a first camera and a second camera using two images acquired simultaneously by the first camera and the second camera. When the number of the feature point pairs matched with the two images synchronously acquired by the first camera and the second camera is not less than 8, a more accurate relative pose can be obtained. However, when the number of pairs of feature points matching two images acquired by the first camera and the second camera synchronously is less than 8, an accurate relative pose cannot be obtained, i.e., pose estimation is unstable. A way of determining the relative pose between two cameras by multi-frame accumulation is described below, by which a more accurate relative pose can be obtained. One way to determine the relative pose between two cameras through multi-frame accumulation is as follows:

1) extracting characteristic points in the two pictures of the ith frame, and respectively recording the characteristic points as x'_iAnd x_iAdding it to the feature point set X₁And a set of feature points X₂。

The two images of the ith frame comprise an image acquired by the first camera in the ith frame and an image acquired by the second camera in the ith frame. The characteristic point extraction in the two images of the ith frame can be the characteristic point extraction in the image acquired by the first camera in the ith frame, and x 'is obtained'_i(ii) a Extracting the characteristic points of the second camera in the image acquired by the ith frame to obtain x_i. i is an integer greater than 0. It should be understood that two images in a frame include one image captured by the first camera and one image captured by the second camera, and the time interval between which the two images in the frame are captured is less than a time threshold, such as 0.5ms, 1ms, etc.

2) Extracting characteristic points in the two graphs of the j frame, and respectively recording the characteristic points as x'_jAnd x_jAdding it to the feature point set X₁And a set of feature points X₂。

j is an integer greater than 1 and not equal to i. It should be understood that after feature points in two images of N frames are continuously extracted, the feature point set X is obtained₁And a set of feature points X₂The matched feature points in the two images of the N frames are stored.

3) Using a set of feature points X₁And a set of feature points X₂And estimating the relative pose of the two cameras.

Optionally, one implementation manner of step 3) may be to set the feature points X₁Feature point and feature point set X in (1)₂Carrying out feature point matching on the feature points to obtain a plurality of groups of feature point pairs; and determining the relative pose between the two cameras by utilizing the plurality of groups of characteristic point pairs. Using the sets of pairs of feature points, an implementation of determining the relative pose between the two cameras may be similar to the implementation of step 303.

Due to the feature point set X₁And X₂The number of the feature points in the image is far more than that of the feature points of a single frame, so that the estimation of the relative pose of the two cameras can be carried out in an iterative optimization mode: first, a feature point set X is used₁And X₂Carrying out pose estimation on all the middle points to obtain a first intermediate pose; then substituting the first intermediate pose into a formula (14) to calculate a residual error, and eliminating the characteristic points of which the residual errors are larger than a residual error threshold value; performing pose estimation again by using the residual feature points to obtain a second intermediate pose; the above processes are repeated until the number of the feature points to be eliminated is less than the number threshold (e.g. 5% of the total number). The unit of equation (14) is pixel, and the residual threshold may be 0.8, 1, 1.2, etc., and the present application is not limited thereto. Finally, a more accurate relative pose under the multi-frame image can be obtained. Here the pose estimation may be implemented in the same way as in step 303.

The central limit theorem indicates that the mean of a large number of mutually independent random variables converges to a normal distribution. Suppose that the sum of R calculated between different frames in a long observation period

Are independently and equally distributed. Recording the 5DOF relative pose obtained by solving the ith frame as

Then, according to the central limit theorem, memorize

Then there are:

where N (0, 1) represents a standard normal distribution. Note the book

It can be seen that the variance of X is equal to

In inverse proportion. n is the size of the time window (corresponding to the acquired n frames of images), and as the time window increases, the variance of X may decrease continuously and may be stable at a certain value in practical application. According to the central limit theorem, the sum of R calculated between different frames in long-time observation is assumed

Are independently and identically distributed, then as n increases, R and

will tend to level off. Choosing the appropriate n such that R and

the variance of (a) can meet the requirement.

The relative pose between the two cameras is determined by adopting a multi-frame accumulation mode, so that the problem of jitter caused by single-frame calculation can be avoided, and the calibration result tends to be stable.

How to achieve the recalibration of the relative pose between two cameras is described below in connection with the structure of the autopilot device. Fig. 5 is a schematic structural diagram of an automatic steering device according to an embodiment of the present application. As shown in fig. 5, the automatic driving apparatus includes: an image acquisition unit (corresponding to the first camera and the second camera) 501, an image feature point extraction unit 502, a pose calculation unit 503, a closed-loop optimization unit 504, a multi-frame accumulation optimization unit 505, a scale calculation unit 506, and a calibration parameter update unit 507. The closed-loop optimization unit 504 and the multi-frame accumulation optimization unit 505 are optional and not necessary. In some embodiments, the functions of the image feature point extraction unit 502, the pose calculation unit 503, the closed-loop optimization unit 504, the multi-frame accumulation optimization unit 505, the scale calculation unit 506, and the calibration parameter update unit 507 are all implemented by the processor 112 (corresponding to an onboard processor). In some embodiments, the functions of the pose calculation unit 503, the closed-loop optimization unit 504, the multi-frame accumulation optimization unit 505, the scale calculation unit 506, and the calibration parameter update unit 507 are all implemented by the processor 112 (corresponding to an onboard processor); the function of the image feature point extraction unit 502 is realized by a graphics processor. The image feature point extraction unit 502, the pose calculation unit 503, the closed-loop optimization unit 504, the multi-frame accumulation optimization unit 505, the scale calculation unit 506, and the calibration parameter update unit 507 are units of the autopilot apparatus that determine the relative pose between two cameras, that is, an online calibration portion (a portion for realizing the relative pose recalibration). The functions of the units are described separately below.

An image acquisition unit 501 is used for acquiring images acquired by a plurality of cameras. Illustratively, the image acquisition unit is used for acquiring images acquired by the cameras synchronously. The plurality of cameras correspond to the cameras 130 in fig. 1.

The image feature point extracting unit 502 is configured to extract feature points in images acquired by multiple cameras, and match the extracted feature points to obtain a first matching feature point set. For example, the image feature point extracting unit is configured to extract feature points in a first image acquired by a first camera to obtain a first feature point set, and extract feature points in a second image acquired by a second camera to obtain a second feature point set; the feature points in the first feature point set and the feature points in the second feature point set are matched to obtain a plurality of sets of feature point pairs (corresponding to the first matched feature point set). For another example, the image feature point extracting unit is configured to extract feature points in a plurality of sets of images captured by the first camera and the second camera synchronously, and perform feature point matching to obtain a plurality of sets of feature point pairs (corresponding to the first matched feature point set), where each set of images (corresponding to one frame) includes one image captured by the first camera and one image captured by the second camera.

And a pose calculation unit 503, configured to determine a basic matrix between every two cameras according to the first matching feature point set (i.e., the matched feature points), and decompose the basic matrix to obtain a 5DOF relative pose. It should be understood that the pose calculation unit may perform pose calculation for all cameras in the multi-camera system having a common view, i.e. determine the relative pose between all cameras in the multi-camera system having a common view. In case the autopilot device does not comprise the closed-loop optimization unit 505 and the multi-frame accumulation optimization unit 506, the pose calculation unit 503 is further adapted to merge the scale factors and the 5DOF relative poses to get a complete 6DOF relative pose. Exemplarily, the pose calculation unit 503 is specifically configured to determine an essential matrix between the first camera and the second camera; and calculating the 5-degree-of-freedom DOF relative pose between the first camera and the second camera according to the singular value decomposition result of the intrinsic matrix. Optionally, the pose calculation unit 503 is specifically configured to perform singular value decomposition on the essential matrix to obtain a singular value decomposition result; obtaining at least two relative poses between the first camera and the second camera according to the singular value decomposition result; calculating the three-dimensional coordinate positions of the feature points in the first matching feature point set by using the at least two relative poses respectively; and setting, as the 5-degree-of-freedom relative pose, a relative pose of the at least two relative poses such that the three-dimensional coordinate position of each feature point in the first set of matched feature points is located in front of the first camera and the second camera.

A scale calculation unit 504 for taking a ratio between the first distance and the second distance as a scale factor; the first distance and the second distance are the result of measuring the same distance by a non-vision sensor and a vision sensor on the automatic driving device, respectively, and the vision sensor includes the first camera and/or the second camera. Illustratively, the scale calculation unit 504 is specifically configured to measure a distance s from a target to the autopilot device by using a laser radar or a millimeter wave radar, and obtain a distance s' from the same target to the autopilot device by using a camera through binocular (or multi-view) vision measurement, thereby obtaining the scale factor. It should be appreciated that merging the scale factors and 5DOF relative poses results in a complete 6DOF relative pose.

And the closed-loop optimization unit 505 is configured to construct an equation set according to a closed loop formed by the relative poses of every two cameras in the multi-camera system, and perform overall optimization. Illustratively, the closed-loop optimization unit 505 optimizes the relative pose obtained by executing the method flow in fig. 3 by solving equation set (21) and equation set (21) in sequence. In case the autopilot device comprises the closed-loop optimization unit 505 and does not comprise the multi-frame accumulation optimization unit 506, the closed-loop optimization unit 505 is further configured to combine the scale factors and the 5DOF relative pose to arrive at a complete 6DOF relative pose.

And the multi-frame accumulation optimizing unit 506 is configured to accumulate, filter, and optimize the multi-frame calculation result to obtain a stable calibration parameter (corresponding to the relative pose). Illustratively, the multi-frame accumulation optimization unit 406 is used to implement a way of determining the relative pose between two cameras by multi-frame accumulation. In the case where the autonomous driving apparatus includes the multi-frame accumulation optimization unit 505, the multi-frame accumulation optimization unit 505 may combine the scale factor and the 5DOF relative pose to obtain a complete 6DOF relative pose.

And a calibration parameter updating unit 507, configured to update the relative pose currently used by the autopilot to the calculated relative pose when a difference between the calculated relative pose and the relative pose currently used by the autopilot is not greater than a pose change threshold. That is, the calibration parameter updating unit 507 is configured to compare the calculated relative pose (corresponding to the calibration parameter) with the relative pose currently used by the automatic driving apparatus, and dynamically update the relative pose currently used (or issue a prompt at the same time) if the difference is too large.

The autopilot of fig. 5 may use approach one to determine the relative pose between two cameras. The following describes a configuration of an autopilot device that can determine the relative pose between two cameras in a second manner. Fig. 6 is a schematic structural diagram of another automatic steering device according to an embodiment of the present application. As shown in fig. 6, the automatic driving apparatus includes:

an image acquisition unit 601 for acquiring images by a first camera and a second camera, respectively; a field of view of the first camera and a field of view of the second camera overlap, the first camera and the second camera being mounted at different positions of an autopilot device;

an image feature point extracting unit 602, configured to perform feature point matching on the image acquired by the first camera and the image acquired by the second camera to obtain a first matching feature point set; the first matching feature point set includes H sets of feature point pairs, each set of feature point pairs including two matching feature points, one of the feature points being a feature point extracted from the image captured by the first camera, the other feature point being a feature point extracted from the image captured by the second camera, and H being an integer not less than 8;

a pose calculation unit 603 configured to determine a first relative pose between the first camera and the second camera according to a first matching feature point set; the first matching feature point set includes H sets of feature point pairs, each set including two matching feature points, one of the feature points being a feature point extracted from an image captured by the first camera and the other of the feature points being a feature point extracted from an image captured by the second camera, a field of view of the first camera and a field of view of the second camera being overlapped, the first camera and the second camera being installed at different positions of an automatic driving apparatus, and H being an integer not less than 8;

a calibration parameter updating unit 604 (corresponding to the calibration parameter updating unit 507) configured to update the second relative pose to the first relative pose if a difference between the first relative pose and the second relative pose is not greater than a pose change threshold; the second relative pose is a relative pose between the first camera and the second camera currently stored by the autopilot device.

In some embodiments, the functions of the pose calculation unit 603 and the calibration parameter update unit 604 are both implemented by the processor 112 (corresponding to an onboard processor).

Optionally, the automatic driving apparatus in fig. 6 further includes: a closed loop optimization unit 605 and a multi-frame accumulation optimization unit 606. Wherein the closed loop optimization unit 605 and the multi-frame accumulation optimization unit 606 are optional, not necessary. The function of the image capturing unit 601 may be the same as that of the image capturing unit 501, the function of the image feature point extracting unit 602 may be the same as that of the image feature point extracting unit 502, the function of the closed-loop optimizing unit 605 may be the same as that of the closed-loop optimizing unit 505, and the function of the multi-frame accumulation optimizing unit 606 may be the same as that of the multi-frame accumulation optimizing unit 506.

Exemplarily, the pose calculation unit 603 is specifically configured to iteratively solve the objective equation to obtain the first relative pose; in the object equation, the first relative pose includes an unknown parameter, and the pair of feature points in the first matching feature point set, the internal parameter of the first camera, and the internal parameter of the second camera are known parameters. It should be appreciated that the pose calculation unit 603 may determine the relative pose between two cameras having a common field of view in a second manner.

It should be understood that the above division of the units in the automatic driving device is only a division of logical functions, and the actual implementation can be wholly or partially integrated into one physical entity or can be physically separated. For example, each of the above units may be a processing element separately set up, or may be implemented by being integrated into a chip of the automatic driving apparatus, or may be stored in a storage element of the controller in the form of program code, and a processing element of the processor calls and executes the functions of each of the above units. In addition, the units can be integrated together or can be independently realized. The processing element may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the method or the units above may be implemented by hardware integrated logic circuits in a processor element or instructions in software. The processing element may be a general-purpose processor, such as a Central Processing Unit (CPU), or may be one or more integrated circuits configured to implement the above method, such as: one or more application-specific integrated circuits (ASICs), one or more microprocessors (DSPs), one or more field-programmable gate arrays (FPGAs), etc.

Fig. 7 is a schematic structural diagram of another automatic steering device according to an embodiment of the present application. As shown in fig. 7, the automatic driving apparatus includes: the system comprises a multi-camera visual system 701, an image processor 702, an image feature extractor 703, an on-board processor 704, a memory 705, a calibration parameter updating device 706 and a parameter abnormality reminding device 707.

A multi-camera vision system 701 for images acquired simultaneously by a plurality of cameras. The multi-camera vision system 701 corresponds to the camera 130 in fig. 1. The multi-camera vision system is composed of a plurality of cameras and can acquire image data respectively and synchronously.

An image processor 702 for performing preprocessing such as scaling on the image data acquired by the multi-camera vision system 701.

An image feature extractor 703, configured to perform feature point extraction on the image preprocessed by the image processor 702.

To increase the speed, the pre-processed image may enter a special image feature extractor 703 for feature point extraction. The image feature extractor 703 may be a graphics processor with general purpose computing capabilities or specially designed hardware such as an ISP.

And the on-board processor 704 is configured to determine a relative pose between each two cameras according to the feature points extracted by the image feature extractor 703.

The on-board processor (corresponding to processor 113 in fig. 1) may be a hardware platform with general purpose computing capabilities. Optionally, the on-board processor includes a pose estimation section and an optimization processing section. The pose estimation section (corresponding to the pose calculation unit 601 in fig. 6 or the pose calculation unit 503 and the scale calculation unit 504 in fig. 5) is used to determine the relative pose between two cameras, i.e., the relative pose estimation. And the optimization processing part is used for realizing closed-loop constraint optimization and/or multi-frame accumulation optimization. Optionally, the on-board processor includes a pose estimation portion and does not include an optimization processing portion. The on-board processor is further configured to read the calibration parameters (i.e., the currently stored relative pose) from the memory 705 (corresponding to the memory 114 in fig. 1), and compare the read calibration parameters (i.e., the relative pose) with the calculated calibration parameters (i.e., the relative pose). And then the vehicle-mounted processor respectively sends the comparison result to the calibration parameter updating device 706 and the parameter abnormity reminding device 707. The calibration parameter updating device 706 (corresponding to the calibration parameter updating unit 507 in fig. 5 and the calibration parameter updating unit 602 in fig. 6) can update the calibration parameters in the memory as needed. The parameter abnormality reminding device 707 may remind the driver of the abnormality of the calibration parameter of the camera by sound or image. For example, the onboard processor may alert the driver that the relative pose between the first camera and the second camera is abnormal when it determines that the relative pose between the first camera and the second camera differs significantly from the relative pose between the first camera and the second camera currently in use by the autopilot device. Taking a vehicle-mounted application scenario as an example, if the difference between the calibration parameters calculated by the vehicle-mounted processor and the parameters calibrated when the vehicle leaves the factory is too large, if the angular deviation exceeds 0.3 degrees, it can be considered that the sensor is installed loosely, which may cause the performance of the automatic driving system to be degraded, and the vehicle owner needs to be reminded to re-calibrate to a 4s shop (for example, a warning window pops up in the central control). If the owner does not recalibrate for a long time and the angle deviation is not large, such as greater than 0.3 degrees but less than 0.5 degrees, the automatic driving system can be operated by replacing the factory calibration parameters with the online calibration parameters under the condition that the owner allows.

The autopilot of fig. 7 can determine the relative pose between two cameras in a first mode or in a second mode.

Fig. 8 is a flowchart of another relative pose calibration method provided in the embodiment of the present application. The method of fig. 8 is a further refinement and refinement of the method of fig. 3. As shown in fig. 8, the method includes:

801. the autopilot device determines a pair of cameras having a common field of view.

802. And extracting feature points from the images synchronously acquired by the pair of cameras, and matching the feature points.

For example, a first camera and a second camera have a common view, the first image and the second image are images (corresponding to one frame) synchronously acquired by the first camera and the second camera, feature points in the first image are extracted to obtain a third feature point set, and feature points in the second image are extracted to obtain a fourth feature point set; and matching the characteristic points in the third characteristic point set with the characteristic points in the fourth characteristic point set to obtain a plurality of groups of characteristic point pairs. It should be understood that the images synchronously acquired by each pair of cameras can obtain a plurality of groups of feature point pairs corresponding to the pair of cameras.

803. And calculating the essential matrix between a pair of cameras through the corresponding multiple groups of characteristic point pairs of the camera.

804. From the essential matrix between a pair of cameras, the 5DOF relative pose between the pair of cameras is determined.

The implementation of

steps

803 and 804 can be the same as the way of determining the 5DOF relative pose between a pair of cameras having a common field of view in the first way, and will not be described in detail here. Optionally, the steps 802 to 804 may be replaced by determining the relative pose between the two cameras by means of multi-frame accumulation in the foregoing embodiment. It should be appreciated that each time the autopilot performs steps 801-804, the 5DOF relative pose between a pair of cameras having a common field of view can be determined.

805. And constructing a multi-camera system closed loop equation.

Illustratively, equations (19) and (20) are a set of multi-camera system closed loop equations constructed.

806. The solution is solved in accordance with the closed loop equation of the multi-camera system.

Step 805 and step 806 are closed loop construction equations formed by the relative poses between each pair of cameras with the common view, and the calculated 5DOF relative pose between each pair of cameras is optimized. The foregoing embodiments describe a way to perform global optimization using a closed loop set of construction equations composed of relative poses between pairs of cameras with a common view.

807. A scale factor is calculated and the 5DOF relative pose and the scale factor are merged to arrive at a 6DOF relative pose.

808. And judging whether to update the currently used calibration parameters (namely the relative pose).

If yes, go to step 809; if not, go to step 810.

809. And updating the currently used calibration parameters.

Updating the currently used calibration parameters may be updating the currently used calibration parameters to the calculated relative pose (i.e. new calibration parameters).

810. And outputting information for reminding a driver of abnormal conditions of the calibration parameters of the camera.

811. The flow is ended.

In the embodiment of the application, a pose constraint equation between the cameras is constructed by utilizing the public vision between every two multiple cameras, such as a formula (4), 5DOF relative poses of the two cameras are obtained by decomposing an essential matrix, and the public vision information between the cameras is fully utilized. In addition, 5DOF pose information is obtained by utilizing a characteristic point matching technology and is optimized through a closed loop equation, so that high recalibration precision is achieved.

The foregoing embodiments describe the manner in which the autopilot device determines the relative pose between two cameras. In some embodiments, the autopilot device may send information to the server necessary to determine the relative pose between two cameras, the relative pose between two cameras on the autopilot device being determined by the server from the information sent by the autopilot device. The following describes a scheme for determining the relative pose between two cameras on an autopilot device by a server.

Fig. 9 is a flowchart of another relative pose calibration method provided in the embodiment of the present application. As shown in fig. 9, the method includes:

901. and the automatic driving device sends a pose calibration request to the server.

Optionally, the pose calibration request is used to request to recalibrate a relative pose between the first camera and the second camera of the autopilot, and the pose calibration request carries internal parameters of the first camera and internal parameters of the second camera. Optionally, the pose calibration request is used to request to recalibrate a relative pose between the cameras on the autopilot, and the pose calibration request carries internal parameters of the cameras on the autopilot.

902. And the server sends confirmation information aiming at the pose calibration request to the automatic driving device.

The server sends confirmation information for the pose calibration request to the automatic driving device under the condition that the pose calibration request from the automatic driving device is determined to be accepted. Optionally, the server receives the pose calibration request from the automatic driving device and sends confirmation information for the pose calibration request to the automatic driving device when determining that the automatic driving device is an authorization device according to the pose calibration request. The authorization device refers to an automatic driving device for which the server needs to provide pose calibration service (i.e. recalibrate relative pose).

903. And the automatic driving device sends recalibration reference information to the server.

Optionally, the recalibration reference information is used for the server to determine a relative pose between a first camera and a second camera of the autopilot device, a visual field of the first camera and a visual field of the second camera overlap, and the first camera and the second camera are installed at different positions of the autopilot device. For example, the recalibration reference information includes a plurality of pairs of feature points (corresponding to a first matching feature point set) obtained by performing feature point matching on images synchronously acquired by the first camera and the second camera or by the automatic driving device according to the images synchronously acquired by the first camera and the second camera. Optionally, the recalibration reference information is used for the server to determine the relative pose between the cameras on the automatic driving device, and the cameras are installed at different positions of the automatic driving device. Illustratively, the recalibration reference information comprises images synchronously acquired by the cameras or a plurality of pairs of feature points obtained by the automatic driving device according to feature point matching of the images synchronously acquired by the cameras.

904. And the server obtains a first matching feature point set according to the recalibration reference information.

Optionally, the first matching feature point set includes H groups of feature point pairs, each group of feature point pairs includes two matching feature points, one of the feature points is a feature point extracted from an image captured by the first camera, the other feature point is a feature point extracted from an image captured by the second camera, and H is an integer not less than 8. Optionally, the recalibration reference information includes a first matching feature point set. Optionally, the recalibration reference information includes images synchronously acquired by the first camera and the second camera, and the server may perform feature point matching on the images synchronously acquired by the first camera and the second camera to obtain the first matching feature point set.

905. And the server determines a first relative pose between the first camera and the second camera according to the first matching feature point set.

The implementation of step 905 may be the same as the implementation of step 303. The server may determine the first relative pose between the first camera and the second camera in a first manner or a second manner, or may determine the first relative pose between the first camera and the second camera in other manners, which is not limited in the embodiment of the present application.

906. And the server sends the first relative pose to the automatic driving device.

907. And under the condition that the difference value between the first relative pose and the second relative pose is not greater than a pose change threshold value, the automatic driving device updates the second relative pose to the first relative pose.

The second relative pose is a relative pose between the first camera and the second camera currently stored by the autopilot device. It should be appreciated that the server may determine the relative pose between two cameras on the autonomous device based on information sent by the autonomous device. That is to say, the manner in which the automatic driving apparatus determines the relative pose between each pair of cameras in the foregoing embodiments may be implemented by a server, for example, a closed loop set of equations formed by the relative poses between each pair of cameras with a common view is used to optimize the calculated manner of the 5DOF relative pose between each pair of cameras, and the manner in which the relative pose between each pair of cameras is determined by multi-frame accumulation.

Fig. 10 is a schematic structural diagram of a server according to an embodiment of the present application, and as shown in fig. 10, the server includes: a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004; the memory 1001, the processor 1002, and the communication interface 1003 are communicatively connected to each other via a bus 1004. Communication interface 1003 is used for data interaction with the autopilot.

The processor 1003 is configured to perform the following operations by reading the code stored in the memory: receiving recalibration reference information from an autopilot device, the recalibration reference information being used by the server to determine a relative pose between a first camera and a second camera of the autopilot device, a field of view of the first camera and a field of view of the second camera overlapping, the first camera and the second camera being mounted at different locations on the autopilot device; obtaining a first matching feature point set according to the recalibration reference information, wherein the first matching feature point set comprises H groups of feature point pairs, each group of feature point pairs comprises two matched feature points, one feature point is a feature point extracted from an image acquired by a first camera, the other feature point is a feature point extracted from an image acquired by a second camera, and H is an integer not less than 8; determining a first relative pose between the first camera and the second camera according to the first matching feature point set; and sending the first relative pose to the automatic driving device.

The embodiment of the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer is enabled to execute the relative pose calibration method provided in the foregoing embodiment.

Optionally, the instructions when executed on a computer may implement: determining a first relative pose between the first camera and the second camera according to a first matching feature point set; the first matching feature point set includes H sets of feature point pairs, each set including two matching feature points, one of the feature points being a feature point extracted from an image captured by the first camera and the other of the feature points being a feature point extracted from an image captured by the second camera, a field of view of the first camera and a field of view of the second camera being overlapped, the first camera and the second camera being installed at different positions of an automatic driving apparatus, and H being an integer not less than 8; updating the second relative pose to the first relative pose when the difference between the first relative pose and the second relative pose is not greater than a pose change threshold; the second relative pose is a relative pose between the first camera and the second camera currently stored by the autopilot device.

Optionally, the instructions when executed on a computer may implement: receiving recalibration reference information from an autopilot device, the recalibration reference information being used by the server to determine a relative pose between a first camera and a second camera of the autopilot device, a field of view of the first camera and a field of view of the second camera overlapping, the first camera and the second camera being mounted at different locations on the autopilot device; obtaining a first matching feature point set according to the recalibration reference information, wherein the first matching feature point set comprises H groups of feature point pairs, each group of feature point pairs comprises two matched feature points, one feature point is a feature point extracted from an image acquired by a first camera, the other feature point is a feature point extracted from an image acquired by a second camera, and H is an integer not less than 8; determining a first relative pose between the first camera and the second camera according to the first matching feature point set; and sending the first relative pose to the automatic driving device.

The embodiment of the present application provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the relative pose calibration method provided by the foregoing embodiment.

While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A relative attitude calibration method is characterized by comprising the following steps:

capturing images by a first camera and a second camera; the field of view of the first camera and the field of view of the second camera overlap, the first camera and the second camera being mounted at different locations on an autopilot device;

performing feature point matching on the image acquired by the first camera and the image acquired by the second camera to obtain a first matched feature point set; the first matching feature point set comprises H groups of feature point pairs, each group of feature point pairs comprises two matched feature points, one feature point is a feature point extracted from the image acquired by the first camera, the other feature point is a feature point extracted from the image acquired by the second camera, and H is an integer not less than 8;

determining a first relative pose between the first camera and the second camera according to the first set of matched feature points;

updating the second relative pose to the first relative pose if the difference between the first relative pose and the second relative pose is not greater than a pose change threshold; the second relative pose is a relative pose between the first camera and the second camera currently stored by the autopilot device.

2. The method of claim 1, wherein the determining a first relative pose between the first camera and the second camera from the first set of matched feature points comprises:

determining an essential matrix between the first camera and the second camera;

calculating a 5-degree-of-freedom relative pose between the first camera and the second camera according to a singular value decomposition result of the intrinsic matrix;

taking the ratio of the first distance to the second distance as a scale factor; the first distance and the second distance are respectively the result of measuring the same distance by a non-vision sensor and a vision sensor on the automatic driving device, and the vision sensor comprises the first camera and/or the second camera;

merging the 5-degree-of-freedom relative pose and the scale factor to obtain the first relative pose.

3. The method of claim 2, wherein the calculating the 5-degree-of-freedom relative pose between the first camera and the second camera from the singular value decomposition results of the essential matrix comprises:

performing singular value decomposition on the intrinsic matrix to obtain a singular value decomposition result;

obtaining at least two relative poses between the first camera and the second camera according to the singular value decomposition result;

calculating the three-dimensional coordinate positions of the feature points in the first matching feature point set by using the at least two relative poses respectively;

taking, as the 5-degree-of-freedom relative pose, one of the at least two relative poses at which the three-dimensional coordinate position of each feature point in the first set of matched feature points is located in front of the first camera and the second camera.

4. The method of claim 1, wherein the determining a first relative pose between the first camera and the second camera from the first set of matched feature points comprises:

iteratively solving an objective equation to obtain the first relative pose; in the object equation, the first relative pose includes parameters that are unknowns, and the pairs of feature points in the first matching feature point set, the internal parameters of the first camera, and the internal parameters of the second camera are known numbers.

5. The method of any one of claims 1 to 4, wherein the autopilot device is equipped with M cameras, the M cameras including the first camera, the second camera, and a third camera, the third camera having a field of view that overlaps both the field of view of the first camera and the field of view of the second camera, M being an integer greater than 2; the method further comprises the following steps:

obtaining M first rotation matrixes, M second rotation matrixes, M first translation matrixes and M second translation matrixes; the M first rotation matrices are rotation matrices between the first camera and the second camera and at least two rotation matrices among the M first rotation matrices are different, the M second rotation matrices are rotation matrices between the second camera and the third camera and at least two rotation matrices among the M second rotation matrices are different, the M first translation matrices are translation matrices between the first camera and the second camera and at least two translation matrices among the M first translation matrices are different, the M second translation matrices are translation matrices between the second camera and the third camera and at least two translation matrices among the M second translation matrices are different, the M first rotation matrices are in one-to-one correspondence with the M first translation matrices, and the M second rotation matrices are in one-to-one correspondence with the M second translation matrices, m is an integer greater than 1;

solving the first equation group to obtain a third rotation matrix; the first equation set comprises M first equations, and the M first equations are in one-to-one correspondence with the M first rotation matrixes and in one-to-one correspondence with the M second rotation matrixes; in each first equation, the first rotation matrix and the second rotation matrix are known numbers, and the third rotation matrix is an unknown number; the third rotation matrix is a rotation matrix between the first camera and the third camera;

solving the second equation set to obtain a third translation matrix; the second equation set comprises M second equations, and the M second equations are in one-to-one correspondence with the M first rotation matrixes and in one-to-one correspondence with the M second rotation matrixes; in each second equation, the first rotation matrix, the second rotation matrix, the first translation matrix and the second translation matrix are known numbers, and the third translation matrix is an unknown number; the third translation matrix is a translation matrix between the first camera and the third camera;

setting a pose including the third rotation matrix and the third translation matrix as a relative pose between the first camera and the third camera.

6. The method of claim 1, wherein the first set of matching feature points comprises feature points extracted from at least two frames of images by the first camera and feature points extracted from at least two frames of images by the second camera; the determining, from the first set of matched feature points, a first relative pose between the first camera and the second camera comprises:

determining a relative pose between the first camera and the second camera according to the feature point pairs in the first matching feature point set to obtain a first intermediate pose;

substituting each characteristic point pair in the first matching characteristic point set into a first formula to obtain a residual error corresponding to each characteristic point pair in the first matching characteristic point set, wherein the first formula representation comprises the first intermediate pose;

rejecting interference characteristic point pairs in the first matching characteristic point set to obtain a second matching characteristic point set; the interference characteristic point pair is a characteristic point pair of which the residual error corresponding to the first matching characteristic point set is greater than a residual error threshold value;

determining a relative pose between the first camera and the second camera according to the feature point pairs in the second matching feature point set to obtain a second intermediate pose;

taking an object intermediate pose as the first relative pose between the first camera and the second camera; the target intermediate pose is a relative pose between the first camera and the second camera determined according to the feature point pairs in the target matching feature point set, and the number of the feature point pairs in the target matching feature point set is smaller than a quantity threshold or the ratio of the number of the feature point pairs in the target matching feature point set to the number of the feature point pairs in the first matching feature point set is smaller than a proportion threshold.

7. The method according to any one of claims 1 to 6, further comprising:

outputting reminding information under the condition that the difference value between the first relative pose and the second relative pose is not greater than the pose change threshold; the reminding information is used for reminding that the relative pose between the first camera and the second camera is abnormal.

8. An autopilot device, comprising:

the image acquisition unit is used for acquiring images through the first camera and the second camera respectively; the field of view of the first camera and the field of view of the second camera overlap, the first camera and the second camera being mounted at different locations on an autopilot device;

the image feature point extraction unit is used for carrying out feature point matching on the image acquired by the first camera and the image acquired by the second camera to obtain a first matched feature point set; the first matching feature point set comprises H groups of feature point pairs, each group of feature point pairs comprises two matched feature points, one feature point is a feature point extracted from the image acquired by the first camera, the other feature point is a feature point extracted from the image acquired by the second camera, and H is an integer not less than 8;

a pose calculation unit for determining a first relative pose between the first camera and the second camera according to the first set of matched feature points; the first matching feature point set comprises H groups of feature point pairs, each group of feature point pairs comprises two matching feature points, one feature point is a feature point extracted from an image acquired by the first camera, the other feature point is a feature point extracted from an image acquired by the second camera, the visual fields of the first camera and the second camera are overlapped, the first camera and the second camera are installed at different positions of the automatic driving device, and H is an integer not less than 8;

9. The apparatus of claim 8,

the pose calculation unit is specifically configured to determine an essential matrix between the first camera and the second camera; calculating a 5-degree-of-freedom relative pose between the first camera and the second camera according to a singular value decomposition result of the intrinsic matrix; the device further comprises:

a scale calculation unit for taking a ratio between the first distance and the second distance as a scale factor; the first distance and the second distance are respectively the result of measuring the same distance by a non-vision sensor and a vision sensor on the automatic driving device, and the vision sensor comprises the first camera and/or the second camera;

the pose calculation unit is further configured to combine the 5-degree-of-freedom relative pose and the scale factor to obtain the first relative pose.

10. The apparatus of claim 9,

the pose calculation unit is specifically configured to perform singular value decomposition on the intrinsic matrix to obtain a singular value decomposition result; obtaining at least two relative poses between the first camera and the second camera according to the singular value decomposition result; calculating the three-dimensional coordinate positions of the feature points in the first matching feature point set by using the at least two relative poses respectively; taking, as the 5-degree-of-freedom relative pose, one of the at least two relative poses at which the three-dimensional coordinate position of each feature point in the first set of matched feature points is located in front of the first camera and the second camera.

11. The apparatus of claim 8,

the pose calculation unit is specifically used for iteratively solving an objective equation to obtain the first relative pose; in the object equation, the first relative pose includes parameters that are unknowns, and the pairs of feature points in the first matching feature point set, the internal parameters of the first camera, and the internal parameters of the second camera are known numbers.

12. The device of any one of claims 8 to 11, wherein the autopilot device is equipped with M cameras, the M cameras including the first camera, the second camera, and a third camera, the third camera having a field of view that overlaps both the field of view of the first camera and the field of view of the second camera, M being an integer greater than 2; the device further comprises:

the closed-loop optimization unit is used for obtaining M first rotation matrixes, M second rotation matrixes, M first translation matrixes and M second translation matrixes; the M first rotation matrices are rotation matrices between the first camera and the second camera and at least two rotation matrices among the M first rotation matrices are different, the M second rotation matrices are rotation matrices between the second camera and the third camera and at least two rotation matrices among the M second rotation matrices are different, the M first translation matrices are translation matrices between the first camera and the second camera and at least two translation matrices among the M first translation matrices are different, the M second translation matrices are translation matrices between the second camera and the third camera and at least two translation matrices among the M second translation matrices are different, the M first rotation matrices are in one-to-one correspondence with the M first translation matrices, and the M second rotation matrices are in one-to-one correspondence with the M second translation matrices, m is an integer greater than 1;

13. The apparatus of claim 8, wherein the first set of matching feature points comprises feature points extracted from at least two frames of images by the first camera and feature points extracted from at least two frames of images by the second camera;

the pose calculation unit is specifically configured to determine, according to the feature point pairs in the first matching feature point set, a relative pose between the first camera and the second camera to obtain a first intermediate pose;

14. The apparatus of any one of claims 8 to 13, further comprising:

the reminding unit is used for outputting reminding information under the condition that the difference value between the first relative pose and the second relative pose is not larger than the pose change threshold; the reminding information is used for reminding that the relative pose between the first camera and the second camera is abnormal.

15. An automobile, comprising:

a memory for storing a program;

a processor for executing the program stored by the memory, the processor being configured to perform the method of any of claims 1 to 7 when the program is executed.

16. A computer-readable storage medium, characterized in that the computer storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1 to 7.