CN117876502A

CN117876502A - Depth calibration method, depth calibration equipment and depth calibration system

Info

Publication number: CN117876502A
Application number: CN202410267894.4A
Authority: CN
Inventors: 昌钟璨; 冯晓刚
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2024-03-08
Filing date: 2024-03-08
Publication date: 2024-04-12
Anticipated expiration: 2044-03-08
Also published as: CN117876502B

Abstract

The application provides a depth calibration method, depth calibration equipment and a depth calibration system, and relates to the technical field of terminals. The moving component of the depth calibration system is used for driving the ToF module to move and rotate. When the ToF module is at a first distance from the calibration plate, the first depth map and the second depth map are acquired respectively in the first space position and the second space position, and then the ToF module is moved to a second distance to acquire the third depth map and the fourth depth map respectively in the third space position and the fourth space position. And the controller performs depth calibration on the ToF module according to the four depth maps and the first distance and the second distance to complete the internal reference calibration of the ToF module. The depth calibration can also acquire 2 depth maps again to calculate the laser offset distance, and laser offset correction is performed before the internal parameter calibration. Under the condition that three calibration operations are not required to be independently executed, laser offset correction, internal reference matrix calibration and offset calibration can be completed by collecting six depth maps, and the calibration capacity and labor cost are greatly improved.

Description

Depth calibration method, depth calibration equipment and depth calibration system

Technical Field

The embodiment of the application relates to the technical field of terminals, in particular to a depth calibration method, depth calibration equipment and a depth calibration system.

Background

Time-of-Flight (ToF) is a technique that uses Time-of-Flight to measure distance. The ToF may be applied to a ranging device, such as a depth camera. The ToF module may cause problems such as laser offset and lens offset due to lens assembly and edge distortion, and further cause errors between the obtained depth of field distance and the actual distance. The ToF module needs calibration correction to minimize measurement errors.

The existing calibration scheme is as follows: laser Offset correction (Laser Offset), camera correction (Camera Calibration), and Offset correction (Offset Calibration) are performed in this order. These three calibration operations increase equipment costs, labor costs, and limit the overall throughput (UPH). The existing calibration scheme comprises three calibration steps which are sequentially executed, is relatively time-consuming and affects the accuracy of the calibration steps mutually, so that the accuracy of the depth calibration is relatively low. The three calibration steps need to be prepared with different calibration plates, the calibration consumables are more, and the calibration steps are more complicated.

Disclosure of Invention

The embodiment of the application provides a depth calibration method, depth calibration equipment and a depth calibration system, which are used for solving the technical problems of low accuracy and complicated steps of the existing depth camera calibration scheme.

In order to achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

in a first aspect, a depth calibration method is provided for calibrating a ToF module by a depth calibration system, where the ToF module may be a depth acquisition module in a depth camera of an electronic device. The depth calibration system comprises a moving assembly, a calibration plate and a controller, wherein the moving assembly is used for fixing the ToF module and driving the ToF module to move. After the ToF module is fixed on the moving assembly, the laser emitting surface of the ToF module is parallel to the calibration plate, where the parallelism may be absolute parallelism or approximate parallelism. In actual assembly, the laser emission surface of the ToF module is approximately parallel to the laser reflection surface of the calibration plate, which can be interpreted as a small included angle between the two planes.

The depth calibration method provided by the application can be mainly controlled by the controller, the moving assembly is controlled to drive the ToF module to move, the depth map collected by the ToF module and related measurement data are obtained for calculation, and the depth calibration of the ToF module is realized.

The controller can control the moving assembly to drive the ToF module to move along a first direction, so that a first distance is reserved between the laser emitting surface of the ToF module and the laser reflecting surface of the calibration plate. The laser emission surface of the ToF module is parallel to the calibration plate, and the first direction is perpendicular to the laser reflection surface of the calibration plate and the laser emission surface of the ToF module. For example, in the three-dimensional reference frame, the laser emitting surface of the ToF module and the calibration plate may be parallel to the XY plane, and the first direction may be interpreted as being parallel to the Z axis.

The controller controls the moving assembly to drive the ToF module, the center point of the laser emission surface surrounding the ToF module is parallel to the calibration plate to rotate, that is to say, the moving assembly drives the ToF module to rotate in the XY plane, and the Y-axis coordinate of the center point of the laser emission surface of the ToF module is kept unchanged in the rotating process. The controller controls the moving assembly to drive the ToF module to rotate around the center point on the XY plane, so that the ToF module is respectively located in a first space pose and a second space pose, and the first space pose and the second space pose are different. For example, when the controller controls the moving assembly to drive the ToF module to rotate so as to be in the first space position, the moving assembly can drive the ToF module to rotate 180 ° in the XY plane, so that the space position where the ToF module is located is changed from the first space position to the second space position.

Under the condition that a laser emission surface of the ToF module is separated from the calibration plate by a first distance, the controller controls the ToF module to be in a first space pose, emits laser to the calibration plate, and shoots the calibration plate to obtain a first depth map; and when the controller controls the ToF module to be in the second space pose, laser is emitted to the calibration plate, and the calibration plate is shot to obtain a second depth map. In this way, the ToF module acquires a first depth map and a second depth map at a measuring point at a first distance from the calibration plate by two different spatial poses respectively.

The controller controls the moving assembly to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is separated from the calibration plate by a second distance, and the first distance is different from the second distance. The controller controls the moving assembly to drive the ToF module to rotate around the center point of the laser emitting surface of the ToF module and parallel to the calibration plate, so that the ToF module is respectively in a third space position and a fourth space position, and the third space position and the fourth space position are different.

Under the condition that the laser emission surface of the ToF module is separated from the calibration plate by a second distance, the controller controls the ToF module to shoot the calibration plate at a third space pose to obtain a third depth image, and shoots the calibration plate at a fourth space pose to obtain a fourth depth image.

And finally, the controller performs depth calibration on the ToF module according to the first depth map, the second depth map, the third depth map, the fourth depth map, the first distance and the second distance. For example, the controller may obtain the calibration point light path distance expression formula and the angle expression formula corresponding to each pixel point according to the multiple depth maps, the first distance and the second distance, and then convert the light path distance expression formula and the angle expression formula according to the angle correspondence relation or the light path distance correspondence relation existing between the different depth maps due to the equal distances or the space pose, so as to calculate the relative displacement of each pixel point of the ToF module relative to the central pixel point, and further obtain an internal reference matrix of the ToF module, and then perform depth calibration, especially internal reference calibration and offset calibration, on the ToF module by using the internal reference matrix.

According to the depth calibration scheme, three sets of calibration systems are not required to be independently distributed and three calibration operations are sequentially executed, so that the calibration cost, the site space requirement degree and the complexity of the calibration operations are greatly reduced. The internal reference matrix calibration and the offset calibration in the depth calibration task can be completed by collecting four depth maps, and the calibration productivity and the labor cost are greatly improved. According to the depth calibration method, the ToF module to be calibrated is required to be basically parallel to the calibration plate, angle measurement and calculation are achieved through laser, and the environment construction difficulty and the spot inspection difficulty are reduced. The limitation of the accurate distance parameter of the grid calibration plate to the resolution of the ToF module in the Zhang Zhengyou calibration method is effectively avoided, and the calibration can be completed in a faster mode while the accuracy is ensured.

In a possible implementation manner of the first aspect, the first space position and the third space position are the same, and the second space position and the fourth space position are the same. The first space pose and the second space pose are different space poses when the depth map is acquired under the condition that the laser emitting surface of the ToF module is separated from the calibration plate by a first distance, and the third space pose and the fourth space pose are different space poses when the depth map is acquired under the condition that the laser emitting surface of the ToF module is separated from the calibration plate by a second distance. And in the two collection operations, the first distance and the second distance between the ToF module and the calibration plate are different.

That is, the first spatial pose and the third spatial pose are the same spatial pose acquired under the condition that the laser emission surface of the ToF module is spaced from the calibration plate by different distances; the second space pose and the fourth space pose are the same space pose acquired under the condition that the laser emission surface of the ToF module and the calibration plate are separated by different distances. Therefore, errors caused by different space poses can be effectively reduced, and the accuracy of depth calibration of the ToF module is further improved.

In a possible implementation manner of the first aspect, a solution of switching the space pose by rotation of the ToF module is further defined. Specifically, the laser emitting surface of the ToF module has a center point, and when the ToF module rotates, the ToF module rotates around the center point and is parallel to the calibration plate. In other words, the center point on the laser emitting surface of the calibration plate and the center point of the laser emitting surface of the ToF module are set to be at the same height, and the ToF module is controlled to rotate around the center axis by taking the connecting line between the two center points as the center axis. In the rotating process of the ToF module, the laser emission surface of the ToF module is always parallel to the laser emission surface of the calibration plate.

Under the condition that the laser emission surface of the ToF module is spaced from the calibration plate by a first distance, when the ToF module is in a first space pose, the moving assembly drives the ToF module to rotate 180 degrees on the premise that the ToF module is parallel to the laser emission surface of the calibration plate, and the ToF module is in a second space pose. Under the condition that the ToF module rotates 180 degrees, the pixel points of the ToF module also rotate 180 degrees along with the ToF module, all the pixel points of the ToF module are in an axisymmetric state, and the angle corresponding relation and the light path corresponding relation can be obtained according to the corresponding relation among part of the pixel points in the axisymmetric state.

Based on the same principle, when the ToF module is in the third space pose under the condition that the laser emission surface of the ToF module is separated from the calibration plate by a second distance, the moving assembly drives the ToF module to rotate 180 degrees on the premise of being parallel to the laser emission surface of the calibration plate, and the ToF module is in the fourth space pose. Under the condition that the ToF module rotates 180 degrees, the pixel points of the ToF module also rotate 180 degrees along with the ToF module, all the pixel points of the ToF module are in an axisymmetric state, and the angle corresponding relation and the light path corresponding relation can be obtained according to the corresponding relation among part of the pixel points in the axisymmetric state.

In a possible implementation manner of the first aspect, a parallel condition between the laser emitting surface of the ToF module and the calibration plate in the depth calibration process is further defined. Specifically, the laser emitting surface of the ToF module and the laser reflecting surface of the calibration plate may be in an approximately parallel state.

Here, the laser emission surface of the ToF module is approximately parallel to the calibration plate, and the specific implementation scheme may be: the included angle between the laser emission surface of the ToF module and the calibration plate is smaller than a preset angle threshold. The preset angle threshold may be set to an angle range of 1 deg. -5 deg., for example, the preset angle threshold may be 5 deg.. Therefore, the accuracy requirement on the relative parallelism of the calibration plate and the ToF module in the construction process of the depth calibration system can be reduced, and the depth calibration operation is simplified.

In a possible implementation manner of the first aspect, a scheme of laser offset correction is added before performing internal reference calibration on the ToF module.

The laser offset correction process of the depth calibration system may specifically include: the controller controls the moving assembly to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is separated from the calibration plate by a third distance. The third distance may be the same as or different from the first distance or the second distance involved in the internal reference calibration process.

The controller controls the moving assembly to drive the ToF module to rotate parallel to the calibration plate, so that the ToF module is respectively in a fifth space pose and a sixth space pose, and the fifth space pose and the sixth space pose are different space poses. For example, the ToF module rotates 180 ° in the fifth spatial pose, switching to the sixth spatial pose. In the process of driving the ToF module to rotate, the movable assembly also needs to surround the center point of the laser emitting surface of the ToF module and always rotates parallel to the calibration plate. The fifth spatial pose of the ToF module when performing the laser offset correction phase may be the same as the first spatial pose of the ToF module when performing the reference calibration phase, and the sixth spatial pose of the ToF module when performing the laser offset correction phase may be the same as the second spatial pose of the ToF module when performing the reference calibration phase. Therefore, errors caused by different space positions in the depth calibration process can be reduced as much as possible, and the accuracy of the depth calibration of the ToF module is improved.

And when the ToF module is in the sixth space pose, the calibration plate is shot to obtain a sixth depth map.

And then, the controller corrects the laser offset of the ToF module according to the fifth depth map and the sixth depth map. The specific implementation scheme can be various, for example, the controller can compare and find the central pixel point through the depth values of all the pixel points of two depth maps of the same calibration plate acquired when the ToF module is in different space positions. Because the central pixel point of the ToF module and the central calibration point of the calibration plate are at the same height, the depth value acquired by the central pixel point is relatively small in error. The controller can correct the laser offset of the ToF module according to the related data of the central pixel point.

According to the depth calibration method, two depth maps of the ToF module in different spatial poses are acquired at one measuring point, and laser offset correction is performed on the ToF module. The method is simple to operate, the accuracy of laser offset correction can be improved, meanwhile, the influence of laser offset on subsequent internal reference calibration can be reduced, and the overall accuracy of ToF module depth calibration is improved.

In a possible implementation manner of the first aspect, the controller is specifically limited to a solution for performing laser offset correction on the ToF module according to the fifth depth map and the sixth depth map. The ToF module collects a fifth space pose where the fifth depth map is located, and is switched to a sixth space pose after rotating 180 degrees, namely the ToF module collects the space pose where the sixth depth map is located. Then the center pixel point of the ToF module collects depth values of the center calibration point of the calibration plate before and after 180 degrees of rotation. On the premise that the calibration plate is unchanged, the depth value acquired by the central pixel point of the ToF module is unchanged.

And the controller searches the pixel points with unchanged depth values relative to the sixth depth map according to the depth values of all the pixel points in the fifth depth map and the sixth depth map, namely the central pixel point. The controller acquires a laser recording histogram of the central pixel, and the laser recording histogram records waveform diagrams of a plurality of groups of laser signals emitted by the laser generators corresponding to the central pixel. The controller calculates the laser offset distance according to the difference between the target time of the waveform diagram of the first laser signal recorded in the laser recording histogram and the statistical 0 point time of the laser recording histogram. To improve the correction accuracy, the peak moment of the waveform map of the first laser signal may be selected to count the difference from the count 0 point of the laser recording histogram. And the controller corrects the laser offset of the ToF module according to the laser offset distance.

The depth calibration system firstly carries out laser offset correction on the ToF module, and then the ToF module after laser offset correction is used for collecting the first depth image to the fourth depth image so as to realize internal reference calibration, and meanwhile, the accuracy of an internal reference calibration stage is improved.

In a possible implementation manner of the first aspect, a solution of performing depth calibration on the ToF module by the controller is further defined. Specifically, the controller obtains a calibration point light path distance expression formula and an angle expression formula corresponding to each pixel point of the ToF module according to the first depth map, the second depth map, the third depth map and the fourth depth map, converts the light path distance expression formula and the angle expression formula according to the angle correspondence or the light path distance correspondence existing between the equal distances or the space pose between the different depth maps so as to calculate the relative displacement of each pixel point of the ToF module relative to the central pixel point, further obtains an internal reference matrix of the ToF module, and performs depth calibration, particularly internal reference calibration and offset calibration, on the ToF module by using the internal reference matrix.

In a possible implementation manner of the first aspect, the solution that the controller obtains the relative displacement between each pixel point of the ToF module and the central pixel point, and the included angle between the corresponding optical path of each pixel point and the first direction is further defined.

Under the condition that a first distance is formed between the laser emitting surface of the ToF module and the calibration plate, the calibration plate comprises a first calibration point, a second calibration point and a third calibration point, the first calibration point corresponds to a central pixel point of the first depth image and the second depth image, and the second calibration point and the third calibration point are two axisymmetric calibration points. The first light path between the first calibration point and the first measurement point forms a first included angle with the second light path between the second calibration point and the first measurement point, and the first light path between the first calibration point and the first measurement point forms a second included angle with the third light path between the third calibration point and the first measurement point. The controller determines that the first included angle is equal to the second included angle according to the first depth map acquired before rotation and the second depth map acquired after 180 degrees of rotation. And calculating a plurality of angle conversion formulas when the first included angle is equal to the second included angle according to the relational expression of the first included angle and the second included angle, and recording the calculated angle conversion formulas as the first angle conversion formulas.

Similarly, under the condition that the laser emission surface of the ToF module is separated from the calibration plate by a second distance, the calibration plate comprises a fourth calibration point, a fifth calibration point and a sixth calibration point, the fourth calibration point corresponds to the central pixel point of the third depth map and the fourth depth map, and the fifth calibration point and the sixth calibration point are two axisymmetric calibration points. A third included angle is formed between a fourth light path between the fourth calibration point and the second measurement point and a fifth light path between the fifth calibration point and the second measurement point, and a fourth included angle is formed between a fourth light path between the fourth calibration point and the second measurement point and a sixth light path between the sixth calibration point and the second measurement point.

The controller calculates the relative displacement of each pixel point and the central pixel point of the ToF module and the included angle of the corresponding light path of each pixel point and the first direction according to the depth values corresponding to the first light path, the second light path, the third light path, the fourth light path, the fifth light path and the sixth light path, the first angle conversion formula and the second angle conversion formula. The controller calculates a second angle conversion formula when the third included angle is equal to the fourth included angle according to the third depth map and the fourth depth map; wherein, because the ToF module collects the space pose of the third depth map and the space pose of the fourth depth map by 180 degrees, the third included angle is equal to the fourth included angle. And calculating a plurality of angle conversion formulas when the third included angle is equal to the fourth included angle according to the relational expression of the third included angle and the fourth included angle, and recording the calculated angle conversion formulas as a second angle conversion formula. And carrying the first angle conversion formula and the second angle conversion formula into a calculation internal reference matrix to finish depth calibration.

According to the depth calibration method provided by the application, three calibration stages of depth calibration can be completed by collecting six depth maps, so that the site and the whole flow are simplified, and the accuracy of the depth calibration is improved.

In a second aspect, the present application provides a depth calibration system for calibrating a ToF module. The depth calibration system comprises a moving assembly, a calibration plate and a controller.

The controller is configured to perform the depth calibration method of any one of the first aspects.

In a third aspect, the present application provides a depth calibration apparatus for calibrating a ToF module. The depth calibration equipment is connected with a moving assembly, and the ToF module is fixed behind the moving assembly, and the laser emission surface of the ToF module is parallel to the calibration plate.

The depth calibration device comprises a memory and a processor, wherein the memory is coupled with the processor; the memory stores computer-executable instructions; the processor executes computer-executable instructions stored in the memory to cause the depth calibration apparatus to perform the depth calibration method performed by the controller in any one of the first aspects.

In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored therein, which when run on a computer causes the computer to perform the depth calibration method as in any one of the first aspects.

In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the depth calibration method according to any one of the first aspects.

The technical effects of any one of the design manners of the second aspect to the fifth aspect may be referred to the technical effects of the different design manners of the first aspect, and will not be repeated here.

Drawings

Fig. 1 is a schematic diagram of depth data acquisition by a ToF module;

FIG. 2 is a schematic diagram of offset laser correction involved in depth calibration of a ToF module;

FIG. 3 is a schematic diagram of a grid calibration plate involved in depth calibration of a ToF module;

FIG. 4 is a schematic diagram of the laser emitted to the grid calibration plate involved in the depth calibration of the ToF module;

FIG. 5 is a schematic diagram of laser bias involved in depth calibration of a ToF module;

FIG. 6 is a schematic flow chart of a depth calibration method according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of a calibration principle of the depth calibration system according to the embodiment of the present application;

FIG. 8 is a depth map representation of a depth calibration method according to an embodiment of the present application;

FIG. 9 is a schematic diagram of a laser recording histogram related to a depth calibration method according to an embodiment of the present application;

FIG. 10 is a schematic flow chart of internal reference calibration related to the depth calibration method according to the embodiment of the present application;

FIG. 11 is a schematic diagram of a new plane fitting during an internal reference calibration process according to the depth calibration method provided in the embodiment of the present application;

fig. 12 is a schematic diagram of coordinate transformation related to a depth calibration method according to an embodiment of the present application.

Detailed Description

Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

For ease of understanding, a part of the technical common sense related to the embodiments of the present application is first described.

Time-of-Flight (ToF) is a technique that uses Time-of-Flight to measure distance. Among them, dtofi, direct Time-of-Flight, is a technique for measuring distance by directly measuring Time of Flight.

The distance measurement principle of the TOF is that the TOF module emits laser to the target object at a detection point, receives the laser reflected from the surface of the target object, and calculates the distance between the detection point and the surface of the target object by detecting the round-trip flight time of the laser and combining the laser flight speed.

ToF may be applied to ranging devices (e.g., depth cameras). The depth camera acquires depth of field distances between the depth camera and each point in the view range by using a ToF technology. The depth camera can obtain the three-dimensional space coordinates of each position point in the world coordinate system by using the depth of field distance of each position point in the view finding range and the two-dimensional coordinates of each position point in the two-dimensional image.

The depth camera is equipped with a ToF module, as shown in fig. 1, which can perform laser ranging on a target object in a field of view. The ToF module mainly comprises a laser emitter, a laser receiver, a lens component and a processor. The laser transmitter is used for transmitting laser, the laser receiver is used for receiving the laser, and the lens assembly is used for collecting images in a view finding range. The processor is coupled to the laser transmitter, the laser receiver, and the lens assembly.

The processor can acquire a first moment when the laser transmitter transmits laser and a second moment when the laser receiver receives corresponding laser, calculate laser flight time and further calculate a corresponding depth of field distance. And the processor calculates depth information of all the pixel points on the image according to the depth of field distance of each pixel point in the image acquired by the lens component. It should be noted that, the laser emitted and received by the ToF module may be a laser pulse.

As shown in fig. 1, the target may be a three-dimensional object. There are two points D1 and D2 on the surface of the object, D1 and D2 being different distances from the ToF module. The laser transmitter transmits laser S1 to D1, and receives the laser S1', D1 and the depth of field distance of the ToF module imaging plane is D1. The laser transmitter transmits laser S2 to D2, and receives the laser S2', and the depth of field distance between the D2 and the imaging plane of the ToF module is D2.

The depth camera may be applied to image processing such as image blurring processing, virtual Reality (VR), augmented Reality (Augmented Reality, AR), or the like, or to display a scene.

The ToF module may cause problems such as laser offset and lens offset due to lens assembly and edge distortion, and further cause errors between the obtained depth of field distance and the actual distance. The ToF module needs calibration correction to minimize measurement errors.

The existing calibration scheme is as follows: laser Offset correction (Laser Offset), camera correction (Camera Calibration), and Offset correction (Offset Calibration) are performed in this order. These three calibration operations increase equipment costs, labor costs, and limit the overall throughput (UPH).

The first calibration operation is laser offset correction.

When the ToF calibration is carried out, the ToF module emits laser to the calibration plate. In the laser offset correction stage, the required calibration plate is a white plate with higher reflectivity. The ToF module records the time that the laser is reflected back to the ToF module after being emitted from the emission to the calibration plate, i.e. the laser flight time.

The ToF module typically records the time of emission and reception of the laser using a histogram, which can record a range that covers a small amount of the laser's time of flight. As shown in fig. 2, a schematic diagram of laser time of flight recorded in a histogram is shown. The laser is firstly lightened, the histogram records the flight time of the firstly lightened laser, and the initial moment of the flight time of the firstly lightened laser corresponds to the position of the actual position on the histogram to be 0 point; however, because there is a laser offset, the position where the laser first lights up and sits on the histogram is counted as position a shown in fig. 2, which is a distance from the actual 0 point (shown as position b shown in fig. 2). The period from the actual 0 point to the end point of the histogram (shown as c in fig. 2) shortens, resulting in a reduction in the number of complete lasers on the histogram that this shortened period can actually cover.

The purpose of the laser offset correction is to measure the offset distance between the actual 0 point location and the statistical 0 point location (as indicated by L in fig. 2), and compensate and calibrate the laser offset by the offset distance. Thus, after the laser offset correction, the actual 0 point position (shown as d in fig. 2) where the laser is first lit will be relatively close to the 0 point position of the histogram, so that the histogram can cover as much of the flight period of the complete laser as possible.

The second calibration operation is camera correction calibration, and the third calibration operation is offset correction calibration.

The camera calibration is also called camera internal reference calibration, and a Zhang Zhengyou calibration method and a calibration method for identifying spatial information by image information related to the Zhang Zhengyou calibration method are adopted for common camera internal reference calibration. Taking Zhang Zhengyou calibration method as an example, the adopted calibration plate is a checkerboard calibration plate as shown in fig. 3, and a checkerboard image is drawn on the calibration plate. The grid points on the checkerboard calibration plate can be dots or square points, and are not limited. As shown in fig. 4, the ToF module photographs the checkerboard calibration plate from multiple angles, and the algorithm identifies the corner centers of the checkerboard image for calculating spatial information. And then, according to the principle that the depth of field of all points on the checkerboard calibration plate is consistent, the edge distortion of the camera is corrected by fitting through a least square method, so that an internal reference matrix matched with all pixel areas of the ToF module is obtained.

dtofs are ranging methods that convert to distance by directly measuring time of flight. For the multi-point laser, the optical path is obviously longer than the z-direction distance due to the larger angle of view, and the included angle of the optical path needs to be calculated when the z-direction distance is converted.

As shown in fig. 5, a checkerboard calibration plate is disposed opposite to the ToF module. The checkerboard calibration plate is provided with a target point A1 and a target point A2, and the actual depth of field distance d1 between the target point A1 and the plane of the ToF module is equal to the actual depth of field distance d2 between the target point A2 and the plane of the ToF module. In actual measurement, the optical path of the laser actually emitted by the ToF module to the target point A1 is OA1, and the distance of the corresponding optical path is equal to the actual depth of field distance d1. Due to factors such as edge distortion, oblique laser emission and the like, an optical path of the laser actually emitted by the ToF module to the target point A2 is OA2, and the corresponding optical path distance d2' is not equal to the actual depth of field distance d2.

In order to calibrate the camera, an included angle θ between the optical path OA2 and the optical path OA1 needs to be calculated, and the actual depth of field distance d1 can be calculated through the actual optical path distance d2' and the included angle θ.

The geometrical relationship of the included angle theta is as follows: cosθ=d1/d 2, sinθ=d3/d 2. Knowing d1, d2 and d3, the angle θ can be calculated. During actual measurement, any target point Ai in the camera visual field range, the distance di' of the optical path OAi emitted by the camera to the target point Ai can be calculated through laser flight time and light speed, and then the actual depth of field distance di can be calculated according to the included angle thetai between the optical path OAi and the z direction and used as the actual depth of field distance of the target point Ai measured by the camera.

Due to the resolution of the ToF module, the accurate distance between the target points on the grid plane (d 3 shown in fig. 5) cannot be accurately measured, which results in lower accuracy of the calculated included angle, that is, lower accuracy of the reference of the calibrated depth camera, and further results in lower accuracy of the actual depth-of-field distance of each target point calculated during actual measurement. In addition, the existing camera calibration scheme comprises three calibration steps which are sequentially executed, so that the time consumption is high, and the accuracy between the calibration steps is mutually influenced. The three calibration steps need to be prepared with different calibration plates, the calibration consumables are more, and the calibration steps are more complicated.

Based on this, the application provides a depth calibration method for performing depth calibration on a ToF module, where the ToF module to be calibrated may be a ToF module in a depth camera. That is, the depth calibration method provided in the embodiment of the present application may perform depth calibration for a depth camera of an electronic device, where the electronic device may be a personal computer (Personal Computer, PC), a tablet computer, a notebook computer, a portable computer (such as a mobile phone), a wearable electronic device (such as a smart watch), an augmented Reality (Augmented Reality, AR) \virtual Reality (VR) device, a vehicle-mounted computer, or other electronic devices with a depth camera, and the following embodiments do not limit the specific form of the electronic device. The ToF module may mainly include a laser transmitter and a laser receiver.

Fig. 6 is a schematic flow chart of the depth calibration method according to the present embodiment. The depth calibration method provided by the application is applied to a depth calibration system, and is shown in fig. 7, which is an assembly schematic diagram of the depth calibration system, and fig. 7 also shows a ToF module for calibration of the depth calibration system. The depth calibration system comprises: a movement assembly, a calibration plate, a ranging assembly (not shown in fig. 7), and a controller (not shown in fig. 7).

The moving component of the depth calibration system is used for moving the ToF module. During depth calibration, the ToF module may have both rotational and translational modes of movement. For realizing rotation and translation of the ToF module, the moving assembly may include a motor and a clamping member, the clamping member is used for clamping the ToF module, and the motor is used for driving the clamping member to drive the ToF module to rotate and translate. The translation of the ToF module may refer to moving the ToF module toward the calibration plate along the first direction, and the rotation of the ToF module may refer to rotating the ToF module in a plane perpendicular to the first direction, or rotating the ToF module in a plane parallel to the calibration plate.

In particular, the motor may include a drive motor and a shaft, the drive motor being configured to drive rotation and translation of the shaft. The clamping piece can include fixed part and clamping jaw, and the fixed part is used for fixing the clamping jaw to the pivot of motor, clamping jaw centre gripping ToF module, and the pivot of motor drives the ToF module through fixed part and clamping jaw.

In order to describe the relative position relation of each device during assembly of the depth calibration system, a three-dimensional reference coordinate system is defined. As shown in fig. 7, the three-dimensional reference coordinate system includes an X-axis, a Y-axis, and a Z-axis, and the three coordinate axes are perpendicular to each other. The XZ plane is basically parallel to the ground or the plane of an operation table of a calibration environment where the depth calibration system is located, and the moving assembly can drive the ToF module to rotate in the XY plane or drive the ToF module to translate along the Z axis. In the embodiment of the application, in the process that the moving assembly drives the ToF module to rotate in the XY plane, the coordinates of each point on the laser emitting surface of the ToF module on the X axis and the Y axis of the three-dimensional reference coordinate system change, and the coordinates of each point on the Z axis are kept unchanged. In the process that the moving assembly drives the ToF module to translate along the Z axis, the coordinates of each point on the laser emission surface of the ToF module on the Z axis of the three-dimensional reference coordinate system are changed, and the coordinates of each point on the X axis and the Y axis are kept unchanged.

The calibration plate of the depth calibration system is used for receiving and reflecting the laser emitted by the ToF module. The ToF module includes a laser emitting surface, and the laser emitter and the laser receiver of the ToF module are disposed on the laser emitting surface, and a plane where the laser emitting surface of the ToF module is located is denoted as a first plane (P1 shown in fig. 7). The calibration plate can be a flat plate with higher surface reflectivity, and a checkerboard is not required to be arranged on the calibration plate.

The calibration plate is provided with a reflecting surface, and the reflecting surface of the calibration plate is used for receiving laser emitted by the laser emitter of the ToF module and reflecting the emitted light to the laser receiver of the ToF module. The plane in which the reflecting surface of the calibration plate is located is denoted as a second plane (P2 as shown in fig. 7).

When the calibration is performed, the first plane where the laser emitting surface of the ToF module is located and the second plane where the laser reflecting surface of the calibration plate is located are basically parallel, and a certain distance exists between the first plane and the second plane. And, for ease of description, the first plane and the second plane may also be made substantially parallel to the XY plane. It should be noted that, the two planes are substantially parallel or approximately parallel, which is to be understood that the included angle between the two planes is less than or equal to a preset included angle threshold, and the included angle threshold may be 0 ° -5 °, which is not limited. In practice, due to factors such as measurement and the thickness of the ToF module, absolute parallelism may not be easily achieved between planes, and subsequent calibration operations may be performed with substantially parallel planes having a relatively small inclination. The following explanations regarding parallelism are referred to herein for a substantially parallel explanation, and are not repeated.

The distance measuring component of the depth calibration system is used for measuring relevant distance data in the depth calibration process, such as distance data between two points on the laser reflecting surface of the calibration plate. The ranging assembly may include devices such as a laser range radar, a measuring ruler, and the like.

The controller of the depth calibration system is used for executing a depth calibration method, for example, the moving assembly is controlled to drive the ToF module to move, and relevant calculation is carried out according to relevant data so as to realize the depth calibration of the ToF module. The controller can be connected with the control end and the data transmission end of the laser transmitter, the laser receiver, the moving component, the ranging component and the like of the ToF module so as to realize instruction interaction and data transmission.

Before the depth calibration system performs the depth calibration, the moving assembly can be placed on an operation platform, and the plane of the operation platform is parallel to the XZ plane of the three-dimensional reference coordinate system. The clamping jaw of the moving assembly clamps the ToF module, so that the ToF module is basically parallel to the XY plane, and the motor rotating shaft of the moving assembly can drive the ToF module to rotate in the XY plane or translate along the Z axis through the clamping jaw. In the rotation or translation process of the ToF module, the laser emission surface of the ToF module is kept parallel to the XY plane.

The calibration plate is fixed on the operation platform, a certain Z-direction distance exists between the laser reflecting surface of the calibration plate and the laser emitting surface of the ToF module, and the second plane where the laser reflecting surface of the calibration plate is located is basically parallel to the first plane where the laser emitting plate of the ToF module is located. The rotating shaft of the motor can drive the ToF module to translate along the Z axis through the clamping jaw, and the ToF module is close to or far away from the calibration plate so as to adjust the Z-direction distance between the laser emitting surface of the ToF module and the laser reflecting surface of the calibration plate.

In addition, during assembly, the center height of the ToF module and the center height of the calibration plate can be controlled to be consistent as much as possible, so that errors caused by inconsistent center heights are reduced, and the depth calibration precision is improved as much as possible. The center height of the ToF module may refer to the Y-axis coordinate of the center point of the laser emitting surface of the ToF module in the XYZ coordinate system, and the center height of the calibration plate may refer to the Y-axis coordinate of the center point of the laser reflecting surface of the calibration plate in the XYZ coordinate system.

Specifically, as shown in fig. 6, from the perspective of the controller in the depth calibration system, the provided depth calibration method mainly includes the following steps:

s601: the controller controls the moving assembly to drive the ToF module to move along a first direction, so that the laser emission surface of the ToF module is separated from the calibration plate by a first distance; the control mobile assembly drives the ToF module to rotate around the center point of the laser emission surface of the ToF module and parallel to the calibration plate, so that the ToF module is respectively in a first space position and a second space position, the ToF module is controlled to shoot the calibration plate in the first space position to obtain a first depth map, and the calibration plate is controlled to shoot in the second space position to obtain a second depth map.

As shown in fig. 7, the moving assembly drives the ToF module to move to the first measuring point T1, the laser emitting surface of the ToF module is a first plane P1, the laser reflecting surface of the ToF module is a second plane P2, and the Z-direction distance between the laser emitting surface of the ToF module and the laser reflecting surface of the calibration plate is a first distance d1.

At the first measurement point T1, the ToF module may record the current pose in the XY plane as a first spatial pose. And rotating the ToF module in the XY plane by a certain angle, and marking the rotated pose as a second space pose. The first spatial pose is different from the second spatial pose.

Or the controller can control the moving component to drive the ToF module to rotate, so that the ToF module is in the first space pose, and the first depth map is acquired. And then rotating the ToF module to enable the ToF module to be in a second space pose, and collecting a second depth map.

The moving assembly can control the ToF module to rotate around the center point of the laser emitting surface of the ToF module in the process of driving the ToF module to rotate in the XY plane. Thus, the Y-axis height of the center point of the laser emitting surface of the ToF module is always consistent with the Y-axis height of the center point of the laser reflecting surface of the calibration plate. The Z-direction distance between the center point of the laser emitting surface of the ToF module and the center point of the laser reflecting surface of the calibration plate is always consistent, and the depth of field distance between the center calibration point of the calibration plate acquired by the center pixel point of the ToF module is always consistent. And reversely deducing that in the process of rotating the ToF module parallel to the calibration plate, if the depth value acquired by a certain pixel point is always unchanged in all pixel areas of the ToF module, the pixel point is a central pixel point, and the depth of field distance acquired by the central pixel point is the depth of field distance of the central calibration plate of the calibration plate.

Based on the above, at the first measurement point T1, the ToF module first emits laser to the calibration plate in the first spatial pose, and collects a first depth map corresponding to the calibration plate. And the ToF module rotates to a second space pose, emits laser to the calibration plate again, and collects a second depth map corresponding to the calibration plate.

The ToF module generally has a standard placement posture, when the ToF module is in the standard placement posture, the lower surface of the ToF module is parallel to the XZ plane, the laser emission surface of the ToF module is parallel to the XY plane, and the laser emission direction of the laser emission surface is parallel to the Z axis. The term parallel is also understood to mean substantially parallel or approximately parallel.

In one example, as shown in fig. 7, the first space position may be a first space position when the ToF module is in the standard placement position, and the second space position may be a position after rotating 180 ° in the XY plane when the ToF module is in the first space position. The ToF module is in a first space pose corresponding to the standard placement pose and a second space pose rotated 180 degrees, the relative parallel state of the ToF module and the calibration plate is more standard, the measurement error caused by relative inclination is smaller, and the depth of field distance of the calibration plate collected by the ToF module is closer to the real distance between the calibration plate and the ToF module. The accuracy of the depth calibration can be effectively improved by reducing errors caused by the relative positions as much as possible.

In other examples, on the premise of ensuring that the first space pose and the second space pose are different, the first space pose and the second space pose where the mobile module drives the ToF module to rotate can also have other implementation schemes, which are not limited.

As shown in fig. 8, the ToF module acquires a first depth map (shown in (a) of fig. 8) and a second depth map (shown in (b) of fig. 8) at a first measurement point T1. For convenience of description, fig. 8 illustrates pixels of 11×9, and fig. 8 only illustrates depth values of a small number of pixels, and the specific size and depth values of the depth map are not limited.

In fig. 8, a first pixel X1 and a second pixel X2 exist in the first depth map, and a third pixel X3 and a fourth pixel X4 exist in the second depth map. By comparing the depth values of all the pixels in the first depth map and the second depth map, if the depth value of the first pixel X1 and the depth value of the third pixel X3 are determined to be the same, the first pixel X1 and the third pixel X3 are central pixels. For specific reasons, reference may be made to the above explanation regarding the fact that only the depth value of the center pixel is unchanged during rotation of the ToF module. Color blocks of different depths refer to different depth values acquired by each pixel point in fig. 8.

Corresponding to fig. 7 and fig. 8 (a), the first pixel point X1 records a depth value C1 of the first calibration point C1 on the calibration plate acquired by the ToF module in the first spatial pose, and the second pixel point X2 records a depth value A1 of the second calibration point A1 on the calibration plate.

Corresponding to fig. 7 and fig. 8 (b), the third pixel point X3 records a depth value C1 of the first calibration point C1 on the calibration plate acquired by the ToF module in the second spatial pose, and the fourth pixel point X4 records a depth value A1 'of the third calibration point A1' on the calibration plate.

And taking a connecting line between the central pixel point of the ToF module and the central calibration plate of the calibration plate as a central shaft, and rotating the ToF module around the central shaft. Based on the rotation consistency principle, in the process that the ToF module rotates around the central shaft, the distance between a certain calibration point on the central calibration plate and the central calibration point is unchanged, and the Z-direction distance between the central calibration point and the central pixel point is also unchanged, so that even if the ToF module rotates 180 degrees from the first space pose to the second space pose, the included angle between the optical path between the calibration point and the central pixel point and the central shaft is unchanged.

For example, as shown in fig. 7, the angle between the optical path A1T1 between the second calibration point A1 corresponding to the first space pose and the first measurement point T1 and the central axis C1T1 where the first calibration point C1 is located is +.a1t1c1, and the angle between the optical path A1' T1 between the third calibration point A1' corresponding to the second space pose and the first measurement point T1 and the central axis C1T1 where the first calibration point C1 is located is equal to +.a1t1c1= a1' t1c1.

S602: the controller controls the moving assembly to drive the ToF module to move along the first direction, so that the laser emission surface of the ToF module is separated from the calibration plate by a second distance; the mobile component is controlled to drive the ToF module to rotate around the center point of the laser emission surface of the ToF module and parallel to the calibration plate, so that the ToF module is respectively in a third space pose and a fourth space pose, the ToF module is controlled to shoot the calibration plate in the third space pose to obtain a third depth image, and the calibration plate is controlled to shoot the calibration plate in the fourth space pose to obtain a fourth depth image;

as shown in fig. 7, the moving assembly drives the ToF module to move to the first measuring point T2, the first plane P1 where the laser emitting surface of the ToF module is located is parallel to the second plane P2 where the laser reflecting surface of the calibration plate is located, and the Z-direction distance between the laser emitting surface of the ToF module and the laser reflecting surface of the calibration plate is the second distance d2.

The ToF module is at the second measuring point T2, and the moving module drives the ToF module to rotate in the XY plane, so that the ToF module is in the third space pose. And the ToF module emits laser to the calibration plate and acquires a third depth map corresponding to the calibration plate. The moving module drives the ToF module to rotate in the XY plane so that the ToF module is in the fourth space position. The ToF module emits laser to the calibration plate, and a fourth depth map corresponding to the calibration plate is collected.

In one example, the third spatial pose is the same as the first spatial pose and the fourth spatial pose is the same as the second spatial pose.

Corresponding to fig. 7 and 8 (a), the first pixel point records the depth value C2 of the fourth calibration point C2 on the calibration plate, and the second pixel point X2 records the depth value A2 of the fifth calibration point A2 on the calibration plate.

Corresponding to fig. 7 and fig. 8 (b), the third pixel point X3 records a depth value C2 of the fourth calibration point C2 on the calibration plate acquired by the ToF module in the second spatial pose, and the fourth pixel point X4 records a depth value A2 'of the sixth calibration point A2' on the calibration plate.

Based on the analysis of the principle of angle consistency in the rotation process of the ToF module, as shown in fig. 7, the optical path A2T2 between the fifth calibration point A2 corresponding to the third space pose and the second measurement point T2, the angle +.a2t2c2 between the central axis C2T2 where the fourth calibration point C2 is located, and the optical path A2' T2 between the sixth calibration point A2' corresponding to the fourth space pose and the second measurement point T2, and the angle +.a2t2c2 between the central axis C2T2 where the fourth calibration point C2 is located are equal, that is +.a2t2c2= a2' T2C2.

S603: and the controller performs depth calibration on the ToF module according to the first depth map, the second depth map, the third depth map, the fourth depth map, the first distance and the second distance.

The controller controls the ToF module to acquire depth maps at two measuring points with different distances and different spatial poses respectively, so that four depth maps can be obtained. The Z-direction distances of the first depth map and the second depth map are the same, and the Z-direction distances of the third depth map and the fourth depth map are the same. And the first space pose where the first depth map is acquired is the same as the third space pose where the third depth map is acquired, and the second space pose where the second depth map is acquired is the same as the fourth space pose where the fourth depth map is acquired.

The controller can obtain an angle conversion formula and the relative displacement of the pixel points according to two depth maps with the same Z-direction distance, and further obtain an internal reference matrix of the ToF module through multiple calculation and conversion.

In addition, the depth calibration system may also perform laser offset correction on the ToF module before performing the internal reference calibration operations of S601-S603. The laser offset correction by the depth calibration system may comprise the steps of:

the controller controls the moving assembly to drive the ToF module to move along the first direction, so that the laser emission surface of the ToF module is separated from the calibration plate by a third distance; the mobile component is controlled to drive the ToF module to rotate around the center point of the laser emission surface of the ToF module and parallel to the calibration plate, so that the ToF module is respectively in a fifth space pose and a sixth space pose, the ToF module is controlled to shoot the calibration plate in the fifth space pose to obtain a fifth depth map, and the calibration plate is shot in the sixth space pose to obtain a sixth depth map;

And the controller corrects the laser offset of the ToF module according to the fifth depth map and the sixth depth map.

And the controller controls the moving assembly to drive the ToF module to move to the first measuring point, and rotates to enable the ToF module to be in a fifth space pose and a sixth space pose respectively. And when the ToF module is in the fifth space pose, laser is emitted to the calibration plate, and a fifth depth map is acquired. And when the ToF module is in the sixth space pose, laser is emitted to the calibration plate, and a sixth depth map is acquired. The specific implementation process of the controller controlling the ToF module to acquire the fifth depth map and the sixth depth map is not described herein.

The moving assembly can control the ToF module to rotate around the center point of the laser emitting surface of the ToF module in the process of driving the ToF module to rotate in the XY plane. Thus, the Y-axis height of the center point of the laser emitting surface of the ToF module is always consistent with the Y-axis height of the center point of the laser reflecting surface of the calibration plate. The Z-direction distance between the center point of the laser emitting surface of the ToF module and the center point of the laser reflecting surface of the calibration plate is always consistent, and the depth of field distance between the center calibration point of the calibration plate acquired by the center pixel point of the ToF module is always consistent. And reversely deducing that if the depth value acquired by a certain pixel point is always unchanged in all pixel areas of the ToF module in the process of rotating parallel to the calibration plate, the pixel point is the central pixel point. Thus, the controller can acquire the center pixel points of the fifth depth map and the sixth depth map; the center pixel point is a pixel point with the same depth value in the fifth depth map and the sixth depth map.

The controller acquires a laser recording histogram of the central pixel point; wherein the laser recording histogram comprises a waveform map of at least two laser signals. The controller calculates a laser offset distance according to the difference between the target time of the waveform diagram of the first laser signal recorded in the laser recording histogram and the statistical 0 point time of the laser recording histogram; wherein the target time of the waveform of the first laser signal includes a peak time. And the controller corrects the laser offset of the ToF module according to the laser offset distance.

In a specific example, as shown in fig. 9 (a), a histogram of the center pixel point recorded by the ToF module is obtained for the controller. The first laser signal waveform corresponding to the central pixel point recorded by the histogram, the distance between the peak position and the statistical 0 point of the histogram is delta 1, namely the laser offset distance is delta 1. And the actual depth of field distance between the center pixel point and the calibration plate is d1. Then, in order to perform laser offset correction on the ToF module, the depth value obtained by the ToF module is superimposed by the laser offset distance, so as to obtain the actual depth distance, and implement laser offset correction, where after correction, the depth value is shown in fig. 9 (b).

As shown in the example of fig. 9, the lateral time unit is bin,1bin is approximately 500 picoseconds is approximately 5 x 10 ^-10 Second. The laser flying speed is 3 x 10 ⁸ Meter/second, Δ1=22×5×10 ^-10 Second/2 x 3 x 10 ⁸ Meter/second approximately 1.65 meter approximately 165 cm.

For laser offset correction, the position of the peak of the histogram signal is at d1+Δ1, and Δ1 is usually 150 cm to 200 cm, for example Δ1=165 cm.

In other examples, the number of corresponding time units may be different from case to case, and may be 10 bins or 30 bins.

The lateral time is 10bin, Δ1=105×10 ^-10 Second/2 x 3 x 10 ⁸ Meter/second approximately 0.75 meter approximately 75 cm.

The lateral time is 30bin, Δ1=30×5×10 ^-10 Second/2 x 3 x 10 ⁸ Meter/second approximately 2.25 meters approximately 225 cm.

That is, the value of Δ1 for laser bias correction is adjusted according to different conditions or different devices, and is not limited.

The depth calibration system calculates the laser offset distance according to the steps, and executes the reference matrix calibration corresponding to S602-S603 after the laser offset correction, so that the influence of the laser offset on the reference matrix calibration can be reduced, and the overall accuracy of the depth calibration of the ToF module is improved.

In one example, continuing with fig. 7 and 8, the actual depth-of-field distance may be further superimposed with the laser offset distance to obtain a closer depth-of-field distance based on the depth values recorded for each depth-map.

For example, at the first measurement point T1, corresponding to the first calibration point C1 on the calibration plate, the first depth map records a depth value C1, and the superimposed offset distance Δd is denoted as a first depth value c1+Δd. After the ToF module rotates 180 °, the depth value recorded by the second depth map is still c1, and the superimposed offset distance Δd is still recorded as the first depth value (c1+Δd).

The offset distance superimposed on the depth value recorded by the depth map includes two error values, one is an error value Δ1 caused by laser offset, and the other is a laser offset error value Δ2 which can be corrected in the internal reference calibration stage. That is, Δd=Δ1+Δ2.Δ1 is the offset distance that has been calculated during the laser offset correction phase, and can be corrected during the laser offset phase. Δ2 is the offset distance that needs to be calculated and corrected by the subsequent internal reference calibration. Before the determination of the internal parameter matrix, Δ2 is an unknown parameter, and a specific value of Δ2 is determined after the calculation of the internal parameter matrix.

For the offset distance delta 1 caused by laser offset, the controller can automatically add the laser offset distance delta 1 into calculation before internal parameter calibration by adjusting a register of the ToF module, that is, the depth value c1 in the depth map acquired by the subsequent internal parameter calibration is automatically overlapped with delta 1, and only delta 2 is overlapped on the depth value c1 and substituted into a related calculation formula of the subsequent internal parameter calibration stage to calculate the unknown parameter delta 2. For convenience of description, the following description will directly replace all offset distances with Δd, and the distinguishing processing operation for Δ1 and Δ2 will not be described in detail, and the following is the same.

Corresponding to a second calibration point A1 on the calibration plate, the depth recorded by the first depth map is A1, and the superposition offset distance delta d is recorded as a second depth value a1+delta d. After the ToF module rotates 180 °, the depth recorded by the second depth map is a1', and the superimposed offset distance Δd is recorded as a third depth value (a 1' +Δd).

Correspondingly, at the second measurement point T2, corresponding to a fourth calibration point C2 on the calibration plate, the depth value recorded by the third depth map is C2, and the superposition offset distance deltad is recorded as a fourth depth value c2+deltad. After the ToF module rotates 180 °, the depth value recorded by the fourth depth map is still c2, and the superimposed offset distance Δd is still recorded as a fifth depth value (c2+Δd).

Corresponding to a second fifth fixed point A2 on the calibration plate, the depth value recorded by the third depth map is A2, and the superposition offset distance delta d is recorded as a fifth depth value a2+delta d. After the ToF module rotates 180 °, the depth value recorded by the fourth depth map is a2', and the superimposed offset distance Δd is recorded as a sixth depth value (a 2' +Δd).

The controller performs depth calibration on the ToF module according to the first depth map, the second depth map, the third depth map and the fourth depth map, and mainly completes calibration and offset correction of the internal reference matrix. In a specific embodiment, as shown in fig. 10, the step of performing the calibration of the reference matrix in S603 by the controller may mainly include:

S1001: the controller obtains the relative displacement of each pixel point and the central pixel point of the ToF module and the included angle of the corresponding light path of each pixel point and the first direction according to the first depth map, the second depth map, the third depth map and the fourth depth map;

specifically, the controller calculates a first angle conversion formula when the first included angle is equal to the second included angle according to the first depth map and the second depth map; the calibration plate comprises a first calibration point, a second calibration point and a third calibration point, wherein the first calibration point corresponds to a first depth image and a central pixel point of the second depth image, a first included angle is formed between a first light path between the first calibration point and the first measurement point and a second light path between the second calibration point and the first measurement point, and a second included angle is formed between the first light path between the first calibration point and the first measurement point and a third light path between the third calibration point and the first measurement point;

the controller calculates a second angle conversion formula when the third included angle is equal to the fourth included angle according to the third depth map and the fourth depth map; the calibration plate comprises a fourth calibration point, a fifth calibration point and a sixth calibration point, the fourth calibration point corresponds to a third depth image and a central pixel point of the fourth depth image, a third included angle is formed between a fourth light path between the fourth calibration point and the second measurement point and a fifth light path between the fifth calibration point and the second measurement point, and a fourth included angle is formed between a fourth light path between the fourth calibration point and the second measurement point and a sixth light path between the sixth calibration point and the second measurement point;

According to the depth values corresponding to the first light path, the second light path, the third light path, the fourth light path, the fifth light path and the sixth light path, the first angle conversion formula and the second angle conversion formula, the relative displacement of each pixel point of the ToF module and the central pixel point and the included angle of the corresponding light path of each pixel point and the first direction are calculated.

S1002: the controller calculates an internal reference matrix of the ToF module according to the relative displacement of each pixel point and the central pixel point of the ToF module and the included angle between the corresponding light path of each pixel point and the first direction;

s1003: and the controller performs depth calibration on the ToF module according to the internal reference matrix of the ToF module.

In one example, with continued reference to fig. 7 and 8 described above, the angle A1T1C1 between the first calibration point C1 and the second calibration point A1, and the angle A1' T1C1 between the first calibration point C1 and the third calibration point A1' are determined from the first depth map and the second depth map, and the angle A1T1C 1= angle A1' T1C1. And determining an included angle A2T2C2 between the fourth calibration point C2 and the fifth calibration point A2 and an included angle A2'2C2 between the fourth calibration point C2 and the sixth calibration point A2', wherein the included angles A2T2C 2= angle A2' T2C2 according to the third depth map and the fourth depth map.

Let a0= (a2+Δd) - (a1+Δd), a0' = (a2 ' +Δd) - (a1 ' +Δd), c= (c2+Δd) - (c1+Δd). The angle between a0 and c corresponds to the angle between a0' and c.

In the above overall scheme, the relative displacement between the second pixel point and the first pixel point in the depth map acquired by the ToF module is calculated. Based on a similar principle, the relative displacement of all pixels in the depth map acquired by the ToF module can be acquired, and the main calculation process is as follows:

setting upVector: />And->Vector: />. Coordinates->，/>。

Calculating the available module length:

；/>。

three pointsI.e. with->And->Collinear, expression 1 can be obtained:

。

there are two points in the line, namely，/>. Substituting the two-point coordinates into expression 1 yields expression 2 and expression 3, respectively. Wherein expression 2 is:

；

expression 3 is:

。

order theThe above expression 2+ expression 3 may be replaced with:

。

continuing to order，

Substitution formula can be obtained:

。

taking the solution with k as an integer can obtainThe slope k of the vector. By mould length is known +.>And->Coordinates, the slope between two points can be obtained as +.>。

And k1 obtained in the deduction process represents the tangent value of the included angle between the plane and the module, and the inverse solution can be obtained: θ=arctan (k 1).

Knowing the angle relation between each pixel point and the central pixel point, a new plane can be fitted by the relation operation of plane equation, and the new plane is marked as a third plane, as shown by P3 in (a) of FIG. 11. In fig. 11, the third plane is a plane parallel to the first plane in which the ToF module is located, and OC is an optical path perpendicular to the third plane. At this point OC is perpendicular to the third plane, with a0 being compensated to a3, a0' to a3', a3=a3 '. From this, in fig. 11 (b), the cosine of the spatial angle α between the second pixel and the first pixel, that is, cosα=c/a 3, can be obtained.

Through the calculation logic, cosine quantities of included angles between all pixel points and the central pixel point can be obtained in the same way.

Continuing with fig. 12, let the point coordinates of the line a to the plane be (ax, ay, az), where ax should be 0, az=c, the spatial coordinates and the pixel coordinates satisfy the following relation:

，

the deduction can be obtained: ay/az= (v-cy)/fy.

Substituting all pixel points of the ToF module and corresponding depth values into the space coordinate conversion formula, and obtaining an optimal internal reference matrix by using a least square method to complete internal reference calibration.

When the ToF module is located at the second measurement point T2, a second distance d2 is between the ToF module and the calibration plate. The histograms actually obtain depth values a2+Δ=a2, a2' +Δ=a2 ', c2+Δ=c2 ', and according to the results of the third steps k and k1, forward-forward formulas, the values of k and k1 are satisfied when Δ=a2-A2, whereby the offset distance Δd can be obtained.

In summary, the depth calibration scheme provided by the embodiment does not need to perform three calibration operations independently and successively, so that the mechanical cost and the field space are greatly reduced. The laser offset correction, the internal reference matrix calibration and the offset calibration can be completed by collecting six depth maps, and the calibration capacity and the labor cost are greatly improved.

In addition, the depth calibration method executed by the depth calibration system provided by the embodiment requires that the ToF module to be calibrated and the calibration plate are kept basically parallel, angle measurement and calculation are realized through laser, and the environment construction difficulty and the spot inspection difficulty are reduced. The dtof calibration of medium and low resolution can not obtain more accurate internal reference information by a Zhengyou calibration method; even if the Zhang Zhengyou calibration method is used, multiple pictures with multiple angles are often required to be taken, and the environment is built and UPH is not friendly. The method based on the depth calibration internal reference can finish the calibration in a faster mode while ensuring the precision.

The depth calibration method and the depth calibration system provided by the embodiment can greatly improve the depth calibration efficiency of the laser range finder, the vehicle-mounted laser radar, the AR/VR and other equipment.

In addition, the application provides a depth calibration system for calibrating a ToF module. The depth calibration system comprises a moving assembly, a calibration plate and a controller.

The controller is used for executing the depth calibration method provided by the previous embodiment.

The application also provides depth calibration equipment for calibrating the ToF module. The depth calibration equipment is connected with a moving assembly, and the ToF module is fixed behind the moving assembly, and the laser emission surface of the ToF module is parallel to the calibration plate.

The depth calibration device comprises a memory and a processor, wherein the memory is coupled with the processor; the memory stores computer-executable instructions; the processor executes computer-executable instructions stored in the memory to cause the depth calibration apparatus to perform the depth calibration method provided by the foregoing embodiment.

The present application also provides a computer-readable storage medium having stored therein a computer program which, when run on a computer, causes the computer to perform the depth calibration method provided by the foregoing embodiments.

The present application also provides a computer program product, comprising a computer program, which when executed by a processor, implements a depth calibration method as provided in the foregoing embodiments.

The specific implementation of the depth calibration system, the depth calibration device, the computer readable storage medium, and the computer program product containing the instructions and the technical effects thereof provided in the embodiments of the present application can refer to the specific implementation process of the depth calibration method and the technical effects thereof provided in the foregoing embodiments, and are not repeated herein.

In some embodiments, it will be clearly understood by those skilled in the art from the foregoing description of the embodiments, for convenience and brevity of description, only the division of the above functional modules is illustrated, and in practical application, the above functional allocation may be implemented by different functional modules, that is, the internal structure of the apparatus is divided into different functional modules to implement all or part of the functions described above. The specific working processes of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which are not described herein.

The functional units in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: flash memory, removable hard disk, read-only memory, random access memory, magnetic or optical disk, and the like.

The foregoing is merely a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the embodiments of the present application should be covered by the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. The depth calibration method is characterized by being used for calibrating the ToF module by a depth calibration system; the depth calibration system comprises a moving assembly, a calibration plate and a controller; after the ToF module is fixed on the moving assembly, the laser emission surface of the ToF module is parallel to the calibration plate; the depth calibration method comprises the following steps:

the controller controls the moving assembly to drive the ToF module to move along a first direction, so that a laser emitting surface of the ToF module is separated from the calibration plate by a first distance; controlling the moving assembly to drive the ToF module to rotate around the center point of the laser emission surface of the ToF module and parallel to the calibration plate, so that the ToF module is respectively in a first space position and a second space position, controlling the ToF module to shoot the calibration plate in the first space position to obtain a first depth map, and shooting the calibration plate in the second space position to obtain a second depth map;

The controller controls the moving assembly to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is separated from the calibration plate by a second distance; controlling the moving assembly to drive the ToF module to rotate around the center point of the laser emission surface of the ToF module and parallel to the calibration plate, so that the ToF module is respectively in a third space position and a fourth space position, controlling the ToF module to shoot the calibration plate in the third space position to obtain a third depth map, and shooting the calibration plate in the fourth space position to obtain a fourth depth map;

the controller performs depth calibration on the ToF module according to the first depth map, the second depth map, the third depth map, the fourth depth map, the first distance and the second distance; wherein the first direction is perpendicular to the calibration plate, and the first distance is different from the second distance; the first and second spatial positions are different, and the third and fourth spatial positions are different.

2. The depth calibration method of claim 1, wherein the first and third spatial positions are the same and the second and fourth spatial positions are the same.

3. The depth calibration method according to claim 1 or 2, wherein the ToF module is rotated 180 degrees parallel to the calibration plate from the first spatial pose around a center point of a laser emitting surface of the ToF module in the second spatial pose;

the ToF module is rotated 180 degrees around the center point of the laser emission surface of the ToF module by the third space pose and is parallel to the calibration plate, and the ToF module is in the fourth space pose.

4. The depth calibration method according to claim 1 or 2, wherein the laser emission surface of the ToF module is parallel to the calibration plate, comprising:

and the included angle between the laser emission surface of the ToF module and the calibration plate is smaller than a preset angle threshold.

5. The depth calibration method according to claim 1, wherein the controlling the moving assembly drives the ToF module to rotate around a center point of a laser emitting surface of the ToF module parallel to the calibration plate, so that the ToF module is in a first space pose and a second space pose respectively, and controls the ToF module to shoot the calibration plate in the first space pose to obtain a first depth map, and before the step of shooting the calibration plate in the second space pose to obtain a second depth map, the depth calibration method further comprises:

The controller controls the moving assembly to drive the ToF module to move along the first direction, so that the laser emission surface of the ToF module is separated from the calibration plate by a third distance; controlling the moving assembly to drive the ToF module to rotate around the center point of the laser emission surface of the ToF module and parallel to the calibration plate, so that the ToF module is respectively in a fifth space position and a sixth space position, controlling the ToF module to shoot the calibration plate in the fifth space position to obtain a fifth depth map, and shooting the calibration plate in the sixth space position to obtain a sixth depth map; the fifth space pose and the sixth space pose are different;

6. The depth calibration method according to claim 5, wherein the step of performing laser offset correction on the ToF module by the controller according to the fifth depth map and the sixth depth map includes:

the controller obtains center pixel points of the fifth depth map and the sixth depth map; the center pixel point is a pixel point with the same depth value in the fifth depth map and the sixth depth map;

The controller acquires a laser recording histogram of the central pixel point; wherein the laser recording histogram comprises a waveform diagram of at least two laser signals;

the controller calculates a laser offset distance according to the difference value between the target time of the waveform diagram of the first laser signal recorded in the laser recording histogram and the statistical 0 point time of the laser recording histogram; wherein the target time of the waveform diagram of the first laser signal comprises a peak time;

and the controller corrects the laser offset of the ToF module according to the laser offset distance.

7. The depth calibration method according to any one of claims 1, 2, 5 or 6, wherein the controller performs depth calibration on the ToF module according to the first, second, third and fourth depth maps, and the first and second distances, and comprises:

the controller obtains the relative displacement of each pixel point and the central pixel point of the ToF module and the included angle of the corresponding light path of each pixel point and the first direction according to the first depth map, the second depth map, the third depth map and the fourth depth map;

The controller calculates an internal reference matrix of the ToF module according to the relative displacement of each pixel point and the central pixel point of the ToF module and the included angle between the corresponding light path of each pixel point and the first direction;

and the controller performs depth calibration on the ToF module according to the internal reference matrix of the ToF module.

8. The depth calibration method according to claim 7, wherein the step of obtaining, by the controller, the relative displacement amounts of each pixel point and the center pixel point of the ToF module and the included angle between the corresponding optical path of each pixel point and the first direction according to the first depth map and the second depth map, the third depth map and the fourth depth map includes:

the controller calculates a first angle conversion formula when the first included angle is equal to the second included angle according to the first depth map and the second depth map; the calibration plate comprises a first calibration point, a second calibration point and a third calibration point, wherein the first calibration point corresponds to a central pixel point of the first depth image and the second depth image, a first included angle is formed between a first light path between the first calibration point and a first measuring point and a second light path between the second calibration point and the first measuring point, and a second included angle is formed between a first light path between the first calibration point and the first measuring point and a third light path between the third calibration point and the first measuring point;

The controller calculates a second angle conversion formula when the third included angle is equal to the fourth included angle according to the third depth map and the fourth depth map; the calibration plate comprises a fourth calibration point, a fifth calibration point and a sixth calibration point, the fourth calibration point corresponds to the third depth image and the center pixel point of the fourth depth image, a fourth light path between the fourth calibration point and a second measurement point forms the third included angle with a fifth light path between the fifth calibration point and the second measurement point, and a fourth light path between the fourth calibration point and the second measurement point forms the fourth included angle with a sixth light path between the sixth calibration point and the second measurement point;

and calculating the relative displacement of each pixel point and the central pixel point of the ToF module and the included angle of each pixel point corresponding to the optical path and the first direction according to the depth values corresponding to the first optical path, the second optical path, the third optical path, the fourth optical path, the fifth optical path and the sixth optical path, the first angle conversion formula and the second angle conversion formula.

9. The depth calibration system is characterized by being used for calibrating the ToF module and comprises a moving assembly, a calibration plate and a controller;

The controller is configured to perform the depth calibration method of any one of claims 1-8.

10. The depth calibration device is characterized by being used for calibrating a ToF module, wherein the depth calibration device is connected with a moving assembly, the ToF module is fixed behind the moving assembly, and a laser emission surface of the ToF module is parallel to the calibration plate;

the depth calibration apparatus includes a memory and a processor, the memory coupled to the processor; the memory stores computer-executable instructions; the processor executing computer-executable instructions stored in the memory, causing a depth calibration apparatus to perform the depth calibration method of any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program which, when run on a computer, causes the computer to perform the depth calibration method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program which, when executed by a processor, implements a depth calibration method according to any one of claims 1 to 8.