CN117876502A - Depth calibration method, depth calibration equipment and depth calibration system - Google Patents

Abstract

The present application provides a depth calibration method, a depth calibration device and a depth calibration system, which relate to the field of terminal technology. The mobile component of the depth calibration system is used to drive the ToF module to move and rotate. When the ToF module is at a first distance from the calibration plate, the first depth map and the second depth map are collected in the first spatial posture and the second spatial posture respectively, and then the third depth map and the fourth depth map are collected in the third spatial posture and the fourth spatial posture respectively when the ToF module is moved to the second distance. The controller performs depth calibration on the ToF module according to the four depth maps and the first distance and the second distance, and completes the internal parameter calibration of the ToF module. The depth calibration can also collect 2 more depth maps to calculate the laser offset distance, and perform laser offset correction before the internal parameter calibration. In this way, without the need to perform three calibration operations separately, laser offset correction, internal parameter matrix calibration and offset calibration can be completed by collecting six depth maps, which greatly improves the calibration capacity and labor costs.

Description

Depth calibration method, depth calibration equipment and depth calibration system

Technical Field

The embodiments of the present application relate to the field of terminal technology, and in particular to a depth calibration method, a depth calibration device, and a depth calibration system.

Background technique

Time-of-Flight (ToF) is a technology that uses the laser flight time to measure distance. ToF can be applied to distance measurement equipment, such as depth cameras. Due to lens assembly, edge distortion and other reasons, the ToF module may cause laser offset, lens offset and other problems, which may lead to errors between the acquired depth of field distance and the actual distance. The ToF module needs to be calibrated to minimize the measurement error.

The existing calibration scheme is: laser offset correction (Laser Offset), camera calibration (CameraCalibration) and offset calibration (Offset Calibration) in sequence. These three calibration operations increase equipment costs and labor costs, and will limit the overall production capacity (Units Per Hour, UPH). The existing calibration scheme includes three calibration steps that are performed sequentially, which is time-consuming and affects the accuracy between the calibration steps, resulting in low depth calibration accuracy. Different calibration plates need to be prepared for the three calibration steps, and the calibration consumables are more and the calibration steps are more cumbersome.

Summary of the invention

The embodiments of the present application provide a depth calibration method, a depth calibration device and a depth calibration system, which are used to solve the technical problems of low accuracy and complicated steps in existing depth camera calibration solutions.

To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

In the first aspect, a depth calibration method is provided, which is used for calibrating a ToF module in a depth calibration system. The ToF module can be a depth acquisition module in a depth camera of an electronic device. The depth calibration system includes a mobile component, a calibration plate and a controller, and the mobile component is used to fix the ToF module and drive the ToF module to move. After the ToF module is fixed to the mobile component, the laser emitting surface of the ToF module is parallel to the calibration plate, and the parallelism here can be absolutely parallel or approximately parallel. During actual assembly, the laser emitting surface of the ToF module is approximately parallel to the laser reflecting surface of the calibration plate, which can be interpreted as a small angle between the two planes. The depth calibration method provided in this application does not require that the laser emitting surface of the ToF module is absolutely parallel to the laser reflecting surface of the calibration plate.

The depth calibration method provided in the present application can be mainly controlled by a controller to control the mobile component to drive the ToF module to move, obtain the depth map and related measurement data collected by the ToF module for calculation, and realize depth calibration of the ToF module.

The controller can control the moving component to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is spaced a first distance from the laser reflecting surface of the calibration plate. The laser emitting surface of the ToF module is parallel to the calibration plate, and the first direction is perpendicular to the laser reflecting surface of the calibration plate and the laser emitting surface of the ToF module. For example, in a three-dimensional reference coordinate system, the laser emitting surface of the ToF module and the calibration plate can be parallel to the XY plane, and the first direction is perpendicular to the laser emitting surface of the ToF module, which can be interpreted as the first direction being parallel to the Z axis.

The controller controls the moving component to drive the ToF module to rotate around the center point of the laser emission surface of the ToF module parallel to the calibration plate, that is, the moving component drives the ToF module to rotate in the XY plane, and the Y-axis coordinate of the center point of the laser emission surface of the ToF module is kept unchanged during the rotation. The controller controls the moving component to drive the ToF module to rotate around the center point in the XY plane, so that the ToF module is in a first spatial posture and a second spatial posture respectively, and the first spatial posture is different from the second spatial posture. For example, when the controller controls the moving component to drive the ToF module to rotate to be in the first spatial posture, the moving component can drive the ToF module to rotate 180° in the XY plane, so that the spatial posture of the ToF module changes from the first spatial posture to the second spatial posture.

When the laser emission surface of the ToF module is separated from the calibration plate by a first distance, the controller controls the ToF module to be in a first spatial posture, emits laser light to the calibration plate, and photographs the calibration plate to obtain a first depth map; when the controller controls the ToF module to be in a second spatial posture, it emits laser light to the calibration plate, and photographs the calibration plate to obtain a second depth map. In this way, the ToF module acquires the first depth map and the second depth map at the measurement point at the first distance from the calibration plate in two different spatial postures.

The controller controls the moving component to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is separated from the calibration plate by a second distance, and the first distance is different from the second distance. The controller then controls the moving component to drive the ToF module to rotate around the center point of the laser emitting surface of the ToF module parallel to the calibration plate, so that the ToF module is in a third spatial posture and a fourth spatial posture, respectively, and the third spatial posture and the fourth spatial posture are different.

When the laser emitting surface of the ToF module is spaced a second distance from the calibration plate, the controller controls the ToF module to photograph the calibration plate in a third spatial position to obtain a third depth map, and to photograph the calibration plate in a fourth spatial position to obtain a fourth depth map.

Finally, the controller performs depth calibration on the ToF module according to the first depth map, the second depth map, the third depth map, and the fourth depth map, as well as the first distance and the second distance. For example, the controller can obtain the optical path distance expression formula and the angle expression formula of the calibration point corresponding to each pixel point according to multiple depth maps, the first distance and the second distance, and then convert the optical path distance expression formula and the angle expression formula according to the angle correspondence relationship or optical path distance correspondence relationship between different depth maps due to equal distance or spatial posture, so as to calculate the relative displacement of each pixel point of the ToF module relative to the central pixel point, and then obtain the intrinsic parameter matrix of the ToF module, and then use the intrinsic parameter matrix to perform depth calibration on the ToF module, especially intrinsic parameter calibration and offset calibration.

The above-mentioned depth calibration scheme does not require the separate layout of three calibration systems and the execution of three calibration operations in succession, which greatly reduces the calibration cost, site space requirements and the complexity of calibration operations. By collecting four depth maps, the intrinsic parameter matrix calibration and offset calibration in the depth calibration task can be completed, which greatly improves the calibration capacity and labor costs. The depth calibration method provided in this application requires that the ToF module to be calibrated is basically parallel to the calibration plate, and angle measurement is achieved by laser, which reduces the difficulty of environment construction and inspection. It effectively avoids the limitation of the resolution of the ToF module by the precise distance parameters of the grid calibration plate required to be collected in Zhang Zhengyou's calibration method, and can use a faster method to complete the calibration while ensuring accuracy.

In a possible implementation of the first aspect, the first spatial pose and the third spatial pose are the same, and the second spatial pose and the fourth spatial pose are the same. The first spatial pose and the second spatial pose are different spatial poses when the depth map is collected when the laser emission surface of the ToF module is separated from the calibration plate by a first distance, and the third spatial pose and the fourth spatial pose are different spatial poses when the depth map is collected when the laser emission surface of the ToF module is separated from the calibration plate by a second distance. During the two acquisition operations, the first distance and the second distance between the ToF module and the calibration plate are different.

That is to say, the first spatial pose and the third spatial pose are the same spatial poses collected when the laser emission surface of the ToF module is at different distances from the calibration plate; the second spatial pose and the fourth spatial pose are the same spatial poses collected when the laser emission surface of the ToF module is at different distances from the calibration plate. In this way, the errors caused by different spatial poses can be effectively reduced, and the accuracy of depth calibration of the ToF module can be further improved.

In a possible implementation of the first aspect, the scheme for rotating and switching the spatial posture of the ToF module is further limited. Specifically, there is a center point on the laser emission surface of the ToF module, and when the ToF module rotates, it rotates around the center point and parallel to the calibration plate. In other words, the center point on the laser emission surface of the calibration plate and the center point of the laser emission surface of the ToF module are placed at the same height, and the line between the two center points is used as the center axis to control the ToF module to rotate around the center axis. During the rotation of the ToF module, the laser emission surface of the ToF module is always parallel to the laser emission surface of the calibration plate.

When the laser emission surface of the ToF module is at a first distance from the calibration plate, the ToF module is in a first spatial position, and the mobile component drives the ToF module to rotate 180° on the premise of being parallel to the laser emission surface of the calibration plate, and the ToF module is in a second spatial position. When the ToF module rotates 180°, its pixels also rotate 180°, and all the pixels of the ToF module are in an axisymmetric state. According to the correspondence between some pixels in the axisymmetric state, the angle correspondence and the optical path correspondence can be obtained.

Based on the same principle, when the laser emission surface of the ToF module is separated from the calibration plate by the second distance, and the ToF module is in the third spatial posture, the mobile component drives the ToF module to rotate 180° on the premise of being parallel to the laser emission surface of the calibration plate, and the ToF module is in the fourth spatial posture. When the ToF module rotates 180°, its pixels also rotate 180°. All the pixels of the ToF module are in an axisymmetric state. According to the correspondence between some pixels in the axisymmetric state, the angle correspondence and the optical path correspondence can be obtained.

In a possible implementation of the first aspect, the parallel condition between the laser emitting surface of the ToF module and the calibration plate during the depth calibration process is further limited. Specifically, the laser emitting surface of the ToF module and the laser reflecting surface of the calibration plate can be approximately parallel.

Here, the laser emitting surface of the ToF module is approximately parallel to the calibration plate. The specific implementation scheme can be: the angle between the laser emitting surface of the ToF module and the calibration plate is less than the preset angle threshold. The preset angle threshold can be set to an angle range of 1°-5°, for example, the preset angle threshold can be 5°. This can reduce the accuracy requirements for the relative parallelism of the calibration plate and the ToF module during the construction of the depth calibration system and simplify the depth calibration operation.

In a possible implementation manner of the first aspect, a solution of performing laser offset correction before performing internal parameter calibration on the ToF module is newly added.

The process of laser offset correction by the depth calibration system may specifically include: the controller controls the moving component to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is separated from the calibration plate by a third distance. The third distance here may be the same as the first distance or the second distance involved in the internal reference calibration process, or it may be different.

The controller controls the mobile component to drive the ToF module to rotate parallel to the calibration plate, so that the ToF module is in the fifth spatial posture and the sixth spatial posture respectively, and the fifth spatial posture and the sixth spatial posture are different spatial postures. For example, the ToF module rotates 180° in the fifth spatial posture and switches to the sixth spatial posture. In the process of the mobile component driving the ToF module to rotate, it is also necessary to rotate around the center point of the laser emitting surface of the ToF module and always parallel to the calibration plate. The fifth spatial posture of the ToF module during the laser offset correction stage can be the same as the first spatial posture during the internal reference calibration stage, and the sixth spatial posture during the laser offset correction stage can be the same as the second spatial posture during the internal reference calibration stage. In this way, the error caused by different spatial postures during the depth calibration process can be minimized, and the accuracy of the depth calibration of the ToF module can be improved.

The controller controls the ToF module to be in a fifth spatial posture and to photograph the calibration plate to obtain a fifth depth map; and controls the ToF module to be in a sixth spatial posture and to photograph the calibration plate to obtain a sixth depth map.

Afterwards, the controller performs laser offset correction on the ToF module based on the fifth depth map and the sixth depth map. There can be many specific implementation schemes. For example, the controller can compare the depth values of all pixels of the two depth maps of the same calibration plate collected when the ToF module is in different spatial postures to find the center pixel. Since the center pixel of the ToF module is at the same height as the center calibration point of the calibration plate, the depth value collected by the center pixel has a relatively small error. The controller can perform laser offset correction on the ToF module based on the relevant data of the center pixel.

The depth calibration method provided in this application performs laser offset correction on the ToF module by collecting two depth maps of the ToF module in different spatial positions at one measurement point. The operation is simple and can improve the accuracy of laser offset correction. At the same time, it can also reduce the impact of laser offset on subsequent internal reference calibration and improve the overall accuracy of ToF module depth calibration.

In a possible implementation of the first aspect, a specific limitation is made on the scheme for the controller to perform laser offset correction on the ToF module according to the fifth depth map and the sixth depth map. The fifth spatial posture at which the ToF module collects the fifth depth map is switched to the sixth spatial posture after rotating 180°, that is, the spatial posture at which the ToF module collects the sixth depth map. Then, before and after rotating 180°, the center pixel point of the ToF module collects the depth value of the center calibration point of the calibration plate. Under the premise that the calibration plate does not change, the depth value collected by the center pixel point of the ToF module does not change.

The controller searches for the pixel whose depth value has not changed in the fifth depth map relative to the sixth depth map based on the depth values of all the pixels in the fifth depth map and the sixth depth map, which is the center pixel. The controller obtains the laser recording histogram of the center pixel, which records the waveforms of multiple groups of laser signals emitted by the laser generator corresponding to the center pixel. The controller calculates the laser offset distance based on the difference between the target moment of the waveform of the first laser signal recorded in the laser recording histogram and the statistical 0 point moment of the laser recording histogram. To improve the correction accuracy, the peak moment of the waveform of the first laser signal can be selected to count the difference with the statistical 0 point of the laser recording histogram. The controller performs laser offset correction on the ToF module based on the laser offset distance.

The depth calibration system first performs laser offset correction on the ToF module, and then uses the ToF module after laser offset correction to collect the first depth map to the fourth depth map to achieve internal parameter calibration, while improving the accuracy of the internal parameter calibration stage.

In a possible implementation of the first aspect, a scheme for the controller to perform depth calibration on the ToF module is further defined. Specifically, the controller obtains the optical path distance expression formula and the angle expression formula of the calibration point corresponding to each pixel of the ToF module according to the first depth map, the second depth map, the third depth map, and the fourth depth map, and then converts the optical path distance expression formula and the angle expression formula according to the angle correspondence relationship or optical path distance correspondence relationship between different depth maps due to equal distance or spatial posture, so as to calculate the relative displacement of each pixel of the ToF module relative to the center pixel, and then obtain the intrinsic parameter matrix of the ToF module, and then use the intrinsic parameter matrix to perform depth calibration on the ToF module, especially intrinsic parameter calibration and offset calibration.

In a possible implementation of the first aspect, a scheme for the controller to obtain the relative displacement of each pixel point and the central pixel point of the ToF module, and the angle between the optical path corresponding to each pixel point and the first direction is further limited.

When the laser emission surface of the ToF module is spaced a first distance from the calibration plate, the calibration plate includes a first calibration point, a second calibration point, and a third calibration point. The first calibration point corresponds to the central pixel point of the first depth map and the second depth map, and the second calibration point and the third calibration point are two axisymmetric calibration points. The first optical path between the first calibration point and the first measurement point forms a first angle with the second optical path between the second calibration point and the first measurement point, and the first optical path between the first calibration point and the first measurement point forms a second angle with the third optical path between the third calibration point and the first measurement point. The controller determines that the first angle is equal to the second angle based on the first depth map collected before the rotation and the second depth map collected after the rotation of 180°. According to the relationship expression between the first angle and the second angle, multiple angle conversion formulas are calculated when the first angle is equal to the second angle, which are recorded as the first angle conversion formula.

Similarly, when the laser emission surface of the ToF module is spaced from the calibration plate by the second distance, the calibration plate includes the fourth calibration point, the fifth calibration point and the sixth calibration point, the fourth calibration point corresponds to the central pixel point of the third depth map and the fourth depth map, and the fifth calibration point and the sixth calibration point are two axisymmetric calibration points. The fourth optical path from the fourth calibration point to the second measurement point and the fifth optical path from the fifth calibration point to the second measurement point form a third angle, and the fourth optical path from the fourth calibration point to the second measurement point and the sixth optical path from the sixth calibration point to the second measurement point form a fourth angle.

The controller calculates the relative displacement of each pixel of the ToF module and the central pixel, as well as the angle between the optical path corresponding to each pixel and the first direction, based on the depth values corresponding to the first optical path, the second optical path, the third optical path, the fourth optical path, the fifth optical path and the sixth optical path, as well as the first angle conversion formula and the second angle conversion formula. The controller calculates the second angle conversion formula when the third angle is equal to the fourth angle based on the third depth map and the fourth depth map; wherein, since the spatial posture of the ToF module collecting the third depth map and the spatial posture of the fourth depth map are 180° different, the third angle is equal to the fourth angle. According to the relationship expression between the third angle and the fourth angle, multiple angle conversion formulas are calculated when the third angle is equal to the fourth angle, which are recorded as the second angle conversion formula. The first angle conversion formula and the second angle conversion formula are brought into the calculation of the internal parameter matrix to complete the depth calibration.

The depth calibration method provided in the present application can complete the three calibration stages of depth calibration by collecting six depth maps, thereby simplifying the site and the overall process and improving the accuracy of depth calibration.

In a second aspect, the present application provides a depth calibration system for calibrating a ToF module. The depth calibration system includes a moving component, a calibration board, and a controller.

The controller is used to execute any depth calibration method in the first aspect.

In a third aspect, the present application provides a depth calibration device for calibrating a ToF module. The depth calibration device is connected to a moving component, the ToF module is fixed to the moving component, and the laser emitting surface of the ToF module is parallel to the calibration plate.

The depth calibration device includes a memory and a processor, the memory is coupled to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the depth calibration device executes the depth calibration method executed by the controller in any one of the first aspects.

In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the depth calibration method as described in any one of the first aspects.

In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the depth calibration method as described in any one of the first aspects.

Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a ToF module performing depth data acquisition;

FIG2 is a schematic diagram of the offset laser correction involved in depth calibration of the ToF module;

FIG3 is a schematic diagram of a grid calibration plate involved in depth calibration of a ToF module;

FIG4 is a schematic diagram of emitting laser light to a grid calibration plate involved in depth calibration of a ToF module;

FIG5 is a schematic diagram of laser offset involved in depth calibration of a ToF module;

FIG6 is a schematic diagram of a flow chart of a depth calibration method provided in an embodiment of the present application;

FIG7 is a schematic diagram of the calibration principle of the depth calibration system provided in an embodiment of the present application;

FIG8 is a schematic diagram of a depth map involved in the depth calibration method provided in an embodiment of the present application;

FIG9 is a schematic diagram of a laser recording histogram involved in the depth calibration method provided in an embodiment of the present application;

FIG10 is a schematic diagram of a flow chart of an internal parameter calibration involved in a depth calibration method provided in an embodiment of the present application;

FIG11 is a schematic diagram of fitting a new plane in the intrinsic parameter calibration process involved in the depth calibration method provided in an embodiment of the present application;

FIG. 12 is a schematic diagram of coordinate transformation involved in the depth calibration method provided in an embodiment of the present application.

Detailed ways

The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

To facilitate understanding, some technical common sense involved in the embodiments of the present application is first introduced.

Laser Time-of-Flight (ToF) is a technology that uses the laser flight time to measure distance. Among them, dToF, or direct Time-of-Flight, is a technology that measures distance by directly measuring the flight time.

The ToF ranging principle is that the ToF module emits laser to the target at the detection point and receives the laser reflected from the surface of the target. It calculates the distance between the detection point and the surface of the target by detecting the round-trip flight time of the laser and combining the laser flight speed.

ToF can be applied to distance measurement devices (such as depth cameras). Depth cameras use ToF technology to collect the depth distance between the depth camera and each point in the viewing range. The depth camera can use the depth distance of each point in the viewing range and the two-dimensional coordinates of each point in the two-dimensional image to obtain the three-dimensional spatial coordinates of each point in the world coordinate system.

The depth camera is equipped with a ToF module, as shown in Figure 1. The ToF module can perform laser ranging on targets within the field of view. The ToF module mainly includes a laser transmitter, a laser receiver, a lens assembly, and a processor. Among them, the laser transmitter is used to transmit lasers, the laser receiver is used to receive lasers, and the lens assembly is used to capture images within the field of view. The processor is coupled with the laser transmitter, laser receiver, and lens assembly.

The processor can obtain the first moment when the laser transmitter emits the laser and the second moment when the laser receiver receives the corresponding laser, calculate the laser flight time, and then calculate the corresponding depth of field distance. The processor calculates the depth information of all pixels on the image based on the depth of field distance of each pixel in the image captured by the lens assembly. It should be noted that the laser emitted and received by the ToF module can be a laser pulse.

As shown in Figure 1, the target object can be a three-dimensional object. There are two points D1 and D2 on the surface of the target object, and the distances between D1 and D2 and the ToF module are different. The laser transmitter emits laser S1 to D1 and receives laser S1'. The depth of field distance between D1 and the imaging plane of the ToF module is d1. The laser transmitter emits laser S2 to D2 and receives laser S2'. The depth of field distance between D2 and the imaging plane of the ToF module is d2.

Depth cameras can be applied to image processing or display scenarios such as image blur processing, virtual reality (VR), and augmented reality (AR).

Due to lens assembly, edge distortion and other reasons, the ToF module may cause laser offset, lens offset and other problems, which may lead to errors between the depth of field distance obtained and the actual distance. The ToF module needs to be calibrated to minimize the measurement error.

The existing calibration scheme is: laser offset correction (Laser Offset), camera calibration (Camera Calibration) and offset calibration (Offset Calibration) in sequence. These three calibration operations increase equipment costs, labor costs, and will limit the overall production capacity (Units Per Hour, UPH).

The first calibration operation is laser offset correction.

During ToF calibration, the ToF module emits laser light to the calibration plate. During the laser offset correction phase, the required calibration plate is a white plate with high reflectivity. The ToF module records the time it takes for the laser to be emitted, reflected from the calibration plate, and then returned to the ToF module, which is the laser flight time.

ToF modules usually use histograms to record the emission and reception time of the laser. The range that the histogram can record covers a small amount of laser flight time. As shown in Figure 2, this is a schematic diagram of the laser flight time recorded by the histogram. When the laser is first lit, the histogram records the flight time of the laser that is first lit. The starting moment of the flight time of the laser that is first lit corresponds to the actual position on the histogram at point 0; however, due to the laser offset, the position where the laser is first lit on the histogram is calculated to be position a as shown in Figure 2, which is a distance away from the actual point 0 (as shown in position b as shown in Figure 2). The time period from the actual point 0 to the end point of the histogram (as shown in c in Figure 2) is shortened, resulting in a reduction in the number of complete lasers that can actually be covered by this shortened time period on the histogram.

The purpose of laser offset correction is to measure the offset distance between the actual 0-point position and the statistical 0-point position (as shown in L in Figure 2), and use the offset distance to compensate and calibrate the laser offset. In this way, after laser offset correction, the actual 0-point position when the laser is first lit (as shown in d in Figure 2) will be closer to the 0-point position of the histogram, so that the histogram can cover as many complete laser flight periods as possible.

The second calibration operation is the camera correction calibration, and the third calibration operation is the offset correction calibration.

Camera correction calibration is also called camera intrinsic calibration. The usual camera intrinsic calibration uses the Zhang Zhengyou calibration method and its derived related calibration methods that identify spatial information through image information. Taking the Zhang Zhengyou calibration method as an example, the calibration board used is the checkerboard calibration board shown in Figure 3, on which a checkerboard image is drawn. The grid points on the checkerboard calibration board can be round or square, without limitation. As shown in Figure 4, the ToF module shoots the checkerboard calibration board from multiple angles, and the algorithm identifies the center of the corners of the checkerboard image for calculating spatial information. Then, based on the principle that the depth of field of all points on the checkerboard calibration board is consistent, the camera edge distortion is fitted and corrected by the least squares method to obtain an intrinsic parameter matrix that matches all pixel areas of the ToF module.

dToF is a distance measurement method that directly measures the flight time and converts it into distance. For multi-point lasers, due to the large field of view, the optical path is significantly longer than the z-direction distance. When converting to the z-direction distance, the angle of the optical path needs to be calculated.

As shown in Figure 5, a checkerboard calibration board is set opposite the ToF module. There are target points A1 and A2 on the checkerboard calibration board. The actual depth of field distance d1 between target point A1 and the plane where the ToF module is located should be equal to the actual depth of field distance d2 between target point A2 and the plane where the ToF module is located. In actual measurement, the optical path of the ToF module to emit laser to target point A1 is OA1, and the corresponding optical path distance is equal to the actual depth of field distance d1. Due to factors such as edge distortion and oblique laser emission, the optical path of the ToF module to emit laser to target point A2 is OA2, corresponding to the optical path distance d2', which is not equal to the actual depth of field distance d2.

In order to perform camera calibration, it is necessary to calculate the angle θ between the optical path OA2 and the optical path OA1. The actual depth of field distance d1 can be calculated through the actual optical path distance d2’ and the angle θ.

The geometric relationship of the angle θ is: cosθ=d1/d2, sinθ=d3/d2. If d1, d2 and d3 are known, the angle θ can be calculated. In actual measurement, for any target point Ai within the camera's field of view, the distance di' of the optical path OAi emitted by the camera to the target point Ai can be calculated by the laser flight time and the speed of light, and then the actual depth of field distance di can be calculated based on the angle θi between the optical path OAi and the z direction, as the actual depth of field distance of the target point Ai measured by the camera.

Limited by the resolution of the ToF module, the exact distance between the target points on the grid plane cannot be accurately measured (such as d3 shown in Figure 5), resulting in low accuracy of the calculated angle, that is, the accuracy of the calibrated depth camera internal parameters is reduced, which in turn leads to low accuracy of the actual depth of field distance calculated for each target point during actual measurement. In addition, the existing camera calibration scheme consists of three calibration steps performed sequentially, which is time-consuming and affects the accuracy of the calibration steps. Different calibration plates need to be prepared for the three calibration steps, which requires more calibration consumables and the calibration steps are more complicated.

Based on this, the present application provides a depth calibration method for calibrating the depth of a ToF module, and the ToF module to be calibrated can be a ToF module in a depth camera. That is to say, the depth calibration method provided in the embodiment of the present application can be used to calibrate the depth of a depth camera of an electronic device, and the electronic device can be a personal computer (PC), a tablet computer, a laptop computer, a portable computer (such as a mobile phone), a wearable electronic device (such as a smart watch), an augmented reality (AR) device, a virtual reality (VR) device, a car computer, and other electronic devices with a depth camera. The following embodiments do not impose any special restrictions on the specific form of the electronic device. The ToF module can mainly include a laser transmitter and a laser receiver.

As shown in Figure 6, it is a schematic diagram of the process flow of the depth calibration method provided in this embodiment. The depth calibration method provided in this application is applied to the depth calibration system, as shown in Figure 7, which is an assembly schematic diagram of the depth calibration system, and Figure 7 also shows the ToF module used for calibration by the depth calibration system. The depth calibration system includes: a mobile component, a calibration plate, a ranging component (not shown in Figure 7) and a controller (not shown in Figure 7).

The moving component of the depth calibration system is used to move the ToF module. During the depth calibration process, the ToF module may have two movement modes: rotation and translation. To achieve the rotation and translation of the ToF module, the moving component may include a motor and a clamping member, the clamping member is used to clamp the ToF module, and the motor is used to drive the clamping member to drive the ToF module to rotate and translate. Among them, the translation of the ToF module can refer to the movement of the ToF module toward the calibration plate along the first direction, and the rotation of the ToF module can refer to the rotation of the ToF module in a plane perpendicular to the first direction, or the rotation of the ToF module in a plane parallel to the calibration plate.

Specifically, the motor may include a driving motor and a rotating shaft, wherein the driving motor is used to drive the rotating shaft to rotate and translate. The clamping member may include a fixing portion and a clamping claw, wherein the fixing portion is used to fix the clamping claw to the rotating shaft of the motor, the clamping claw clamps the ToF module, and the rotating shaft of the motor drives the ToF module through the fixing portion and the clamping claw.

In order to facilitate the description of the relative position relationship of each component of the depth calibration system during assembly, a three-dimensional reference coordinate system is first defined. As shown in Figure 7, the three-dimensional reference coordinate system includes an X-axis, a Y-axis, and a Z-axis, and the three coordinate axes are perpendicular to each other. Among them, the XZ plane is basically parallel to the ground or operating table plane of the calibration environment where the depth calibration system is located, and the mobile component can drive the ToF module to rotate in the XY plane, or drive the ToF module to translate along the Z axis. Among them, in an embodiment of the present application, in the process of the mobile component driving the ToF module to rotate in the XY plane, the coordinates of each point on the laser emission surface of the ToF module in the X-axis and Y-axis of the three-dimensional reference coordinate system change, while the coordinates of each point in the Z axis remain unchanged. In the process of the mobile component driving the ToF module to translate along the Z axis, the coordinates of each point on the laser emission surface of the ToF module in the Z axis of the three-dimensional reference coordinate system change, while the coordinates of each point in the X-axis and Y-axis remain unchanged.

The calibration plate of the depth calibration system is used to receive and reflect the laser emitted by the ToF module. The ToF module includes a laser emitting surface, and the laser emitter and laser receiver of the ToF module are both arranged on the laser emitting surface. The plane where the laser emitting surface of the ToF module is located is recorded as the first plane (as shown in P1 in Figure 7). The calibration plate can be a flat plate with a high surface reflectivity, and there is no need to set a checkerboard on the calibration plate.

The calibration plate is provided with a reflective surface, and the reflective surface of the calibration plate is used to receive the laser emitted by the laser transmitter of the ToF module and reflect the light to the laser receiver of the ToF module. The plane where the reflective surface of the calibration plate is located is recorded as the second plane (P2 as shown in FIG. 7 ).

When calibrating, the first plane where the laser emitting surface of the ToF module is located can be made substantially parallel to the second plane where the laser reflecting surface of the calibration plate is located, and there is a certain distance between the first plane and the second plane. And, for ease of description, the first plane and the second plane can also be made substantially parallel to the XY plane. It should be noted that the two planes mentioned here are substantially parallel or approximately parallel, which can be understood as the angle between the two planes being less than or equal to the preset angle threshold, and the angle threshold can be 0°-5°, without limitation. In the specific implementation, due to factors such as measurement and thickness of the ToF module, it may not be easy to achieve absolute parallelism between the planes, and subsequent calibration operations can be performed with relatively small inclination. Subsequent explanations about parallelism can refer to the explanation about basic parallelism here, and no further explanation is given.

The distance measuring component of the depth calibration system is used to measure the relevant distance data in the depth calibration process, for example, the distance data between two points on the laser reflection surface of the calibration plate, etc. The distance measuring component may include devices such as laser ranging radar and measuring ruler.

The controller of the depth calibration system is used to execute the depth calibration method, such as controlling the mobile component to drive the ToF module to move, and performing relevant calculations based on relevant data to achieve depth calibration of the ToF module. The controller can be connected to the control end and data transmission end of the ToF module's laser transmitter, laser receiver, mobile component, ranging component, etc. to achieve command interaction and data transmission.

Before the depth calibration system performs depth calibration, the mobile component can be placed on the operating platform, and the plane of the operating platform is parallel to the XZ plane of the three-dimensional reference coordinate system. The clamp of the mobile component clamps the ToF module so that the ToF module is basically parallel to the XY plane. The motor shaft of the mobile component can drive the ToF module to rotate in the XY plane or translate along the Z axis through the clamp. During the rotation or translation of the ToF module, the laser emitting surface of the ToF module remains parallel to the XY plane.

The calibration plate is fixed on the operating platform, and there is a certain Z-direction distance between the laser reflection surface of the calibration plate and the laser emission surface of the ToF module, and the second plane where the laser reflection surface of the calibration plate is located is substantially parallel to the first plane where the laser emission plate of the ToF module is located. The rotating shaft of the motor can drive the ToF module to translate along the Z axis through the clamping claw, approaching or moving away from the calibration plate, so as to adjust the Z-direction distance between the laser emission surface of the ToF module and the laser reflection surface of the calibration plate.

In addition, during assembly, the center height of the ToF module can be controlled to be consistent with the center height of the calibration plate as much as possible to reduce the error caused by inconsistent center heights and maximize the depth calibration accuracy. The center height of the ToF module mentioned here can refer to the Y-axis coordinate of the center point of the laser emitting surface of the ToF module in the XYZ coordinate system, and the center height of the calibration plate can refer to the Y-axis coordinate of the center point of the laser reflecting surface of the calibration plate in the XYZ coordinate system.

Specifically, as shown in FIG6 , from the perspective of a controller in a depth calibration system, the provided depth calibration method mainly includes the following steps:

S601: The controller controls the moving component to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is separated from the calibration plate by a first distance; controls the moving component to drive the ToF module to rotate around the center point of the laser emitting surface of the ToF module parallel to the calibration plate, so that the ToF module is in the first spatial posture and the second spatial posture respectively, and controls the ToF module to shoot the calibration plate in the first spatial posture to obtain a first depth map, and to shoot the calibration plate in the second spatial posture to obtain a second depth map.

As shown in Figure 7, the moving component drives the ToF module to move to the first measuring point T1. The laser emitting surface of the ToF module, namely the first plane P1, is parallel to the laser reflecting surface of the calibration plate, namely the second plane P2. The Z-direction distance between the laser emitting surface of the ToF module, namely the first plane P1, and the laser reflecting surface of the calibration plate, namely the second plane P2, is the first distance d1.

The current position of the ToF module at the first measurement point T1 in the XY plane can be recorded as the first spatial position. The ToF module is rotated by a certain angle in the XY plane, and the rotated position is recorded as the second spatial position. The first spatial position is different from the second spatial position.

Alternatively, the controller may also control the mobile component to drive the ToF module to rotate, first rotating the ToF module so that the ToF module is in a first spatial position and collecting a first depth map, and then rotating the ToF module so that the ToF module is in a second spatial position and collecting a second depth map.

When the mobile component drives the ToF module to rotate in the XY plane, the ToF module can be controlled to rotate around the center point of the laser emitting surface of the ToF module. In this way, the Y-axis height of the center point of the laser emitting surface of the ToF module is always consistent with the Y-axis height of the center point of the laser reflecting surface of the calibration plate. The Z-direction distance between the center point of the laser emitting surface of the ToF module and the center point of the laser reflecting surface of the calibration plate is also always consistent. Correspondingly, the depth of field distance of the center calibration point of the calibration plate collected by the center pixel point of the ToF module is also always consistent. Reversely, when the ToF module rotates parallel to the calibration plate, in all pixel areas of the ToF module, if the depth value collected by a certain pixel point remains unchanged, then this pixel point is the center pixel point, and the depth of field distance collected by the center pixel point is the depth of field distance of the center calibration plate of the calibration plate.

Based on this, the ToF module first emits laser to the calibration plate in the first spatial posture at the first measurement point T1 to collect the first depth map corresponding to the calibration plate. The ToF module rotates to the second spatial posture, emits laser to the calibration plate again, and collects the second depth map corresponding to the calibration plate.

The ToF module usually has a standard placement posture. When the ToF module is in the standard placement posture, the lower surface of the ToF module is parallel to the XZ plane, and the laser emitting surface of the ToF module is parallel to the XY plane, and the laser emitting direction of the laser emitting surface is parallel to the Z axis. The parallel mentioned here can also be understood as substantially parallel or approximately parallel.

In one example, as shown in FIG. 7 , the first spatial posture may be the first spatial posture when the ToF module is in a standard placement posture, and the second spatial posture may be the posture after the ToF module is rotated 180° in the XY plane when the ToF module is in the first spatial posture corresponding to the standard placement posture and the second spatial posture after being rotated 180°. The relative parallel state between the ToF module and the calibration plate is more standard, and the measurement error caused by relative tilt is smaller. The depth of field distance of the calibration plate collected by the ToF module is closer to the actual distance between the calibration plate and the ToF module. By minimizing the error caused by relative position as much as possible, the accuracy of depth calibration can be effectively improved.

In other examples, under the premise of ensuring that the first spatial posture and the second spatial posture are different, the first spatial posture and the second spatial posture in which the mobile module drives the ToF module to rotate may also have other implementation schemes, which are not limited.

As shown in Figure 8, the first depth map (as shown in (a) in Figure 8) and the second depth map (as shown in (b) in Figure 8) collected by the ToF module at the first measurement point T1. For ease of description, Figure 8 takes 11*9 pixels as an example, and Figure 8 only shows the depth values of a small number of pixels, and does not limit the specific size and depth value of the depth map.

In Figure 8, there are a first pixel X1 and a second pixel X2 in the first depth map, and there are a third pixel X3 and a fourth pixel X4 in the second depth map. By comparing the depth values of all pixels in the first depth map and the second depth map, if it is determined that the depth value of the first pixel X1 is the same as the depth value of the third pixel X3, then the first pixel X1 and the third pixel X3 are the center pixels. For specific reasons, please refer to the aforementioned explanation about the fact that only the depth value of the center pixel remains unchanged during the rotation of the ToF module. In Figure 8, color blocks of different depths represent different depth values collected by each pixel.

As shown in (a) of FIG. 7 and FIG. 8 , the first pixel point X1 records the depth value c1 of the first calibration point C1 on the calibration plate collected by the ToF module in the first spatial posture, and the second pixel point X2 records the depth value a1 of the second calibration point A1 on the calibration plate.

As shown in (b) of Figure 7 and Figure 8, the third pixel point X3 records the depth value c1 of the first calibration point C1 on the calibration plate collected by the ToF module in the second spatial posture, and the fourth pixel point X4 records the depth value a1' of the third calibration point A1' on the calibration plate.

The ToF module is rotated around the central axis using the line between the central pixel of the ToF module and the central calibration plate of the calibration plate as the central axis. Based on the principle of rotation consistency, during the rotation of the ToF module around the central axis, the distance between a calibration point on the central calibration plate and the central calibration point remains unchanged, and the Z-direction distance between the central calibration point and the central pixel remains unchanged. Therefore, even if the ToF module rotates 180° from the first spatial posture to the second spatial posture, the angle between the optical path between this calibration point and the central pixel and the central axis remains unchanged.

For example, as shown in Figure 7, the angle ∠A1T1C1 between the optical path A1T1 between the second calibration point A1 corresponding to the first spatial posture and the first measuring point T1 and the central axis C1T1 where the first calibration point C1 is located, and the angle ∠A1’T1C1 between the optical path A1’T1 between the third calibration point A1’ corresponding to the second spatial posture and the first measuring point T1 and the central axis C1T1 where the first calibration point C1 is located are equal, that is, ∠A1T1C1=∠A1’T1C1.

S602: The controller controls the moving component to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is spaced a second distance from the calibration plate; controls the moving component to drive the ToF module to rotate around the center point of the laser emitting surface of the ToF module parallel to the calibration plate, so that the ToF module is in a third spatial posture and a fourth spatial posture, respectively, and controls the ToF module to shoot the calibration plate in the third spatial posture to obtain a third depth map, and to shoot the calibration plate in the fourth spatial posture to obtain a fourth depth map;

Continuing as shown in Figure 7, the moving component drives the ToF module to move to the first measuring point T2, the first plane P1 where the laser emitting surface of the ToF module is located is parallel to the second plane P2 where the laser reflecting surface of the calibration plate is located, and the Z-direction distance between the laser emitting surface of the ToF module and the laser reflecting surface of the calibration plate is the second distance d2.

The ToF module is at the second measurement point T2, and the mobile module first drives the ToF module to rotate in the XY plane so that the ToF module is in the third spatial posture. The ToF module emits laser to the calibration plate and collects the third depth map corresponding to the calibration plate. The mobile module then drives the ToF module to rotate in the XY plane so that the ToF module is in the fourth spatial posture. The ToF module emits laser to the calibration plate and collects the fourth depth map corresponding to the calibration plate.

In one example, the third spatial pose is the same as the first spatial pose, and the fourth spatial pose is the same as the second spatial pose.

Corresponding to (a) in FIG. 7 and FIG. 8 , the first pixel records the depth value c2 of the fourth calibration point C2 on the calibration plate, and the second pixel X2 records the depth value a2 of the fifth calibration point A2 on the calibration plate.

As shown in (b) of Figure 7 and Figure 8, the third pixel point X3 records the depth value c2 of the fourth calibration point C2 on the calibration plate collected by the ToF module in the second spatial posture, and the fourth pixel point X4 records the depth value a2' of the sixth calibration point A2' on the calibration plate.

Based on the above analysis on the principle of angle consistency during the rotation of the ToF module, as shown in Figure 7, the angle ∠A2T2C2 between the optical path A2T2 between the fifth calibration point A2 corresponding to the third spatial posture and the second measuring point T2 and the central axis C2T2 where the fourth calibration point C2 is located, and the angle ∠A2’T2C2 between the optical path A2’T2 between the sixth calibration point A2’ corresponding to the fourth spatial posture and the second measuring point T2 and the central axis C2T2 where the fourth calibration point C2 is located are equal, that is, ∠A2T2C2=∠A2’T2C2.

S603: The controller performs depth calibration on the ToF module according to the first depth map, the second depth map, the third depth map, and the fourth depth map, as well as the first distance and the second distance.

The controller controls the ToF module to collect depth maps at two different distances and in different spatial poses, and four depth maps can be obtained. Among them, the Z-direction distances of the first depth map and the second depth map are the same, and the Z-direction distances of the third depth map and the fourth depth map are the same. In addition, the first spatial pose at which the first depth map is collected is the same as the third spatial pose at which the third depth map is collected, and the second spatial pose at which the second depth map is collected is the same as the fourth spatial pose at which the fourth depth map is collected.

The controller can obtain the angle conversion formula and the relative displacement of the pixels based on two depth maps with the same Z distance, and then obtain the internal parameter matrix of the ToF module through multiple calculations and conversions.

In addition, before performing the internal reference calibration operation of S601-S603, the depth calibration system can also perform laser offset correction on the ToF module. The depth calibration system can perform laser offset correction including the following steps:

The controller controls the moving component to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is spaced a third distance from the calibration plate; controls the moving component to drive the ToF module to rotate around the center point of the laser emitting surface of the ToF module parallel to the calibration plate, so that the ToF module is in a fifth spatial posture and a sixth spatial posture, respectively, and controls the ToF module to shoot the calibration plate in the fifth spatial posture to obtain a fifth depth map, and to shoot the calibration plate in the sixth spatial posture to obtain a sixth depth map;

The controller performs laser offset correction on the ToF module according to the fifth depth map and the sixth depth map.

The depth calibration system performs laser offset correction on the ToF module. The controller controls the moving component to drive the ToF module to move to the first measurement point, and rotates the ToF module so that the ToF module is in the fifth spatial posture and the sixth spatial posture, respectively. When the ToF module is in the fifth spatial posture, the laser is emitted to the calibration plate to collect the fifth depth map. When the ToF module is in the sixth spatial posture, the laser is emitted to the calibration plate to collect the sixth depth map. The specific implementation process of the controller controlling the ToF module to collect the fifth depth map and the sixth depth map will not be repeated here.

In the process of driving the ToF module to rotate in the XY plane, the mobile component can control the ToF module to rotate around the center point of the laser emitting surface of the ToF module. In this way, the Y-axis height of the center point of the laser emitting surface of the ToF module is always consistent with the Y-axis height of the center point of the laser reflecting surface of the calibration plate. The Z-direction distance between the center point of the laser emitting surface of the ToF module and the center point of the laser reflecting surface of the calibration plate is also always consistent. Correspondingly, the depth of field distance of the center calibration point of the calibration plate collected by the center pixel point of the ToF module is also always consistent. Reversely, when the ToF module rotates parallel to the calibration plate, if the depth value collected by a certain pixel point in all pixel areas of the ToF module remains unchanged, then this pixel point is the center pixel point. In this way, the controller can obtain the center pixel point of the fifth depth map and the sixth depth map; wherein, the center pixel point is the pixel point with the same depth value in the fifth depth map and the sixth depth map.

The controller obtains a laser recording histogram of a central pixel point; wherein the laser recording histogram includes waveforms of at least two laser signals. The controller calculates the laser offset distance according to the difference between the target time of the waveform of the first laser signal recorded in the laser recording histogram and the statistical 0-point time of the laser recording histogram; wherein the target time of the waveform of the first laser signal includes the peak time. The controller performs laser offset correction on the ToF module according to the laser offset distance.

In a specific example, as shown in (a) of FIG9 , the controller obtains a histogram of the center pixel recorded by the ToF module. The first laser signal waveform corresponding to the center pixel recorded in the histogram has a peak position that is Δ1 away from the statistical 0 point of the histogram, that is, the laser offset distance is Δ1. The actual depth of field distance between the center pixel and the calibration plate is d1. Then, in order to perform laser offset correction on the ToF module, the depth value obtained by the ToF module is superimposed on the laser offset distance to obtain the actual depth of field distance and realize laser offset correction. The corrected value is shown in (b) of FIG9 .

As shown in the example of Figure 9, the horizontal time unit is bin, 1 bin ≈ 500 picoseconds ≈ 5* ^10-10 seconds. The laser flying speed is 3* ¹⁰⁸ m/s, Δ1 = 22*5* ^10-10 seconds/2*3* ¹⁰⁸ m/s ≈ 1.65 m ≈ 165 cm.

For laser offset correction, the position of the histogram signal peak is located at d1+Δ1, and Δ1 is usually reserved for 150 cm-200 cm, for example, Δ1=165 cm.

In other examples, the number of time units corresponding to different situations is different, and may be 10 bins or 30 bins.

The lateral time is 10 bin, Δ1=10*5*10 ^-10 seconds/2*3*10 ⁸ meters/second≈0.75 meters≈75 centimeters.

The lateral time is 30 bin, Δ1=30*5*10 ^-10 sec/2*3*10 ⁸ m/sec≈2.25 m≈225 cm.

That is to say, in different situations or different devices, the value of Δ1 of the laser bias correction will be adjusted accordingly without limitation.

The depth calibration system calculates the laser offset distance according to the above steps, performs laser offset correction, and then executes the internal parameter matrix calibration corresponding to S602-S603. This can reduce the impact of laser offset on the internal parameter matrix calibration and improve the overall accuracy of the depth calibration of the ToF module.

In one example, as shown in FIG. 7 and FIG. 8 , according to the depth values recorded in each depth map, the actual depth of field distance can be superimposed with the laser offset distance to obtain a closer depth of field distance.

For example, at the first measurement point T1, corresponding to the first calibration point C1 on the calibration plate, the depth value recorded in the first depth map is c1, and the offset distance Δd is superimposed, which is recorded as the first depth value c1+Δd. After the ToF module rotates 180°, the depth value recorded in the second depth map is still c1, and the offset distance Δd is superimposed, which is still recorded as the first depth value (c1+Δd).

Among them, the offset distance superimposed on the depth value recorded in the depth map includes two error values, one is the error value Δ1 caused by the laser offset, and the other is the laser offset error value Δ2 that can be corrected in the internal parameter calibration stage. In other words, Δd=Δ1+Δ2. Δ1 is the offset distance calculated in the laser offset correction stage and can be corrected in the laser offset stage. Δ2 is the offset distance that needs to be calculated and corrected through subsequent internal parameter calibration. Before the internal parameter matrix is determined, Δ2 is an unknown parameter, and the specific value of Δ2 will be determined after the internal parameter matrix is calculated.

For the offset distance Δ1 caused by laser offset, the controller can automatically add the laser offset distance Δ1 to the calculation before the internal calibration by adjusting the register of the ToF module. In other words, the depth value c1 in the depth map collected by the subsequent internal calibration has been automatically superimposed with Δ1. It is only necessary to superimpose Δ2 on the depth value c1 and substitute it into the relevant calculation formula of the subsequent internal calibration stage to calculate the unknown parameter Δ2. For the convenience of description, Δd is directly used to replace all offset distances in the future, and the distinguishing processing operations for Δ1 and Δ2 are no longer described in detail. The same applies to the future.

Corresponding to the second calibration point A1 on the calibration plate, the depth recorded in the first depth map is a1, and the offset distance Δd is superimposed, which is recorded as the second depth value a1+Δd. After the ToF module rotates 180°, the depth recorded in the second depth map is a1’, and the offset distance Δd is superimposed, which is recorded as the third depth value (a1’+Δd).

Correspondingly, at the second measurement point T2, corresponding to the fourth calibration point C2 on the calibration board, the depth value recorded in the third depth map is c2, and the offset distance Δd is superimposed, which is recorded as the fourth depth value c2+Δd. After the ToF module rotates 180°, the depth value recorded in the fourth depth map is still c2, and the offset distance Δd is superimposed, which is still recorded as the fifth depth value (c2+Δd).

Corresponding to the 25th fixed point A2 on the calibration plate, the depth value recorded in the third depth map is a2, and the offset distance Δd is superimposed, which is recorded as the fifth depth value a2+Δd. After the ToF module rotates 180°, the depth value recorded in the fourth depth map is a2’, and the offset distance Δd is superimposed, which is recorded as the sixth depth value (a2’+Δd).

The controller performs depth calibration on the ToF module according to the first depth map, the second depth map, the third depth map and the fourth depth map, mainly completing the calibration and offset correction of the intrinsic parameter matrix. In a specific implementation, as shown in FIG10 , the controller performs the step of calibrating the intrinsic parameter matrix of S603, which may mainly include:

S1001: The controller obtains the relative displacement of each pixel of the ToF module and the central pixel, and the angle between the optical path corresponding to each pixel and the first direction according to the first depth map, the second depth map, the third depth map, and the fourth depth map;

Specifically, the controller calculates a first angle conversion formula when the first angle is equal to the second angle according to the first depth map and the second depth map; wherein the calibration plate includes a first calibration point, a second calibration point and a third calibration point, the first calibration point corresponds to a central pixel point of the first depth map and the second depth map, a first optical path between the first calibration point and the first measuring point and a second optical path between the second calibration point and the first measuring point form a first angle, and the first optical path between the first calibration point and the first measuring point and a third optical path between the third calibration point and the first measuring point form a second angle;

The controller calculates a second angle conversion formula when the third angle is equal to the fourth angle according to the third depth map and the fourth depth map; wherein the calibration plate includes a fourth calibration point, a fifth calibration point and a sixth calibration point, the fourth calibration point corresponds to the central pixel point of the third depth map and the fourth depth map, the fourth optical path between the fourth calibration point and the second measuring point and the fifth optical path between the fifth calibration point and the second measuring point form a third angle, and the fourth optical path between the fourth calibration point and the second measuring point and the sixth optical path between the sixth calibration point and the second measuring point form a fourth angle;

According to the depth values corresponding to the first optical path, the second optical path, the third optical path, the fourth optical path, the fifth optical path and the sixth optical path, and the first angle conversion formula and the second angle conversion formula, the relative displacement of each pixel point of the ToF module and the center pixel point, and the angle between the optical path corresponding to each pixel point and the first direction are calculated.

S1002: The controller calculates an internal parameter matrix of the ToF module according to a relative displacement between each pixel of the ToF module and a central pixel, and an angle between an optical path corresponding to each pixel and a first direction;

S1003: The controller performs depth calibration on the ToF module according to the internal parameter matrix of the ToF module.

In one example, referring to the aforementioned Figures 7 and 8, the angle ∠A1T1C1 between the first calibration point C1 and the second calibration point A1, and the angle ∠A1’T1C1 between the first calibration point C1 and the third calibration point A1’ are determined based on the first depth map and the second depth map, and ∠A1T1C1=∠A1’T1C1. The angle ∠A2T2C2 between the fourth calibration point C2 and the fifth calibration point A2, and the angle ∠A2’2C2 between the fourth calibration point C2 and the sixth calibration point A2’ are determined based on the third depth map and the fourth depth map, and ∠A2T2C2=∠A2’T2C2.

Let a0 = (a2 + Δd) - (a1 + Δd), a0' = (a2' + Δd) - (a1' + Δd), c = (c2 + Δd) - (c1 + Δd). The angle between a0 and c is the same as the angle between a0' and c.

In the above overall solution, the relative displacement between the second pixel and the first pixel in the depth map collected by the ToF module is calculated. Based on a similar principle, the relative displacement of all pixels in the depth map collected by the ToF module can be obtained. The main calculation process is as follows:

set up Vector: /> , and/> Vector: /> . Coordinates/> ,/> .

The modulus length can be calculated:

; /> .

Three points That is, with/> With/> Collinear, we can get expression 1:

.

There are two points in the above straight line, namely ,/> Substituting the coordinates of the two points into expression 1, we get expressions 2 and 3 respectively. Among them, expression 2 is:

;

Expression 3 is:

.

make , the above expression 2 + expression 3 can be replaced by:

.

Continue Order ,

Substituting into the formula we get:

.

Taking k as an integer solution, we can get The slope k of the vector is known by the modulus. and/> Coordinates, we can get the slope between the two points as/> .

The k1 obtained in the derivation process represents the tangent value of the angle between the plane and the module. The reverse solution is: θ=arctan(k1).

The angle relationship between each pixel and the central pixel is known. Through the relationship operation of the plane equation, a new plane can be fitted, which is recorded as the third plane, such as P3 shown in (a) of Figure 11. In Figure 11, the third plane is a plane parallel to the first plane where the ToF module is located, and OC is the optical path perpendicular to the third plane. At this time, OC is perpendicular to the third plane, the original a0 is compensated to a3, and a0' is compensated to a3', and a3=a3'. From this, the cosine of the spatial angle α between the second pixel and the first pixel in (b) of Figure 11 can be obtained, that is, cosα=c/a3.

Through the above calculation logic, the cosine of the angle between all pixels and the central pixel can be obtained in the same way.

Continuing as shown in Figure 12, let the coordinates of the point from line segment a to the plane be (ax, ay, az), where ax should be 0, az=c, and the spatial coordinates and pixel coordinates satisfy the following relationship:

,

Derivation shows that: ay/az=(v-cy)/fy.

Substitute all the pixels of the ToF module and their corresponding depth values into the above spatial coordinate conversion formula, and use the least squares method to obtain the optimal intrinsic parameter matrix to complete the intrinsic parameter calibration.

When the ToF module is located at the second measurement point T2, the distance between the ToF module and the calibration plate is the second distance d2. The actual depth values obtained from the histogram are a2+Δ=A2, a2'+Δ=A2', c2+Δ=C2'. According to the results of k and k1 in the third step, the formula is deduced forward. When Δ=A2-a2, the values of k and k1 can be satisfied, and the offset distance Δd can be obtained.

In summary, the depth calibration solution provided in this embodiment does not need to perform three calibration operations separately, which greatly reduces the mechanical cost and site space. Laser offset correction, intrinsic matrix calibration and offset calibration can be completed by collecting six depth maps, which greatly improves the calibration capacity and labor cost.

In addition, the depth calibration method performed by the depth calibration system provided in this embodiment requires that the ToF module to be calibrated is basically parallel to the calibration plate, and angle measurement is achieved through laser, which reduces the difficulty of environment construction and inspection. The low and medium resolution dtof calibration cannot obtain more accurate internal reference information through Zhang Zhengyou's calibration method; even if Zhang Zhengyou's calibration method is used, it is often necessary to take multiple pictures from multiple angles, which is not friendly to environment construction and UPH. Based on the method of depth calibration internal reference, calibration can be completed in a faster way while ensuring accuracy.

The depth calibration method and depth calibration system provided in this embodiment can greatly improve the depth calibration efficiency of laser rangefinders, vehicle-mounted laser radars, AR/VR and other equipment.

In addition, the present application provides a depth calibration system for calibrating a ToF module. The depth calibration system includes a moving component, a calibration board, and a controller.

The controller is used to execute the depth calibration method provided in the above embodiment.

The present application also provides a depth calibration device for calibrating a ToF module. The depth calibration device is connected to a moving component, the ToF module is fixed behind the moving component, and the laser emission surface of the ToF module is parallel to the calibration plate.

The depth calibration device includes a memory and a processor, the memory is coupled to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the depth calibration device executes the depth calibration method provided by the aforementioned embodiment.

The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the depth calibration method provided in the aforementioned embodiment.

The present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the depth calibration method provided in the above embodiment is implemented.

The specific implementation methods of the depth calibration system, depth calibration device, computer-readable storage medium, and computer program product containing instructions provided in the embodiments of the present application and the technical effects brought about by them can be found in the specific implementation process of the depth calibration method provided in the aforementioned embodiments and the technical effects brought about by them, which will not be repeated here.

In some embodiments, through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

The above are only specific implementations of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiments of the present application should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.

Claims (12)

1. A depth calibration method, characterized in that it is used for calibrating a ToF module in a depth calibration system; the depth calibration system comprises a mobile component, a calibration plate and a controller; after the ToF module is fixed to the mobile component, the laser emission surface of the ToF module is parallel to the calibration plate; the depth calibration method comprises: The controller controls the moving component to drive the ToF module to move along a first direction, so that the laser emitting surface of the ToF module is spaced a first distance from the calibration plate; controls the moving component to drive the ToF module to rotate around the center point of the laser emitting surface of the ToF module parallel to the calibration plate, so that the ToF module is in a first spatial posture and a second spatial posture, respectively, and controls the ToF module to photograph the calibration plate in the first spatial posture to obtain a first depth map, and to photograph the calibration plate in the second spatial posture to obtain a second depth map; The controller controls the moving component to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is spaced a second distance from the calibration plate; controls the moving component to drive the ToF module to rotate around the center point of the laser emitting surface of the ToF module parallel to the calibration plate, so that the ToF module is in a third spatial posture and a fourth spatial posture, respectively, and controls the ToF module to photograph the calibration plate in the third spatial posture to obtain a third depth map, and to photograph the calibration plate in the fourth spatial posture to obtain a fourth depth map; The controller performs depth calibration on the ToF module according to the first depth map, the second depth map, the third depth map and the fourth depth map, as well as the first distance and the second distance; wherein the first direction is perpendicular to the calibration plate, the first distance is different from the second distance; the first spatial posture is different from the second spatial posture, and the third spatial posture is different from the fourth spatial posture. 2. The depth calibration method according to claim 1 is characterized in that the first spatial pose is the same as the third spatial pose, and the second spatial pose is the same as the fourth spatial pose. 3. The depth calibration method according to claim 1 or 2, characterized in that the ToF module is rotated 180 degrees parallel to the calibration plate from the first spatial posture around the center point of the laser emission surface of the ToF module to the second spatial posture; The ToF module is rotated 180 degrees from the third spatial posture around the center point of the laser emitting surface of the ToF module parallel to the calibration plate to be in the fourth spatial posture. 4. The depth calibration method according to claim 1 or 2, characterized in that the laser emission surface of the ToF module is parallel to the calibration plate, comprising: The angle between the laser emitting surface of the ToF module and the calibration plate is smaller than a preset angle threshold. 5. The depth calibration method according to claim 1 is characterized in that the controlling the moving component drives the ToF module to rotate around the center point of the laser emission surface of the ToF module parallel to the calibration plate, so that the ToF module is in a first spatial posture and a second spatial posture respectively, and the ToF module is controlled to shoot the calibration plate in the first spatial posture to obtain a first depth map. Before the step of shooting the calibration plate in the second spatial posture to obtain a second depth map, the depth calibration method further includes: The controller controls the moving component to drive the ToF module to move along the first direction, so that the laser emitting surface of the ToF module is spaced a third distance from the calibration plate; controls the moving component to drive the ToF module to rotate around the center point of the laser emitting surface of the ToF module parallel to the calibration plate, so that the ToF module is in a fifth spatial posture and a sixth spatial posture, respectively, and controls the ToF module to photograph the calibration plate in the fifth spatial posture to obtain a fifth depth map, and to photograph the calibration plate in the sixth spatial posture to obtain a sixth depth map; the fifth spatial posture and the sixth spatial posture are different; The controller performs laser offset correction on the ToF module according to the fifth depth map and the sixth depth map. 6. The depth calibration method according to claim 5, characterized in that the controller performs a step of performing laser offset correction on the ToF module according to the fifth depth map and the sixth depth map, comprising: The controller obtains a central pixel point of the fifth depth map and the sixth depth map; wherein the central pixel point is a pixel point with the same depth value in the fifth depth map and the sixth depth map; The controller acquires a laser recording histogram of the central pixel point; wherein the laser recording histogram includes at least two waveform diagrams of laser signals; The controller calculates the laser offset distance according to the difference between the target time of the waveform of the first laser signal recorded in the laser recording histogram and the statistical 0 point time of the laser recording histogram; wherein the target time of the waveform of the first laser signal includes the peak time; The controller performs laser offset correction on the ToF module according to the laser offset distance. 7. The depth calibration method according to any one of claims 1, 2, 5 or 6, characterized in that the controller performs a step of depth calibration on the ToF module according to the first depth map, the second depth map, the third depth map and the fourth depth map, and the first distance and the second distance, comprising: The controller obtains the relative displacement of each pixel point of the ToF module and the central pixel point, and the angle between the optical path corresponding to each pixel point and the first direction according to the first depth map, the second depth map, the third depth map, and the fourth depth map; The controller calculates the intrinsic parameter matrix of the ToF module according to the relative displacement between each pixel of the ToF module and the central pixel, and the angle between the optical path corresponding to each pixel and the first direction; The controller performs depth calibration on the ToF module according to the internal parameter matrix of the ToF module. 8. The depth calibration method according to claim 7, characterized in that the controller obtains the relative displacement of each pixel point and the central pixel point of the ToF module and the angle between the optical path corresponding to each pixel point and the first direction according to the first depth map, the second depth map, the third depth map and the fourth depth map, comprising: The controller calculates a first angle conversion formula when the first angle is equal to the second angle according to the first depth map and the second depth map; wherein the calibration plate includes a first calibration point, a second calibration point and a third calibration point, the first calibration point corresponds to a central pixel point of the first depth map and the second depth map, a first optical path between the first calibration point and the first measuring point and a second optical path between the second calibration point and the first measuring point form the first angle, and a first optical path between the first calibration point and the first measuring point and a third optical path between the third calibration point and the first measuring point form the second angle; The controller calculates a second angle conversion formula when the third angle is equal to the fourth angle according to the third depth map and the fourth depth map; wherein the calibration plate includes a fourth calibration point, a fifth calibration point and a sixth calibration point, the fourth calibration point corresponds to a central pixel point of the third depth map and the fourth depth map, a fourth optical path between the fourth calibration point and the second measuring point and a fifth optical path between the fifth calibration point and the second measuring point form the third angle, and a fourth optical path between the fourth calibration point and the second measuring point and a sixth optical path between the sixth calibration point and the second measuring point form the fourth angle; According to the depth values corresponding to the first optical path, the second optical path, the third optical path, the fourth optical path, the fifth optical path and the sixth optical path, and the first angle conversion formula and the second angle conversion formula, the relative displacement of each pixel point and the center pixel point of the ToF module, and the angle between the optical path corresponding to each pixel point and the first direction are calculated. 9. A depth calibration system, characterized in that it is used to calibrate a ToF module, and the depth calibration system includes a moving component, a calibration board and a controller; The controller is used to execute the depth calibration method according to any one of claims 1 to 8. 10. A depth calibration device, characterized in that it is used to calibrate a ToF module, the depth calibration device is connected to a moving component, after the ToF module is fixed to the moving component, the laser emitting surface of the ToF module is parallel to the calibration plate; The depth calibration device includes a memory and a processor, the memory is coupled to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the depth calibration device performs the depth calibration method described in any one of claims 1 to 8. 11. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer is enabled to execute the depth calibration method according to any one of claims 1 to 8. 12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the depth calibration method according to any one of claims 1 to 8 is implemented.