CN117083533A

CN117083533A - Method, device, computing equipment and readable storage medium for acquiring depth map

Info

Publication number: CN117083533A
Application number: CN202180096494.6A
Authority: CN
Inventors: 罗鹏飞; 董晨; 周鸿彬; 唐样洋
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-03-29
Filing date: 2021-03-29
Publication date: 2023-11-17
Also published as: WO2022204895A1

Abstract

The application provides a method, a device, computing equipment and a readable storage medium for acquiring a depth map, and belongs to the technical field of images. The method comprises the following steps: performing downsampling processing on bare data of a plurality of modulation frequencies of a measured object to obtain a depth map of a first resolution, wherein the bare data of each modulation frequency is obtained by exposing the measured object for a plurality of times under each modulation frequency when a TOF system measures distance, the depth map of the first resolution is a depth map of which the pixels are judged to have no abnormal unwrapping coefficient, the resolution of a pixel array of the TOF system is a second resolution, the first resolution is lower than the second resolution, and the unwrapping coefficients of the pixels of the second resolution are obtained by using the depth map of the first resolution and the phase delay values of the pixels of the second resolution; and obtaining a target depth map of the second resolution corresponding to the measured object by using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution. By adopting the method and the device, the accuracy of the depth map is improved.

Description

Method, device, computing equipment and readable storage medium for acquiring depth map

Technical Field

The present application relates to the field of image technologies, and in particular, to a method, an apparatus, a computing device, and a readable storage medium for acquiring a depth map.

Background

At present, the distance is often measured by a time of flight (TOF) imaging technology, in the TOF imaging technology, a light source in a TOF three-dimensional imaging system (for short, a TOF system) emits light to irradiate a target object, the light reflected by the target object returns to a TOF chip (i.e., an image sensor) in the TOF three-dimensional imaging system, each pixel in a pixel array of the TOF chip can receive an optical signal, and the distance between each pixel in the pixel array and the target object is determined based on a phase difference between the emitted optical signal and the received optical signal, so as to obtain a depth map.

Since the phases overlap every 2 pi, measuring the distance using only one modulation frequency of light may result in inaccurate measured distance, and thus measuring the distance using light of multiple modulation frequencies may be used. For example, when measuring a distance using light of two modulation frequencies, for each pixel in a pixel array, a phase delay value corresponding to the pixel at the two modulation frequencies, respectively, is determined, and a blur distance corresponding to each modulation frequency is determined, and then a unwrapping coefficient corresponding to the two modulation frequencies at the pixel is determined using the phase delay value at the pixel. The phase delay value, the blur distance and the unwrapping coefficient are used to determine the distance measured at both modulation frequencies.

Since the TOF system has noise interference, the phase delay value is affected by noise and fluctuates around an ideal value. When the phase delay value is subjected to larger interference fluctuation, the phase delay value is inaccurate, so that the unwrapping coefficient is inaccurate, and the obtained depth map is inaccurate.

Disclosure of Invention

The embodiment of the application provides a method, a device, computing equipment and a readable storage medium for acquiring a depth map, which can acquire an accurate depth map.

In a first aspect, the present application provides a method for obtaining a depth map, the method comprising: performing downsampling processing on bare (raw) data of a plurality of modulation frequencies of a measured object to obtain a depth map with a first resolution, wherein the raw data of each modulation frequency is obtained by multiple exposure at each modulation frequency when a TOF system measures distance of the measured object, the depth map with the first resolution is a depth map with pixels determined to have no abnormal unwrapping coefficient, and the resolution of a pixel array in the TOF system is a second resolution, wherein the first resolution is lower than the second resolution; calculating an unwrapping coefficient of each pixel of a second resolution by using the depth map of the first resolution and the phase delay value of each pixel of the second resolution, wherein the phase delay value of each pixel of the second resolution is calculated by using raw data of a plurality of modulation frequencies; and calculating to obtain a target depth map of the second resolution corresponding to the measured object by using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution.

Wherein the object to be measured is any object, such as an automobile. The modulation frequency is a frequency used for intensity-modulating an optical signal of a certain frequency. The phase delay value of each pixel of the second resolution is plural, and is a phase delay value of each pixel at each modulation frequency. The disentanglement coefficient of each pixel of the second resolution is a plurality of, respectively, the disentanglement coefficient of each pixel at each modulation frequency. The abnormal unwrapping coefficient may also be referred to as an erroneous unwrapping coefficient.

According to the scheme, an execution main body of the method for acquiring the depth map can be a distance measuring device, and when the object to be measured is measured, the TOF system is subjected to multiple exposure under any modulation frequency to obtain raw data of the modulation frequency. And then, carrying out downsampling processing on the raw data with a plurality of modulation frequencies to obtain a depth map with first resolution, wherein each pixel in the depth map is judged as a pixel without an abnormal unwrapping coefficient. Then, using the depth map of the first resolution and the phase delay value of each pixel of the second resolution, the unwrapping coefficient of each pixel of the second resolution is calculated. And calculating to obtain a target depth map of the second resolution corresponding to the measured object by using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution. In this way, since the downsampled depth map of the first resolution does not have the pixels with abnormal unwrapping coefficients, and the unwrapping coefficients of the second resolution use the depth map of the first resolution in calculation, the target depth map of the second resolution calculated by using the unwrapping coefficients is also more accurate, and the accuracy of the target depth map of the second resolution is ensured.

In one possible implementation manner, performing downsampling processing on raw data of multiple modulation frequencies of a measured object to obtain a depth map with a first resolution, where the downsampling processing includes: performing downsampling processing on the raw data of a plurality of modulation frequencies of the measured object by using the first downsampling parameters to obtain a gray level map of target resolution and a depth map of target resolution corresponding to the raw data of each modulation frequency; if no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, determining the depth map with the target resolution as the depth map with the first resolution; if at least one gray level image has pixels with gray values smaller than a first threshold corresponding to the modulation frequency, updating a first downsampling parameter, and downsampling the raw data of a plurality of modulation frequencies based on the updated downsampling parameter until a depth image with a first resolution is obtained.

The first downsampling parameter is a parameter for reducing resolution, for example, the first downsampling parameter is 2×2,2×2 represents 4, and the original 2×2 pixels are combined into one pixel during processing. The first downsampling parameter may also be referred to as a first binning (binning) parameter, and the downsampling process may also be referred to as a binning process.

According to the scheme disclosed by the application, the ranging device uses the first downsampling parameter to downsample the raw data of each modulation frequency, so that a gray level map of a target resolution corresponding to each modulation frequency is obtained, and a depth map of the target resolution can be obtained. The target resolution is lower than the second resolution. And judging whether pixels with gray values smaller than a first threshold corresponding to the modulation frequency exist in the gray map of each target resolution, and if not, determining the depth map of the target resolution as the gray map of the first resolution. If at least one gray level map has pixels with gray values smaller than the first threshold corresponding to the modulation frequency, the first downsampling parameter is updated, so that the first downsampling parameter is enlarged, that is, more pixels are combined into one pixel. And re-performing downsampling processing on the raw data of the plurality of modulation frequencies by using the updated downsampling parameters, and obtaining a plurality of gray maps and a depth map again, wherein the resolution of the gray maps is lower than the target resolution. Judging whether pixels with gray values smaller than a first threshold corresponding to the modulation frequency exist in the gray maps, if not, determining the depth map as the depth map with the first resolution, and if at least one pixel with gray values smaller than the first threshold corresponding to the modulation frequency exists in the gray map, continuing updating the downsampling parameters, and executing the processing until the depth map with the first resolution is obtained.

Thus, a depth map in which no abnormal unwrapping coefficient exists for the pixel can be obtained.

In one possible implementation manner, performing downsampling processing on raw data of multiple modulation frequencies of a measured object to obtain a depth map with a first resolution, where the downsampling processing includes: performing downsampling processing on the raw data of a plurality of modulation frequencies of the measured object by using the first downsampling parameters to obtain a gray scale map of target resolution corresponding to the raw data of each modulation frequency; if no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, downsampling processing is carried out on raw data of a plurality of modulation frequencies by using a first downsampling parameter, so that a depth map with a first resolution is obtained; if at least one gray level image has pixels with gray values smaller than a first threshold corresponding to the modulation frequency, updating a first downsampling parameter, and downsampling the raw data of a plurality of modulation frequencies based on the updated downsampling parameter until a depth image with a first resolution is obtained.

According to the scheme, the ranging device uses the first downsampling parameter to downsample the raw data of each modulation frequency, and a gray scale map of target resolution corresponding to each modulation frequency is obtained. The target resolution is lower than the second resolution. And judging whether pixels with gray values smaller than a first threshold corresponding to the modulation frequency exist in the gray map of each target resolution, and if not, performing downsampling processing on raw data corresponding to a plurality of modulation frequencies by using a first downsampling parameter to obtain the gray map of the first resolution. If at least one gray level map has pixels with gray values smaller than the first threshold corresponding to the modulation frequency, the first downsampling parameter is updated, so that the first downsampling parameter is enlarged, that is, more pixels are combined into one pixel. And re-performing downsampling processing on the raw data of the plurality of modulation frequencies by using the updated downsampling parameters, and obtaining a plurality of gray maps again, wherein the resolution of the plurality of gray maps is lower than the target resolution. Judging whether pixels with gray values smaller than a first threshold corresponding to the modulation frequency exist in the plurality of gray maps, if not, performing downsampling processing on raw data corresponding to the plurality of modulation frequencies by using updated downsampling parameters to obtain a depth map with first resolution, and if pixels with gray values smaller than the first threshold corresponding to the modulation frequency exist in at least one gray map, continuing updating the downsampling parameters, and executing the processing until the depth map with the first resolution is obtained.

In one possible implementation, the first threshold value for each of the modulation frequencies is the lowest signal reception intensity of unwrapping error in a corresponding distance noise curve obtained in the TOF system.

After the TOF system determines, each modulation frequency corresponds to a distance noise curve, the distance noise curve is drawn on a rectangular coordinate system, the horizontal axis is the intensity of the received optical signal, and the vertical axis is the standard deviation of the depth values of the measured object measuring the same distance for multiple times under a certain optical signal intensity. When the standard deviation of the depth value meets a certain condition, the corresponding signal receiving intensity is the lowest signal receiving intensity without unwrapping errors.

In one possible implementation manner, performing downsampling processing on raw data of multiple modulation frequencies of a measured object to obtain a depth map with a first resolution, where the downsampling processing includes: using the raw data of the plurality of modulation frequencies and the phase delay diagrams of the plurality of modulation frequencies to obtain unwrapping coefficients of each pixel in the initial depth diagram of the second resolution; calculating to obtain the depth value of each pixel in the initial depth map by using the unwrapping coefficient of each pixel in the initial depth map and the phase delay value of each pixel of the second resolution; and performing downsampling processing on the initial depth map to obtain a depth map with a first resolution.

Wherein the phase delay map is a map of correspondence of phase delay values with unwrapping coefficients, in which the phase delay values can be used to correspond to the unwrapping coefficients.

According to the scheme, the ranging device can substitute the raw data of each modulation frequency into a phase delay value calculation formula to determine the phase delay value of each pixel with the second resolution at each modulation frequency. And then for any pixel, determining a line segment closest to a position point where the plurality of phase delay values are positioned in the phase delay diagram by using the plurality of phase delay values of the pixel, determining an unwrapping coefficient corresponding to a sequence number of the line segment, and determining the unwrapping coefficient as the unwrapping coefficient corresponding to the pixel under a plurality of frequencies. Then for pixel k of the second resolution, N will be _kj 、 And blur distance U _j Substitution formulaObtaining depth value D of pixel k at modulation frequency j _kj ，N _kj For the corresponding unwrapping factor at the modulation frequency j pixel k,is the pixel k phase delay value at the modulation frequency j. In this way, a depth value at the modulation frequency j is obtained for each pixel. The depth value of each pixel at a plurality of modulation frequencies is averaged, and the average value corresponding to each pixel is determined as the depth value of each pixel of the second resolution. The depth values of each pixel of the second resolution are assembled into an initial depth map of the second resolution. And then, carrying out downsampling processing on the initial depth map with the second resolution to obtain a depth map with the first resolution, wherein the first resolution is lower than the second resolution.

In this way, a depth map of the first resolution may be acquired.

In one possible implementation, the downsampling process is performed on the initial depth map to obtain a depth map of a first resolution, including: performing downsampling processing on the initial depth map by using the second downsampling parameter to obtain a depth map with a third resolution; determining a gradient map corresponding to the depth map with the third resolution; if the gradient map meets the gradient condition, determining that the depth map with the third resolution is the depth map with the first resolution; if the gradient map does not meet the gradient conditions, updating a second downsampling parameter, and downsampling the initial depth map based on the updated downsampling parameter until a target gradient map meeting the gradient conditions is obtained, and determining the depth map of the obtained target gradient map as a depth map with a second resolution; wherein the gradient condition is that the absolute value of the gradient of the pixel with the distance interval smaller than the second threshold value is smaller than or equal to the third threshold value.

The second downsampling parameter is a parameter for reducing resolution, for example, the second downsampling parameter is 2×2,2×2 represents 4, and the original 2×2 pixels are combined into one pixel during processing. The second downsampling parameter may also be referred to as a second binning parameter, and the downsampling process may also be referred to as a binning process.

According to the scheme disclosed by the application, the distance measuring device can divide the initial depth map with the second resolution into a plurality of pixel combinations by using the second downsampling parameter, and the number of pixels in each pixel combination is the same. And averaging the depth values of the pixels in each pixel combination to obtain a depth value when the pixels in each pixel combination are combined into one pixel, and combining the combined depth values to form a depth map with a third resolution.

And then determining a gradient map corresponding to the depth map with the third resolution, judging the size relation between the absolute gradient value of the pixels with the distance interval smaller than the second threshold value in the gradient map and the third threshold value by the distance measuring device, and determining that the gradient map meets the gradient condition if the absolute gradient value of the pixels with the distance interval smaller than the second threshold value in the gradient map is smaller than or equal to the third threshold value, wherein the depth map with the third resolution is the depth map with the first resolution, that is to say, the third resolution is equal to the first resolution. Otherwise, the gradient map is determined to not meet the gradient condition, and the second downsampling parameter can be updated by using a preset rule on the basis of the second downsampling parameter. And performing downsampling processing on the initial depth map with the second resolution by using the updated downsampling parameters to obtain an updated gradient map, judging whether the updated gradient map meets the gradient conditions, and if so, determining that the depth map with the first resolution is the depth map with the updated gradient map. If the updated gradient map does not meet the gradient conditions, continuously updating the gradient map on the basis of the downsampling parameters updated last time until a target gradient map meeting the gradient conditions is obtained. The depth map from which the target gradient map is obtained is determined as the depth map of the first resolution.

In this way, when there is a sudden change in the gradient of a pixel, it is indicated that the absolute value of the gradient is relatively large, the gradient is also related to the depth value error, and for any pixel, when the correct unwrapping coefficient of the pixel at a certain modulation frequency can be obtained, the depth value error is usually smaller than a certain value, so that the gradient condition can be used to obtain the depth map of the first resolution.

In one possible implementation, calculating the unwrapping coefficient for each pixel of the second resolution using the depth map of the first resolution and the phase delay value for each pixel of the second resolution includes: determining modulation frequencyThe unwrapping coefficient for pixel k of the second resolution of j is:wherein the modulation frequency j belongs to a plurality of modulation frequencies, round [ []To round-off operation, D _L For a depth value of pixel k corresponding in a depth map of a first resolution, N _kj For the unwrap coefficient at the modulation frequency j pixel k,u is the phase delay value of pixel k at modulation frequency j _j The ambiguity distance corresponding to the modulation frequency j.

Wherein, since the first resolution is lower than the second resolution and the pixels of the first resolution are formed by the plurality of pixel downsampling processes at the second resolution, when calculating the unwrapping coefficient, the pixels k of the second resolution correspond to the depth value D in the depth map of the first resolution _L A depth value of a pixel formed for a plurality of pixels including a pixel k.

In one possible implementation manner, calculating to obtain the target depth map of the second resolution corresponding to the measured object using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution includes: calculating a depth value of each pixel of the second resolution for each modulation frequency using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution; determining a depth value error for each pixel of the second resolution for each modulation frequency using the depth map of the first resolution and the depth values for each pixel of the second resolution for each modulation frequency; and calculating and obtaining a target depth map of a second resolution corresponding to the measured object by using the depth value error of each pixel of the first resolution of each modulation frequency.

In the scheme shown in the application, for the modulation frequency j in the plurality of modulation frequencies and the pixel k with the second resolution, the distance measuring device can make N _kj 、 And blur distance U _j Substitution formulaObtaining depth value D of pixel k at modulation frequency j _kj ，N _kj For the corresponding unwrapping factor at the modulation frequency j pixel k,is the pixel k phase delay value at the modulation frequency j. In this way, the depth value of each pixel at each modulation frequency can be determined.

The ranging device may then determine a depth value error for each pixel of the second resolution for each modulation frequency using the depth map of the first resolution and the depth values for each pixel of the second resolution for each modulation frequency. The ranging device may determine a target depth map of the second resolution corresponding to the object under test using the depth value errors of the pixels of the second resolution for each modulation frequency.

In this way, since the depth value error is taken into consideration, the accuracy of the target depth map can be improved.

In one possible implementation, determining a depth value error for each pixel of the second resolution for each modulation frequency using the depth map of the first resolution and the depth values for each pixel of the second resolution for each modulation frequency, includes: the depth value error of the pixel k of the second resolution determining the modulation frequency j is: e (E) _kj ＝|D _L -D _kj I (I); wherein D is _L For the corresponding depth value of pixel k in the depth map of the first resolution,D _kj for adjustingThe depth value of pixel k at the second resolution of frequency j,to the phase delay value of pixel k at modulation frequency j, N _kj U is the unwrapping coefficient of pixel k at modulation frequency j _j For the ambiguity distance corresponding to the modulation frequency j, j takes a value of 1 to n, where n is the number of the plurality of modulation frequencies.

In one possible implementation, using the depth value error of each pixel of the first resolution of each modulation frequency, a target depth map of the second resolution corresponding to the measured object is obtained by calculation, including: for a modulation frequency j in the plurality of modulation frequencies, determining a target pixel with a depth value error larger than a fourth threshold value corresponding to the modulation frequency j in pixels with a second resolution of the modulation frequency j, wherein the value of j is 1 to n, and n is the number of the plurality of modulation frequencies; updating the unwrapping coefficient of the target pixel with the second resolution of the modulation frequency j by using the depth value of the adjacent pixel of the target pixel in the depth map with the first resolution; updating the depth value of the target pixel of the second resolution of the modulation frequency j by using the updated unwrapping coefficient of the target pixel and the phase delay value of the target pixel of each modulation frequency; and combining the depth values of the pixels except the target pixel in the pixels with the second resolution of the modulation frequency j with the updated depth values of the target pixel to obtain a target depth map with the second resolution of the modulation frequency j, which corresponds to the measured object.

According to the scheme, for the modulation frequency j, the distance measuring device can acquire a fourth threshold value corresponding to the modulation frequency j, then the distance measuring device can judge whether the depth value error in the pixel with the second resolution of the modulation frequency j is larger than the fourth threshold value corresponding to the modulation frequency j, and in the depth map with the second resolution of the modulation frequency j, a target pixel with the depth value error larger than the fourth threshold value is obtained. Then, in the depth map of the first resolution, depth values of neighboring pixels of the target pixel are determined.

Assuming that the target pixel is a pixel B, for any adjacent pixel P in the depth map of the target pixel in the first resolution, the distance measuring device calculates and obtains a unwrapping coefficient corresponding to the target pixel in the second resolution at the modulation frequency j by using the depth value of the adjacent pixel P, the phase delay value corresponding to the target pixel in the second resolution at the modulation frequency j and the blur distance corresponding to the modulation frequency j.

Then the distance measuring device calculates and obtains a depth value of the target pixel with a second resolution at the modulation frequency j by using the unwrapping coefficient corresponding to the target pixel with the modulation frequency j, the phase delay value corresponding to the target pixel with the modulation frequency j and the fuzzy distance corresponding to the modulation frequency j. The depth value error of the target pixel calculated at this time is determined. Based on the above manner, each adjacent pixel of the depth map of the target pixel at the first resolution is used to determine the depth value error of the target pixel corresponding to each adjacent pixel. The distance measuring device determines the depth value of the target pixel at the modulation frequency j as the depth value when the depth value error is minimum and is smaller than the depth value of the fourth threshold corresponding to the modulation frequency j. And then the distance measuring device combines the depth values of the pixels except the target pixel in the pixels with the second resolution of the modulation frequency j with the updated depth values of the target pixel to obtain a target depth map with the second resolution of the measured object with the modulation frequency j. A target depth map of the second resolution of the object under test at each modulation frequency is determined based on the manner in which the target depth map of the second resolution of the object under test at modulation frequency j is determined.

In this way, since the correct depth value of the adjacent pixels can be used to calculate the depth value of each pixel of the region with the wrong depth value after the region with the wrong depth value is obtained, it can be ensured that the depth values of all the pixels can be calculated correctly, and the correct depth map of the original resolution is obtained.

In one possible implementation, the fourth threshold corresponding to the modulation frequency j is: thD _j ＝U _j *ph _ith /(2π*2)；U _j To modulate the fuzzy distance corresponding to the frequency j, ph _ith For the minimum phase distance between adjacent line segments of unwrapped regions in a phase delay diagram of a plurality of modulation frequencies, j has a value of 1 to n, n being a plurality of modulation frequenciesNumber of frequencies is produced.

If the error of the phase delay value exceeds one half of the minimum phase distance, the unwrapping coefficient of the neighboring line segment may be determined, and the number of unwrapping errors may be determined.

In a second aspect, the present application provides an apparatus for acquiring a depth map, where the apparatus includes one or more modules configured to implement the method for acquiring a depth map according to the first aspect.

In a third aspect, the present application provides a computing device for acquiring a depth map, the computing device comprising a processor and a memory, wherein: the memory stores computer instructions; the processor executes the computer instructions to implement the method of the first aspect and possible implementations thereof.

In a fourth aspect, the present application provides a computer readable storage medium storing computer instructions that, when executed by a computing device, cause the computing device to perform the method of the first aspect and possible implementations thereof, or cause the computing device to implement the functions of the apparatus of the second aspect and possible implementations thereof.

In a fifth aspect, the application provides a computer program product comprising instructions which, when run on a computing device, cause the computing device to perform the method of the first aspect and possible implementations thereof.

Drawings

FIG. 1 is a graph of phase delay values versus distance provided by an exemplary embodiment of the present application;

FIG. 2 is a schematic diagram of unwrapping coefficients in a phase delay value graph provided in accordance with an exemplary embodiment of the present application;

FIG. 3 is a graph of phase delay values versus unwrapping coefficients provided by an exemplary embodiment of the present application;

FIG. 4 is a schematic diagram of a TOF system according to an exemplary embodiment of the present application;

FIG. 5 is a flow chart of a method for obtaining a depth map according to an exemplary embodiment of the present application;

FIG. 6 is a schematic diagram of downsampling provided by an exemplary embodiment of the application;

FIG. 7 is a graph of received optical signal strength versus distance noise provided by an exemplary embodiment of the present application;

FIG. 8 is a schematic diagram of downsampling provided by an exemplary embodiment of the application;

FIG. 9 is a flow chart of a method for obtaining a depth map according to an exemplary embodiment of the present application;

FIG. 10 is a flow chart of a method for obtaining a depth map according to an exemplary embodiment of the present application;

FIG. 11 is a schematic structural diagram of an apparatus for acquiring a depth map according to an exemplary embodiment of the present application;

FIG. 12 is a schematic diagram of a computing device provided in an exemplary embodiment of the application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.

To facilitate an understanding of embodiments of the present application, the concepts of the terms involved are first described below:

TOF, time of flight refers to round trip time, which is the round trip time of an optical signal sent by a transmitter in a TOF system between a measured object and a receiver in the TOF system, and a depth map can be calculated through the round trip time.

The depth map comprises a plurality of pixels, wherein the pixel value of each pixel is used for recording the distance between the TOF system and a certain point of the measured object, namely the pixel value of each pixel is the depth value.

The blur distance, expressed as the maximum measurable distance for a single modulation frequency TOF system, is formulated as:

in the expression (1), f represents a modulation frequency, u represents a blur distance corresponding to the modulation frequency f, and c represents a light velocity. The modulation frequency is a frequency used for intensity-modulating an optical signal of a certain frequency.

The beat frequency (beat frequency) is a frequency corresponding to a distance obtained by commonly measuring a plurality of modulation frequencies when the distance is measured using the plurality of modulation frequencies. The striking frequency is the greatest common divisor of the plurality of modulation frequencies. The hit frequency is relatively low with respect to the plurality of modulation frequencies, allowing for a longer measurement distance to be extended. For example, as shown in FIG. 1, the relationship between the phase delay value and the distance of two modulation frequencies within 7.5m of 80MHz and 100MHz is shown, and the blurring distance is u at the modulation frequency of 100MHz _100MHz =1.5m, at a modulation frequency of 80MHz, the ambiguity distance is u _80MHz When using two modulation frequencies, 80MHz and 100MHz, =1.875m, the blur distance is 7.5m. The maximum measurement distance is illustrated as being extended to 7.5m, where the unit "m" means "meter".

In the related art, when two modulation frequencies are used to obtain a depth map, as shown in fig. 2, after two modulation frequencies of 80MHz and 100MHz are used, the blurring distance is extended to 7.5m, and if there is no noise in the TOF system, the two modulation frequency phase delay values of each pixel are within the blurring distanceIn TOF systems of 80MHz and 100MHz, there are 8 unwrapped regions, only odd-numbered unwrapped regions being labeled in fig. 2, in one-to-one correspondence with distance. As shown in table one, the unwrapping coefficients for the 8-segment unwrapping regions of the TOF systems of 80MHz and 100MHz are given.

List one

Sequence number	Disentanglement coefficient N1 (80 MHz)	Disentanglement coefficient N2 (100 MHz)
①	0	0
②	0	1
③	1	1
④	1	2
⑤	2	2
⑥	2	3
⑦	3	3
⑧	3	4

For any one pixel, the unwrapping coefficient (N ₁ ，N ₂ ) And a phase delay valueThen, the distance (D) obtained by measurement at two modulation frequencies is calculated using the formulas (2) and (3) ₁ ，D ₂ )：

It can be seen that for the calculation (D ₁ ，D ₂ ) First, it is necessary to calculate and obtain the unwrapping coefficient (N ₁ ，N ₂ ). In the related art, as shown in fig. 3, the phase delay values corresponding to two modulation frequencies are plotted in an xy coordinate system (x-o-y), and in fig. 3, a segment of an 8-segment unwrapping region of a TOF system of 80MHz and 100MHz is given without noise interference, the horizontal axis represents the phase delay value of 80MHz, and the vertical axis represents the phase delay value of 100 MHz. After calculating the unwrapping coefficient (N ₁ ，N ₂ ) In this case, the position point of the phase delay value corresponding to the pixel in fig. 3 is determined, and the line segment ((3) located line segment) closest to the position point is determined, which may be referred to as the closest line segment. In table one, determining the unwrapping coefficient corresponding to the sequence number of the nearest line segment, and determining the unwrapping coefficient as the unwrapping coefficient corresponding to the pixel at two modulation frequencies.

The calculated phase delay value will jump around the ideal value due to noise interference in the actual 80MHz and 100MHz TOF systems. If the phase delay value of a certain pixel is less in noise interference and small in jitter, an accurate unwrapping coefficient can be determined as long as a line segment closest to the position point of the phase delay value is judged, if the phase delay value of the certain pixel is greater in noise interference, the position point of the phase delay value deviates from the correct line segment, the pixel is endowed with an incorrect unwrapping coefficient, and further an incorrect distance is determined for the pixel based on the formulas (2) and (3), so that the depth map obtained by measuring the measured object is incorrect.

Based on the above reasons, the embodiment of the application provides a method for acquiring a depth map. The method may be performed by a device that acquires a depth map, which may be referred to hereafter simply as a ranging device. The distance measuring device may be a hardware device, such as a server, a terminal, or other computing device, or may be a software device, such as a set of software programs running on the hardware device.

The ranging device may acquire raw data of a plurality of modulation frequencies from the TOF system, determine a depth map based on the raw data of the plurality of modulation frequencies, and the detailed process will be described later. The ranging device in the embodiment of the application can be a part of the TOF system or can exist independently of the TOF system.

When the distance measuring device is independent of the TOF system, as shown in fig. 4, the TOF system includes a TOF chip 101, a lens 102, a laser drive 103, a laser 104, and the like, and the TOF chip 101 includes a controller 1011, a digital-to-analog converter (analog to digital converter, ADC) 1012, and a pixel array 1013. The TOF chip 101 may be referred to as an image sensor. The laser 104 may be a light-emitting diode (LED) laser, and may sequentially emit light signals with a plurality of modulation frequencies. The controller 1011 is used to control the laser 104 to emit an optical signal by the laser driver 103, and the controller 1011 is also used to control the pixel array 1013 to be exposed for a certain period of time, to obtain an exposure value. The pixel array 1013 transmits the exposure value to the ADC1012, and the ADC1012 performs analog-to-digital conversion, and transmits the exposure value after analog-to-digital conversion, that is, raw data, to the ranging device. Here, for a TOF system of multiple modulation frequencies, each time the controller 1011 controls the laser 104 to emit an optical signal of one frequency, there are four exposure values for any pixel for each modulation frequency. The four exposure values correspond to 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively, see formulas (7) to (10).

In the embodiment of the present application, it is assumed that the emitted optical signal is an intensity modulated cosine signal, and the signal is represented by formula (4):

s(t)＝cos(ωt) (4)

wherein in the formula (4), s (t) represents an emitted optical signal, and ω represents an angular rate.

The reflected light signal from the object to be measured generates an offset (due in part to the illumination of the background light) and a modulated cosine signal with a phase delay, the expression of the reflected light signal is expressed by equation (5):

wherein in the formula (5), g (t) represents the reflected light signal, ω represents the angular velocity, a represents the attenuation coefficient,the phase delay value and b the offset.

The distance is indirectly calculated by demodulating the reflected light signal, and a correlation function method is used in the process of demodulating the reflected light signal: taking the transmitted optical signal as a reference signal, obtaining a correlation function between the modulated transmitted optical signal and the reflected optical signal, and expressing by a formula (6):

wherein in the formula (6), c (τ) represents a correlation function forThe emitted optical signal selects 4 different phase delays, namely 4 different τ values: τ ₀ ＝0°，τ ₁ ＝90°，τ ₂ ＝180°，τ ₃ =270°, obtained by substituting formula (6):

The final expression of the demodulated optical signal is expressed in the expressions (7) to (10), K being the offset, and indicating the background light included in the reflected signal. The four exposure values mentioned in the foregoing for each pixel are C (τ ₀ )、C(τ ₁ )、C(τ ₂ ) And C (τ) ₃ )。

The method for acquiring the depth map can be applied to various scenes, and three possible scenes are given as follows:

scene one: automobile scene

In automotive applications, TOF systems may be applied to autopilot, crashproof self-braking, and the like. For example, the method is applied to an auxiliary driving system, and under the support of the TOF technical principle, the auxiliary driving system can accurately detect the body and head positions of a driver, and even capture the blinking actions of the driver under the condition that the driver wears glasses or sunglasses so as to judge whether the driver is focused enough, is in fatigue driving or not, and the like, so that corresponding countermeasures are started. The countermeasure may be to vibrate the seat or to sound a warning sound, etc. In addition, a quick and accurate response is possible, and the auxiliary driving system and the emergency braking system can be automatically activated before a potential emergency situation occurs.

In addition, the TOF technology can control a vehicle entertainment system or a vehicle air conditioner through hand movements or body gestures, and even realize brand new auxiliary and safety functions outside a vehicle, for example, door opening auxiliary equipment can prevent a vehicle door from being collided with other objects (such as a vehicle, a wall, a ceiling and the like) after the vehicle door is opened when a parking lot or a household garage is opened.

Scene II: man-machine interaction scene

The TOF system provides a real-time remote image that can be used simply to record human motion. Therefore, the electronic product has a brand new interaction mode, and can be a television, a mobile phone, a tablet and the like.

For example, the depth value of the object photographed by each pixel can be obtained by TOF to distinguish objects of different depths.

Scene III: measuring and machine vision scenarios

In industrial machine vision applications, TOF systems are mounted on robots that classify and precisely position objects by means of the TOF systems. In addition, the method can be applied to face recognition, earth topography mapping and the like.

After the application scenario is introduced, a method for acquiring a depth map according to an embodiment of the present application will be described below with reference to fig. 5, where the method may be performed by a ranging apparatus. As shown in fig. 5, the process flow of the method is as follows:

step 501, performing downsampling processing on raw data of multiple modulation frequencies of a measured object to obtain a depth map with a first resolution, wherein the raw data of each modulation frequency is obtained by multiple exposure at each modulation frequency when a TOF system measures distance of the measured object, the depth map with the first resolution is a depth map determined that no abnormal unwrapping coefficient exists in pixels, and the resolution of a pixel array in the TOF system is a second resolution, wherein the first resolution is lower than the second resolution.

Wherein the object to be measured is any object, such as an automobile. The number of the plurality of modulation frequencies is greater than 1, for example, the plurality of modulation frequencies may be 2 modulation frequencies, 3 modulation frequencies, or the like. The pixel value of each pixel in the depth map may be referred to as a depth value, and for any pixel's depth value, the depth value represents the distance from the TOF system to the location point of the pixel where the light signal is reflected on the object under test. The first resolution is lower than a second resolution, which is the resolution of the pixel array in the TOF system. The second resolution is a resolution which is not downsampled with respect to the first resolution, and therefore may also be referred to as the original resolution, and the first resolution is a low resolution with respect to the second resolution. The absence of an abnormal unwrapping coefficient for a pixel means that the unwrapping coefficient does not calculate errors when the unwrapping coefficient is determined using the depth value of the pixel.

In this embodiment, the controller in the TOF system is driven by the laser to control the laser to sequentially emit optical signals of each modulation frequency, so as to obtain raw data of each modulation frequency, and the following description will be given by taking an optical signal of an emission modulation frequency j (the modulation frequency j is any one of a plurality of modulation frequencies) as an example:

A controller in the TOF system is driven by a laser to control the laser to emit an optical signal with a modulation frequency j, and the optical signal is an intensity-modulated optical signal to irradiate an object to be measured. The measured object reflects the light signal, the controller controls the pixel array to expose for a period of time, and the light signal reflected by the measured object is received. After the end of the multiple exposure (for example, the phase retardation value is 0 degree, 90 degrees, 180 degrees, 270 degrees in order), the exposure value of any one pixel in the pixel array is C (τ ₀ ) _j 、C(τ ₁ ) _j 、C(τ ₂ ) _j And C (τ) ₃ ) _j Corresponding to the formulae (7) to (10) above, respectively. The pixel array transmits the exposure value of each pixel to the ADC, and the ADC carries out analog-to-digital conversion processing on the exposure value of each pixel to obtain DCS (tau) ₀ ) _j 、DCS(τ ₁ ) _j 、DCS(τ ₂ ) _j 、DCS(τ ₃ ) _j Transmitted to a distance measuring device. The value DCS (τ) obtained after the multiple exposure and quantization of each pixel ₀ ) _j 、DCS(τ ₁ ) _j 、DCS(τ ₂ ) _j 、DCS(τ ₃ ) _j Constituting raw data of modulation frequency j. Thus, the ranging device acquires raw data of the modulation frequency j.

Thus, based on the optical signals of the plurality of modulation frequencies sequentially transmitted, raw data of the plurality of modulation frequencies is obtained.

And then the ranging device can perform downsampling processing on the raw data with the plurality of modulation frequencies to obtain a depth map with a first resolution, wherein the depth map with the first resolution is a depth map determined that the pixel does not have an abnormal unwrapping coefficient. In this way, a depth map can be obtained in which the pixels do not have abnormal unwrapping coefficients.

Step 502, calculating and obtaining the unwrapping coefficient of each pixel of the second resolution by using the depth map of the first resolution and the phase delay value of each pixel of the second resolution, wherein the phase delay value of each pixel of the second resolution is calculated and obtained by using the raw data of a plurality of modulation frequencies.

In the present embodiment, for any pixel k of the second resolution at the modulation frequency j, the ranging device can calculate the phase delay value by the equations (7) to (10)The expression using formula (11) is:

wherein in formula (11), DCS (τ) ₀ ) _kj 、DCS(τ ₁ ) _kj 、DCS(τ ₂ ) _kj 、DCS(τ ₃ ) _kj And represents raw data for pixel k at modulation frequency j. In this way, since the resolution of the pixel array in the TOF system is the second resolution and the raw data corresponding to each modulation frequency is not downsampled, the phase delay value of each pixel at the second resolution can be obtained by equation (11).

The ranging device may then acquire a previously stored fuzzy distance corresponding to each modulation frequency, or may substitute each modulation frequency into equation (1) to obtain a fuzzy distance corresponding to each modulation frequency. The distance measuring device calculates and obtains the unwrapping coefficient of each pixel of the second resolution by using the depth map of the first resolution, the phase delay value of each pixel of the second resolution and the fuzzy distance corresponding to each modulation frequency.

Step 503, calculating to obtain a target depth map of the second resolution corresponding to the measured object by using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution.

In this embodiment, for any pixel k of the second resolution of the modulation frequency j, the ranging device may determine N in step 502 _kj 、 And U _j Substituting (12) to obtain the distance D of the pixel k _kj (i.e., depth values). Based on this approach, depth values for each pixel of the second resolution of the modulation frequency j are determined, and these depth values are formed into a target depth map of the second resolution corresponding to the object under test at the modulation frequency j.

Wherein in formula (12), N _kj To unwrap the coefficient for that pixel k at modulation frequency j,to delay the phase of the pixel k at the modulation frequency j, U _j Is the ambiguity distance for modulation frequency j.

And determining the target depth map of the second resolution corresponding to the measured object at each modulation frequency based on the mode of determining the target depth map of the second resolution corresponding to the measured object at the modulation frequency j.

In this way, since the downsampled depth map of the first resolution does not have the pixels with abnormal unwrapping coefficients, the unwrapping coefficients of the pixels of the second resolution are obtained by using the depth map calculation of the first resolution, so that the unwrapping coefficients of the pixels of the second resolution are more accurate, and further the depth map of the second resolution calculated based on the unwrapping coefficients is also more accurate, thereby ensuring the accuracy of the depth map of the second resolution.

Here, in step 503, the target depth map is to be distinguished from the initial depth map to be described later, and it is actually a representation depth map.

In addition, after step 503, the depth values of the corresponding pixels in the depth map of the second resolution corresponding to the measured object at the plurality of modulation frequencies may be averaged or weighted and averaged according to a certain set weight sequence, so as to obtain a depth map of the second resolution corresponding to the measured object.

The flow of fig. 5 is supplemented as follows:

there are a number of ways that this can be accomplished in step 501, three possible implementations are given below:

mode one: performing downsampling processing on the raw data of a plurality of modulation frequencies of the measured object by using the first downsampling parameters to obtain a gray level map of target resolution and a depth map of target resolution corresponding to the raw data of each modulation frequency; if no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, determining the depth map with the target resolution as the depth map with the first resolution; if at least one gray level image has pixels with gray values smaller than a first threshold corresponding to the modulation frequency, updating a first downsampling parameter, and downsampling the raw data of a plurality of modulation frequencies based on the updated downsampling parameter until a depth image with a first resolution is obtained.

The first threshold value corresponding to each modulation frequency is a preset value and is stored in the distance measuring device. The first downsampling parameter is a downsampling parameter used when downsampling processing is performed for the first time. The first downsampling parameter is a parameter for reducing resolution, for example, the first downsampling parameter is 2×2,2×2 represents 4, and the original 2×2 pixels are combined into one pixel at the time of processing. The first downsampling parameter may also be referred to as a first binning parameter, and the downsampling process may also be referred to as a binning process.

In this embodiment, the ranging device may perform downsampling processing on the raw data of each modulation frequency by using the first downsampling parameter to obtain a gray scale map of a target resolution and a depth map of the target resolution corresponding to the raw data of each modulation frequency. Illustratively, the plurality of pixels in the raw data of the modulation frequency j are divided into a plurality of pixel combinations according to a first downsampling parameter, and the number of pixels included in each pixel combination is equal to the first downsampling parameter. For example, the first downsampling parameter is 2×2, then here each pixel combination includes 2×2 pixels, and the first downsampling parameter is 3*3, then here each pixel combination includes 3*3 pixels.

Then, for any pixel combination, DCS (τ) of each pixel in the pixel combination in the raw data of the modulation frequency j ₀ ) _j Averaging to obtain DCS (τ) with pixels in the pixel combination combined into one pixel ₀ ) _j0 The method comprises the steps of carrying out a first treatment on the surface of the DCS (τ) of each pixel in the pixel combination in raw data of modulation frequency j ₁ ) _j Averaging to obtain DCS (τ) with pixels in the pixel combination combined into one pixel ₁ ) _j1 The method comprises the steps of carrying out a first treatment on the surface of the DCS (τ) of each pixel in the pixel combination in raw data of modulation frequency j ₂ ) _j Averaging to obtain DCS (τ) with pixels in the pixel combination combined into one pixel ₂ ) _j2 The method comprises the steps of carrying out a first treatment on the surface of the DCS (τ) of each pixel in the pixel combination in the raw data of the modulation frequency j ₃ ) _j Averaging to obtain DCS (τ) with pixels in the pixel combination combined into one pixel ₃ ) _j3 . Since the pixels in the pixel combination in the raw data with the second resolution are combined into one pixel, the downsampling process is equivalent to that of the raw data with the second resolution, so as to obtain raw data with a lower resolution than the second resolution, which may be referred to as raw data with a target resolution. For example, as shown in fig. 6, the second resolution is 8×8, the first downsampling parameter is 2×2, the target resolution is 4*4, and each box in fig. 6 represents one pixel.

Then, for a pixel i of the target resolution, DCS (τ ₀ ) _ij0 、DCS(τ ₁ ) _ij1 、DCS(τ ₂ ) _ij2 And DCS (τ) ₃ ) _ij3 Substituting (13) to obtain gray value A of pixel i of target resolution _ij Thus, at modulation frequency j, one gray value is assigned to each pixel of the target resolution, and a gray map of the target resolution at modulation frequency j is obtained.

Based on the above processing procedure, a gray scale map of the target resolution at the modulation frequency j can be obtained, and for each modulation frequency, the gray scale map of the target resolution corresponding to the raw data of each modulation frequency can be obtained according to the above processing.

Then, for pixel i of the target resolution, DCS (τ ₀ ) _ij0 、DCS(τ ₁ ) _ij1 、DCS(τ ₂ ) _ij2 And DCS (τ) ₃ ) _ij3 Substituting (14) to obtain the phase delay value of the pixel i of the target resolutionThus, at the modulation frequency j, one phase delay value is corresponding to each pixel of the target resolution.

After each modulation frequency is fixed, a line segment of a unwrapping area of a plurality of modulation frequencies can be obtained, and for each pixel of the target resolution, a phase delay value corresponding to each modulation frequency and a line segment of the unwrapping area of a plurality of modulation frequencies are used to obtain an unwrapping coefficient of each pixel. For example, the plurality of modulation frequencies are two modulation frequencies (modulation frequency 1 and modulation frequency 2), and for a pixel i of a target resolution, a phase delay value of the pixel i is used And( a phase delay value representing the pixel i at a modulation frequency of 1,representing the phase delay value of this pixel i at modulation frequency 2), the distance is found in fig. 3Andbit of compositionAnd determining the unwrapping coefficient corresponding to the sequence number of the line segment as the unwrapping coefficient corresponding to the pixel i.

Unwrapping coefficient N for pixel i to be at a target resolution for modulation frequency j _ij Fuzzy distance U corresponding to modulation frequency j _j Phase delay value of pixel i at modulation frequency jSubstituting (15) to obtain depth value D of pixel i of target resolution at modulation frequency j _ij 。

A depth value at the modulation frequency i for each pixel of the target resolution is determined based on equation (15).

And then averaging or weighted averaging the depth values of each pixel of the target resolution under a plurality of modulation frequencies to obtain the depth value of each pixel of the target resolution. Of course, the depth value of each pixel of the target resolution at any one of the plurality of modulation frequencies may be determined as the depth value of each pixel of the target resolution. In this way, a depth map of the target resolution can be obtained.

The distance measuring device acquires a stored first threshold value corresponding to each modulation frequency, and judges whether the gray value of each pixel is larger than or equal to the first threshold value corresponding to the modulation frequency in the gray map of the target resolution of each modulation frequency. And if the depth maps are all larger than or equal to the first threshold corresponding to the modulation frequency, determining the depth map with the target resolution as the depth map with the first resolution. If at least one gray scale map with target resolution has pixels with gray scale values smaller than a first threshold corresponding to the modulation frequency, the downsampling parameters can be updated according to a preset rule on the basis of the first downsampling parameters. The ranging device uses the updated boxing parameters to perform downsampling processing on the raw data of a plurality of modulation frequencies, a fourth-resolution gray scale map and a fourth-resolution depth map of each modulation frequency are obtained, and whether the gray scale value of each pixel in the fourth-resolution gray scale map is larger than or equal to a first threshold value corresponding to the modulation frequency is judged according to the mode. And if the gray value of each pixel is greater than or equal to the first threshold value, determining the depth map with the fourth resolution as the depth map with the first resolution. If at least one gray image with the fourth resolution has pixels with gray values smaller than the first threshold corresponding to the modulation frequency, the downsampling parameters can be updated according to a preset rule on the basis of the downsampling parameters updated last time, and then the downsampling process is performed on the raw data with the plurality of modulation frequencies until the gray values of the pixels in each gray image are larger than or equal to the first threshold corresponding to the modulation frequency.

In addition, in the above processing, the downsampling parameters are updated according to a preset rule, and the preset rule may be that a preset value is added in both the horizontal direction and the vertical direction on the basis of the original downsampling parameters. For example, the preset value is 1, the first downsampling parameter is 2×2, the first updated downsampling parameter is 3*3, and the second updated downsampling parameter is 4*4.

Here, since the gray value can reflect the signal receiving intensity, the signal receiving intensity is relatively high, the signal to noise ratio is relatively high, and the unwrapping can be performed accurately in general, and the accurate unwrapping coefficient is determined, it can be determined whether the unwrapping can be performed accurately by the gray value in the gray map.

In a second mode, the first downsampling parameter is used for downsampling the raw data of a plurality of modulation frequencies of the measured object, and a gray scale map of target resolution corresponding to the raw data of each modulation frequency is obtained; if no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, downsampling processing is carried out on raw data of a plurality of modulation frequencies by using a first downsampling parameter, so that a depth map with a first resolution is obtained; if at least one gray level image has pixels with gray values smaller than a first threshold corresponding to the modulation frequency, updating a first downsampling parameter, and downsampling the raw data of a plurality of modulation frequencies based on the updated downsampling parameter until a depth image with a first resolution is obtained.

The process of the second mode is similar to that of the first mode, except that: in the first mode, a gray level map and a depth map are determined together during the downsampling process, in the second mode, a target gray level map is determined first, downsampling parameters used for obtaining the target gray level map are target downsampling parameters, after gray level values of pixels in the target gray level map are all larger than or equal to a first threshold value corresponding to the modulation frequency, downsampling process is conducted on raw data of multiple modulation frequencies based on the target downsampling parameters, and a depth map of a first resolution is obtained.

Optionally, the first threshold value corresponding to each modulation frequency is the lowest signal receiving intensity of unwrapping error in the corresponding distance noise curve, and the distance noise curve is obtained in the TOF system.

In this embodiment, the blur distance of the TOF systems with multiple modulation frequencies is relatively large, the reflectivity of the measured object may be relatively large in the space range from 0 to the blur distance, and the intensities of the optical signals reflected by the measured object are different when the distance between the TOF systems and the measured object is different, so that some pixels in the pixel array in the TOF system may receive the optical signals with relatively high signal to noise ratio, but other pixels may receive the optical signals with relatively low signal to noise ratio. For a TOF chip of a TOF system, a fixed distance noise curve exists at each modulation frequency, the x-axis represents the intensity of a received optical signal, and the y-axis represents the standard deviation of depth values of objects to be measured, which measure the same distance for a plurality of times under a certain optical signal intensity. As shown in fig. 7, at a modulation frequency of 100MHz, for a certain pixel of the pixel array, the received optical signal intensity is reduced from 800 to 100, and then the unit of the received optical signal intensity is a digital quantization value (DN), the distance noise increases from 5 to 50, and the unit of the distance noise is mm.

After selecting a plurality of modulation frequencies, a phase delay value and unwrapping coefficient relation diagram similar to that of fig. 3 can be obtained, in which the minimum between two adjacent line segments can be obtained through geometric calculationPhase distance ph _ith For each modulation frequency, a phase delay value corresponding to each modulation frequency can be calculated, the phase delay value corresponding to each modulation frequency contains an error, and the phase delay value corresponding to the modulation frequency j is assumed to be expressed asFor the true phase delay value of the modulation frequency j,is the error of the phase delay value of the modulation frequency j.

When the phase delay value error is not very large, the unwrapping coefficient judgment will not be wrong, so there isWhen the unwrapping coefficient is judged, no error is generatedAt this time, the unwrapping coefficient may be judged to be erroneous. Assuming that the errors contained in the phase delay values corresponding to each modulation frequency are the same, there is a phase thresholdThe phase standard deviation can be obtained by the phase threshold value and the normal distribution 3sigma principle as follows:

the number n is the number of modulation frequencies.

Then substituting the phase standard deviation into (1) and distance(t is the duration from the light signal emitted by the TOF system to the light signal reflected by the detected object, c is the light speed, f is the modulation frequency j), and the standard deviation of the depth value of the modulation frequency j is determined as follows:

In formula (17) U _j Is the ambiguity distance for modulation frequency j.

Then, the standard deviation of the depth value of the modulation frequency j is used, and in the distance noise curve of the modulation frequency j, the standard deviation of the depth value is found to be sigma (d _j ) Signal reception intensity at the time of unwrapping the modulation frequency j as the lowest signal reception intensity A _th . Will A _th Dividing the first threshold value corresponding to the modulation frequency j by g, where g is the target downsampling parameter mentioned above, and the target downsampling parameter is a downsampling parameter used when obtaining the depth map with the first resolution.

Here A _th The reason for dividing g is: when the first threshold is calculated, the gray value of each pixel before the original downsampling process is calculated (the gray value meets the minimum signal-to-noise ratio requirement), but after the downsampling process, (g x g) pixels are combined into one pixel, the signal-to-noise ratio is improved by g times, and the minimum signal receiving intensity of one newly formed pixel can be changed to 1/g of the minimum signal receiving intensity when the original pixel is changed.

Mode three: using the raw data of the plurality of modulation frequencies and the phase delay diagrams of the plurality of modulation frequencies to obtain unwrapping coefficients of each pixel in the initial depth diagram of the second resolution; calculating to obtain the depth value of each pixel in the initial depth map by using the unwrapping coefficient of each pixel in the initial depth map and the phase delay value of each pixel of the second resolution; and performing downsampling processing on the initial depth map to obtain a depth map with a first resolution.

In this embodiment, the ranging device may substitute raw data of the modulation frequency j into equation (11) to determine a phase delay value of each pixel of the second resolution at the modulation frequency j, so that each pixel of the second resolution corresponds to a phase delay value at each modulation frequency. And then for any pixel, determining a line segment closest to a position point where the plurality of phase delay values are positioned in a corresponding relation diagram of the phase delay values and the unwrapping coefficients of a plurality of phase delay values of the pixel, and determining the unwrapping coefficients corresponding to the sequence numbers of the line segment as the unwrapping coefficients corresponding to the pixel under a plurality of frequencies. For example, the plurality of modulation frequencies are two modulation frequencies, in fig. 3, a line segment closest to a position point where the two phase delay values are located is determined, an unwrapping coefficient corresponding to a sequence number of the line segment is determined, and the unwrapping coefficient is determined as an unwrapping coefficient corresponding to the two modulation frequencies of the pixel. Then N of pixel k _kj 、 And U _j Substituting (12) to obtain depth value of pixel k at modulation frequency j, N _kj To unwrap the coefficient for that pixel k at modulation frequency j,to delay the phase of the pixel k at the modulation frequency j, U _j Is the ambiguity distance for modulation frequency j. The depth value of each pixel at the modulation frequency j is determined in this way.

The depth value of each pixel at a plurality of modulation frequencies is averaged, and the average value corresponding to each pixel is determined as the depth value of each pixel at the second resolution. The depth values of each pixel of the second resolution are assembled into an initial depth map of the second resolution.

And then, carrying out downsampling processing on the initial depth map with the second resolution to obtain a depth map with the first resolution, wherein the first resolution is lower than the second resolution.

Optionally, the downsampling process for the initial depth map with the second resolution includes:

performing downsampling processing on the initial depth map by using the second downsampling parameter to obtain a depth map with a third resolution; determining a gradient map corresponding to the depth map with the third resolution; if the gradient map meets the gradient condition, determining that the depth map with the third resolution is the depth map with the first resolution; if the gradient map does not meet the gradient conditions, updating a second downsampling parameter, and downsampling the initial depth map based on the updated downsampling parameter until a target gradient map meeting the gradient conditions is obtained, and determining the depth map of the obtained target gradient map as a depth map with a second resolution; wherein the gradient condition is that the absolute value of the gradient of the pixel with the distance interval smaller than the second threshold value is smaller than or equal to the third threshold value.

Wherein the second downsampling parameter may be the same as the first downsampling parameter or different. The third resolution is lower than the second resolution. In case the second downsampling parameter is the same as the first downsampling parameter, the third resolution is equal to the target resolution. The second threshold is a preset threshold, and may be 3, 4, etc. The third threshold is a preset value.

In this embodiment, the ranging device may divide the initial depth map of the second resolution into a plurality of pixel combinations according to the second downsampling parameter, where the number of pixels in each pixel combination is the same. And averaging the depth values of the pixels in each pixel combination to obtain the depth value when the pixels in each pixel combination are combined into one pixel. And combining the depth values of each pixel into a depth map with a third resolution.

And then determining a gradient map corresponding to the depth map with the third resolution, judging the size relation between the absolute gradient value of the pixels with the distance interval smaller than the second threshold value in the gradient map and the third threshold value by the distance measuring device, and if the absolute gradient value of the pixels with the distance interval smaller than the second threshold value in the gradient map is smaller than or equal to the third threshold value, determining that the gradient map meets the gradient condition, and determining that the depth map with the third resolution is the depth map with the first resolution, that is to say, the third resolution is equal to the first resolution. If the absolute value of the gradient of the pixel with the distance interval smaller than the second threshold value in the gradient map is larger than the third threshold value, determining that the gradient map does not meet the gradient condition, and updating the downsampling parameters by using a preset rule on the basis of the second downsampling parameters. And performing downsampling processing on the initial depth map with the second resolution by using the updated downsampling parameters to obtain an updated gradient map, judging whether the updated gradient map meets the gradient conditions, and if so, determining that the depth map with the first resolution is the depth map with the updated gradient map. If the updated gradient map does not meet the gradient conditions, continuously updating the gradient map on the basis of the downsampling parameters updated last time until a target gradient map meeting the gradient conditions is obtained. The depth map from which the target gradient map is obtained is determined as the depth map of the first resolution.

For example, if the second downsampling parameter is 2×2, and the absolute values of gradients of two adjacent pixels (e.g., 3, 4 pixels in line 2) in the gradient map are both greater than the third threshold, the downsampling parameter is updated from 2×2 to 3*3. For another example, if the second downsampling parameter is 2×2, and the absolute values of gradients in the gradient map with 2 intervals of 1 pixel (e.g., the 2 nd row, the 3 rd row, and the 5 th row) are all greater than the third threshold value, the downsampling parameter is updated from 2×2 to 3*3.

Thus, the gradient of the pixel can be seen as the abrupt pixel, if the gradient of the pixel has abrupt change, the absolute value of the gradient is larger, for example, 5 pixels exist in the depth map, namely [ 00 100 00 ], the third pixel is an abnormal pixel, after the gradient map is calculated, the gradient map is obtained as [0 50 0-50 0], and the gradient map does not meet the gradient condition.

In addition, since the gradient is related to the depth value error, when the unwrapping coefficient can be correctly determined, the depth value error of the pixel at each modulation frequency is usually smaller than the fourth threshold thD corresponding to the modulation frequency, so the third threshold is related to the depth value error, and the third threshold can be thD _{min f} And m, m is a positive integer not greater than a certain value, wherein the certain value is an empirical value, such as 10. Here thD _{min f} ＝U _{min f} *ph _ith /(2pi.2). Wherein U is _{min f} thD for the ambiguity distance corresponding to the smallest modulation frequency of the plurality of modulation frequencies _{min f} For the depth value error threshold value corresponding to the minimum modulation frequency, ph _ith The minimum phase distance in the phase delay diagram of the n modulation frequencies is the minimum phase distance between neighboring line segments in the phase delay diagram, and the neighboring line segments belong to unwrapping areas in the phase delay diagram.

The depth value error threshold value of the smallest modulation frequency is selected here because: u of minimum modulation frequency _{min f} If the depth value error threshold of the minimum modulation frequency is larger, if the depth value error of the minimum modulation frequency is smaller than the depth value error threshold, the gradient map also meets the gradient condition, so the third threshold is thD _{min f} /m。

In this embodiment, when calculating the gradient map of the depth map of the third resolution, the gradient value of any position (x, y) in the depth map of the third resolution may be calculated using the following equation:

G(x,y)＝dx(i,j)+dy(i,j)(18)

where G (x, y) represents a gradient value at pixel (x, y) in the depth map of the third resolution, dx (I, j) = [ I (i+1, j) -I (I-1, j) ]/2, dy (I, j) = [ I (I, j+1) -I (I, j-1) ]/2, I () represents a depth value of the corresponding pixel, e.g., I (i+1, j) is the depth value at pixel (i+1, j). For pixels at edge locations, the depth value may be determined directly as a gradient value, or may be calculated using other means.

The method is only one possible way to calculate the gradient map, and any way to calculate the gradient map may apply the embodiments of the present application and will not be described herein.

In the downsampling process, the number of times of updating the downsampling parameters is limited, and if the number of times of updating reaches a certain number and the depth map of the first resolution is not obtained yet, the acquired raw data is considered to be wrong, and the raw data can be acquired again.

In one possible implementation, the processing of step 502 may be:

the unwrapping coefficient of the pixel k of the second resolution at the modulation frequency j is determined as:

wherein round []To round-off operation, D _L For a depth value of pixel k corresponding in a depth map of a first resolution, N _kj For the unwrap coefficient at the modulation frequency j pixel k,u is a phase delay value corresponding to a pixel k at a modulation frequency j _j The ambiguity distance corresponding to the modulation frequency j.

In this embodiment, the ranging device may determine the unwrapping coefficient of pixel k at frequency j in the second resolution using equation (19).

Wherein the formula (19) can be deduced from the formula (2), in the formula (19),representation pairPerforming rounding operation, D _L For a depth value of pixel k corresponding in a depth map of a first resolution, N _kj For the unwrap coefficient at the modulation frequency j pixel k,u is a phase delay value corresponding to a pixel k at a modulation frequency j _j For modulatingThe blur distance corresponding to frequency j.

For example, when there are two modulation frequencies, the unwrap coefficient of pixel k at modulation frequency 1 is:

the unwrapping coefficient of pixel k at modulation frequency 2 is:

wherein D in the formulae (20) and (21) _L For the depth value of pixel k in the depth map of the first resolution,andthe phase delay values of the pixels k at the modulation frequency 1 and the modulation frequency 2, respectively, U ₁ And U ₂ The respective blur distances corresponding to the modulation frequency 1 and the modulation frequency 2.

It should be noted that, since the first resolution is lower than the second resolution and the pixels of the first resolution are formed by downsampling a plurality of pixels at the second resolution, when calculating the unwrapping coefficient, the pixels k of the second resolution correspond to the depth value D in the depth map of the first resolution _L A depth value of a pixel formed for a plurality of pixels including a pixel k.

In one possible implementation, some pixels in the TOF system capture the edge of the object under test (a scene with both near and far views), and when the difference between the near and far views and the TOF system is relatively large, the downsampled pixels are calculated together with the neighboring pixels Depth value D of (2) _L Since the actual depth value of these pixels is greatly different from that of the pixels before downsampling, the disentangling coefficients of the pixels with the second resolution are calculated directly by using the equation (19), which makes the disentangling coefficients calculation erroneous, so that the disentangling coefficients need to be recalculated for the object edge region. For example, as shown in fig. 8, a scene is shown in which a TOF system shoots a whiteboard at two different distances vertically using two modulation frequencies (80 MHz and 100 MHz) without noise, 3 pixels on the left shoots a whiteboard at 2.5 meters away, 5 pixels on the right shoots a whiteboard at 3.5 meters away, and when the bin parameters are 2 x 2, the 2 nd column (i.e., 3 th and 4 th pixels of the left image, pixels a and C in the third column, and pixels B and D in the fourth column) after the downsampling process contains pixels with two depth values (3 rd column, 2.5 meters, 4 th column, 3.5 meters), assuming that the depth value output after unwrapping the two modulation frequencies is D _L (e.g., 2.49 meters) and then using equation (19) to calculate the unwrapping coefficients for 2 x 2 pixels in the edge region can result in erroneous unwrapping coefficient calculations for pixels B and D. Therefore, an edge error detection function can be introduced, as described in detail below:

accordingly, the processing of step 503 may be:

Calculating a depth value of each pixel of the second resolution for each modulation frequency using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution; determining a depth value error for each pixel of the second resolution for each modulation frequency using the depth map of the first resolution and the depth values for each pixel of the second resolution for each modulation frequency; and calculating and obtaining a target depth map of a second resolution corresponding to the measured object by using the depth value error of each pixel of the first resolution of each modulation frequency.

In this embodiment, the distance measuring device may substitute the pixel unwrapping coefficient of the second resolution of the modulation frequency j, the phase delay value of each pixel at the modulation frequency j, and the blur distance corresponding to the modulation frequency j into equation (12) to obtain the depth value of each pixel at the modulation frequency j. In this way, the depth value of each pixel at each modulation frequency can be determined.

The ranging device may then determine a depth value error for each pixel at the second resolution for each modulation frequency using the depth map at the first resolution and the depth values for each pixel at the second resolution for each modulation frequency. The ranging device may determine a target depth map of the object under test at the second resolution at each modulation frequency using the depth value error for each pixel at the second resolution at each modulation frequency.

In one possible implementation, the process of determining the depth value error for each pixel may be:

the ranging device may determine the depth value error for pixel k at the second resolution of modulation frequency j using equation (22) as:

E _kj ＝|D _L -D _kj | (22)

in formula (22), E _kj Representing the test distance D of pixel k after unwrapping at modulation frequency j _kj And D _L Absolute value of error of E _kj The larger the depth value error, requiring recalculation of the depth value of pixel k. D (D) _L For the corresponding depth value of pixel k in the depth map of the first resolution,D _kj for the depth value of pixel k at the second resolution of modulation frequency j,to the corresponding phase delay value of the pixel k at the modulation frequency j, N _kj U is the unwrapping coefficient of pixel k at modulation frequency j _j The ambiguity distance corresponding to the modulation frequency j.

Here, the depth value corresponding to the pixel k of the second resolution in the depth map of the first resolution is the depth value of the pixel including the pixel k that is formed when the pixel k is downsampled. For example, in fig. 8, pixel k is pixel B in fig. 8, and the depth value corresponding to pixel B in the depth map of the first resolution is the depth value of the pixel composed of pixels A, B, C and D.

In one possible implementation, the process of determining the target depth map for the second resolution for each modulation frequency may be:

For a modulation frequency j in the plurality of modulation frequencies, determining a target pixel with a depth value error larger than a fourth threshold value corresponding to the modulation frequency j in pixels with a second resolution of the modulation frequency j, wherein the value of j is 1 to n, and n is the number of the plurality of modulation frequencies; updating the unwrapping coefficient of the target pixel with the second resolution of the modulation frequency j by using the depth value of the adjacent pixel of the target pixel in the depth map with the first resolution; updating the depth value of the target pixel of the second resolution of the modulation frequency j by using the updated unwrapping coefficient of the target pixel and the phase delay value of the target pixel of each modulation frequency; and combining the depth values of the pixels except the target pixel in the pixels with the second resolution of the modulation frequency j with the updated depth values of the target pixel to obtain a target depth map with the second resolution of the modulation frequency j, which corresponds to the measured object.

The fourth threshold is a preset value, and may be stored in the ranging device.

In this embodiment, for the modulation frequency j, the ranging device may acquire a fourth threshold corresponding to the modulation frequency j, and then the ranging device may determine whether the depth value error in the pixel of the second resolution of the modulation frequency j is greater than the fourth threshold corresponding to the modulation frequency j, and in the depth map of the second resolution of the modulation frequency j, obtain a target pixel with the depth value error greater than the fourth threshold. Then, in the depth map of the first resolution, depth values of neighboring pixels of the target pixel are determined. For example, in fig. 8, the target pixel of the second resolution is a pixel B, the adjacent pixels in the depth map of the first resolution are pixels P, M and N, the target pixel of the second resolution is a, and the adjacent pixels in the depth map of the first resolution are pixels P, M and N.

Assuming that the target pixel is a pixel B, for any adjacent pixel P in the depth map of the target pixel at the first resolution, the distance measuring device substitutes the depth value of the adjacent pixel P, the phase delay value of the target pixel at the second resolution at the modulation frequency j, and the blur distance corresponding to the modulation frequency j into equation (19) to obtain the unwrapping coefficient of the target pixel at the second resolution at the modulation frequency j when the depth value of the adjacent pixel P is used.

Then, the distance measuring device substitutes the unwrapping coefficient of the target pixel at the modulation frequency j, the phase delay value of the target pixel at the modulation frequency j, and the blurring distance corresponding to the modulation frequency j into formula (12) to obtain a depth value of the target pixel at the second resolution of the modulation frequency j, the depth value being obtained based on the depth values of the adjacent pixels P.

Then, the depth value of the adjacent pixel P and the determined depth value of the target pixel at the modulation frequency j are substituted into formula (22) to obtain the depth value error of the target pixel corresponding to the adjacent pixel P.

Based on the above manner, each adjacent pixel of the depth map of the target pixel at the first resolution is used to determine the depth value error of the target pixel corresponding to each adjacent pixel.

The distance measuring device determines the depth value of the target pixel at the modulation frequency j as the depth value when the depth value error is minimum and is smaller than the depth value of the fourth threshold corresponding to the modulation frequency j. Or if the determined depth value errors of the target pixels are smaller than the fourth threshold value, determining the depth value of the target pixel corresponding to any depth value error at the first frequency as the depth value of the target pixel at the modulation frequency j. For example, in fig. 8, the target pixel is pixel B, which is adjacent pixel P, M, N in the depth map of the first resolution, and since pixel B is closer to pixel M from the TOF system, the depth value error calculated based on adjacent pixel M is minimal. In addition, if the depth value error of each adjacent pixel corresponding to the target pixel is larger than the fourth threshold value, the error of the raw data acquired at this time is indicated, the raw data is acquired again, and a depth map is calculated.

And then the distance measuring device combines the depth values of the pixels except the target pixel in the pixels with the second resolution of the modulation frequency j with the updated depth values of the target pixel to obtain a target depth map with the second resolution of the measured object with the modulation frequency j. The target depth map of the second resolution of each modulation frequency measured object may be determined in the manner described above.

In this way, after the region with the error depth value is obtained, the depth value of each pixel in the region with the error depth value can be calculated by using the correct depth values of the adjacent pixels, so that the correct calculation of the depth values of all the pixels can be ensured, and the correct depth map with the original resolution can be obtained.

Optionally, the fourth threshold corresponding to the modulation frequency j is: thD _j ＝U _j *ph _ith /(2pi.2); wherein U is _j To modulate the fuzzy distance corresponding to the frequency j, ph _ith The minimum phase distance between adjacent line segments in the phase delay diagram of n modulation frequencies is the value of j which is 1 to n, and the adjacent line segments belong to the unwrapping area in the phase delay diagram. The fourth threshold represents the minimum depth value error when an unwrapping error occurs. For any modulation frequency, if the depth value error of a certain pixel is larger than the fourth threshold thD corresponding to the modulation frequency, a unwrapping error occurs, and an erroneous unwrapping coefficient is determined.

Here, if the error of the phase delay value exceeds one half of the minimum phase distance, there is a possibility that the unwrapping coefficient determined as the neighboring line segment may be determined to be the wrong unwrapping number, and if the error of the phase delay value is less than one half of the minimum phase distance, the probability of the unwrapping coefficient determined as the neighboring line segment is relatively small, so that the distance corresponding to one half of the minimum phase distance may be used as the threshold value of the depth value error.

In addition, for better understanding of the embodiments of the present application, a flow for the first mode is provided:

as shown in fig. 9, step 901 begins.

Step 902, acquiring raw data of a plurality of modulation frequencies from a TOF system;

step 903, taking the first downsampling parameter as the current downsampling parameter, performing downsampling processing on raw data of a plurality of modulation frequencies to obtain raw data of a target resolution of each modulation frequency, and determining a phase delay value of each pixel of a second resolution of each modulation frequency by using the raw data of the plurality of modulation frequencies;

step 904, determining a depth map of the target resolution and a gray map of the target resolution of each modulation frequency based on the raw data of the target resolution of each modulation frequency;

Step 905, judging whether the gray value of each pixel in the gray map of each target resolution is greater than or equal to a first threshold value of the modulation frequency;

step 906, if the determination result of step 905 is yes, determining the depth map of the target resolution as the depth map of the first resolution, and executing step 907. If the judgment result in step 905 is no, returning to update the downsampling parameter, taking the updated downsampling parameter as the current downsampling parameter, and executing the downsampling process on the raw data of a plurality of modulation frequencies until a depth map of a first resolution is obtained;

step 907, calculating unwrapping coefficients of each pixel of the second resolution at each modulation frequency using the depth map of the first resolution;

step 908, calculating a depth value error of each pixel of the second resolution at each modulation frequency using the unwrapping coefficient of each pixel of the second resolution, the depth map of the first resolution, and the phase delay value of each pixel of the second resolution;

step 909, judging whether pixels with depth value errors larger than a fourth threshold value corresponding to the modulation frequency exist in the pixels with the second resolution of each modulation frequency;

step 910, if the determination result in step 909 is yes, outputting a depth map of the second resolution corresponding to the measured object at each modulation frequency;

If the determination result in step 909 is no, step 911 is performed, the unwrapping coefficient of the target pixel is recalculated based on the neighboring pixels of the target pixel in the depth map of the first resolution, and step 908 is performed back until the depth value error of each pixel of the second resolution of each modulation frequency is smaller than or equal to the fourth threshold value of the associated modulation frequency, and the depth map of the second resolution corresponding to the object to be measured at each modulation frequency is output, where the target pixel is a pixel whose depth value error is greater than the fourth threshold value corresponding to the modulation frequency.

The detailed description of each step in fig. 9 may be found in the foregoing description, and will not be repeated here.

In this way, the downsampling process is added to the raw data of a plurality of modulation frequencies before the disentanglement coefficient is calculated, so that the pixels obtained after downsampling have no depth map of the first resolution of the abnormal disentanglement coefficient, the disentanglement coefficient of each pixel in the second resolution calculated based on the depth map of the first resolution is accurate, the depth map of the second resolution calculated based on the disentanglement coefficient is accurate, and the accuracy of the depth map of the second resolution is ensured.

In addition, in order to better understand the embodiment of the present application, a flow for the third mode is provided:

As shown in fig. 10, step 1001 begins.

Step 1002, acquiring raw data of a plurality of modulation frequencies from a TOF system;

step 1003, calculating an unwrapping coefficient of each pixel of the second resolution at each modulation frequency and a phase delay value of each pixel at each modulation frequency, and calculating a depth map of the second resolution by using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution;

step 1004, taking the second downsampling parameter as the current downsampling parameter, downsampling the depth map of the second resolution to obtain the depth map of the third resolution, and determining the gradient map of the depth map of the third resolution;

step 1005, judging whether the gradient map of the depth map of the third resolution satisfies a gradient condition;

step 1006, determining that the depth map of the third resolution is the depth map of the first resolution if the gradient map of the depth map of the third resolution satisfies the gradient condition, and executing step 1007; returning to update the downsampling parameter under the condition that the gradient map of the depth map with the third resolution does not meet the gradient condition, taking the updated downsampling parameter as the current downsampling parameter, and executing step 1004;

Step 1007, calculating the unwrapping coefficient of each pixel of the second resolution at each modulation frequency using the third resolved depth map;

step 1008, determining a depth value error of each pixel of the second resolution at each modulation frequency based on the unwrapping coefficient of each pixel of the second resolution at each modulation frequency, the depth map of the first resolution, and the phase delay value of each pixel of the second resolution at each modulation frequency;

step 1009, determining whether there is a pixel with a depth value error greater than the fourth threshold value in the pixels of the second resolution;

step 1010, if the determination result in step 1009 is yes, outputting a depth map of the second resolution corresponding to the object to be measured at each modulation frequency;

if the determination result in step 1009 is no, step 1011 is performed, the unwrapping coefficient of the target pixel is recalculated based on the neighboring pixels of the target pixel in the depth map of the first resolution, and step 908 is performed back until the depth value error of each pixel of the second resolution of each modulation frequency is less than or equal to the fourth threshold of the associated modulation frequency, and the depth map of the second resolution corresponding to the object to be measured at each modulation frequency is output, where the target pixel is a pixel whose depth value error is greater than the fourth threshold of the modulation frequency.

The detailed description of each step in fig. 10 may be found in the foregoing description, and will not be repeated here.

In this way, the depth map with abnormal unwrapping pixels is downsampled, so that the downsampled depth map with the first resolution does not have pixels with abnormal unwrapping coefficients, the unwrapping coefficients of the pixels in the second resolution calculated based on the depth map with the first resolution are accurate, and further the depth map with the second resolution calculated based on the unwrapping coefficients is accurate, so that the accuracy of the depth map with the second resolution is ensured.

According to the embodiment of the application, aiming at the problem that the determined unwrapping coefficient is wrong and the distance precision is reduced due to overlarge phase delay value errors in a TOF system with a plurality of modulation frequencies, the low-resolution depth map with no abnormal unwrapping coefficient of pixels is determined, the unwrapping coefficient of each pixel with original resolution is calculated through the low-resolution depth map, the accurate unwrapping coefficient can be determined, and the accurate depth map can be further determined.

In addition, for a pixel whose depth value is wrong, determining the depth value using the correct depth values of the neighboring pixels of the pixel can make the depth values of all pixels the correct depth value.

In the embodiment of the application, all the related calculation processes can be obtained by using a floating point number calculator or obtained by looking up a table.

Fig. 11 is a block diagram of an apparatus for acquiring a depth map according to an embodiment of the present application. The apparatus may be implemented as part or all of an apparatus by software, hardware, or a combination of both. The apparatus provided by the embodiment of the present application may implement the flow described in fig. 5 of the embodiment of the present application, where the apparatus includes: a downsampling module 1110, an unwrapping coefficient determination module 1120, and a depth map determination module 1130, wherein:

a downsampling module 1110, configured to downsample raw data of multiple modulation frequencies of a measured object to obtain a depth map of a first resolution, where the raw data of each modulation frequency is obtained by multiple exposure at each modulation frequency when a TOF system measures the distance of the measured object, the depth map of the first resolution is a depth map determined that no abnormal unwrapping coefficient exists in a pixel, and the resolution of a pixel array in the TOF system is a second resolution, where the first resolution is lower than the second resolution, and may be specifically used to implement the downsampling function of step 501 and execute an implicit step included in step 501;

A unwrapping coefficient determining module 1120, configured to calculate an unwrapping coefficient of each pixel of the second resolution using the depth map of the first resolution and a phase delay value of each pixel of the second resolution, where the phase delay value of each pixel of the second resolution is calculated using raw data of the plurality of modulation frequencies, and may be specifically used to implement the unwrapping coefficient determining function of step 502 and execute the implicit steps included in step 502;

the depth map determining module 1130 is further configured to calculate and obtain a target depth map of the second resolution corresponding to the object to be measured by using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution, and may specifically be used to implement the depth map determining function of step 503 and execute the implicit steps included in step 503.

In one possible implementation, the downsampling module 1110 is configured to:

performing downsampling processing on raw data of a plurality of modulation frequencies of a measured object by using a first downsampling parameter to obtain a gray scale map of target resolution and a depth map of the target resolution, wherein the gray scale map corresponds to the raw data of each modulation frequency;

If no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, determining that the depth map with the target resolution is the depth map with the first resolution;

if at least one pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in the gray map, updating the first downsampling parameter, and downsampling the raw data of the modulation frequencies based on the updated downsampling parameter until a depth map with the first resolution is obtained.

In one possible implementation, the downsampling module 1110 is configured to:

performing downsampling processing on raw data of a plurality of modulation frequencies of a measured object by using a first downsampling parameter to obtain a gray scale map of target resolution corresponding to the raw data of each modulation frequency;

if no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, downsampling the raw data of the plurality of modulation frequencies by using the first downsampling parameter to obtain a depth map with the first resolution;

In one possible implementation, the downsampling module 1110 is configured to:

obtaining unwrapping coefficients of each pixel in the initial depth map of the second resolution by using the raw data of the plurality of modulation frequencies and the phase delay maps of the plurality of modulation frequencies;

calculating to obtain a depth value of each pixel in the initial depth map by using the unwrapping coefficient of each pixel in the initial depth map and the phase delay value of each pixel of the second resolution;

and carrying out downsampling processing on the initial depth map to obtain the depth map with the first resolution.

In one possible implementation, the downsampling module 1110 is configured to:

performing downsampling processing on the initial depth map by using a second downsampling parameter to obtain a depth map with a third resolution;

determining a gradient map corresponding to the depth map with the third resolution;

if the gradient map meets the gradient condition, determining that the depth map with the third resolution is the depth map with the first resolution;

If the gradient map does not meet the gradient conditions, updating the second downsampling parameters, downsampling the initial depth map based on the updated downsampling parameters until a target gradient map meeting the gradient conditions is obtained, and determining the depth map of the target gradient map as the depth map of the second resolution;

wherein the gradient condition is that the absolute value of the gradient of the pixel with the distance interval smaller than the second threshold value is smaller than or equal to the third threshold value.

In one possible implementation, the unwrapping coefficient determining module 1120 is configured to:

the unwrapping coefficient of the pixel k of the second resolution determining the modulation frequency j is:

wherein the modulation frequency j belongs to the plurality of modulation frequencies, round []To round-off operation, D _L For the depth value of the pixel k corresponding to the depth map of the first resolution, N _kj For the unwrap coefficient of the pixel k at the modulation frequency j,to the phase delay value of the pixel k at the modulation frequency j, U _j And the fuzzy distance corresponding to the modulation frequency j.

In one possible implementation, the depth map determining module 1130 is configured to:

calculating a depth value of each pixel of the second resolution for each of the modulation frequencies using an unwrapping coefficient of each pixel of the second resolution and a phase delay value of each pixel of the second resolution;

Determining a depth value error for each pixel of the second resolution for each of the modulation frequencies using the depth map of the first resolution and the depth values for each pixel of the second resolution for each of the modulation frequencies;

and calculating and obtaining a target depth map of the second resolution corresponding to the measured object by using the depth value errors of the pixels of the first resolution of each modulation frequency.

In one possible implementation, the depth map determining module 1130 is configured to: the depth value error of the pixel k of the second resolution determining the modulation frequency j is: e (E) _kj ＝|D _L -D _kj |；

Wherein D is _L For the corresponding depth value of the pixel k in the depth map of the first resolution,D _kj for the depth value of the pixel k of the second resolution of the modulation frequency j,to the phase delay value of the pixel k at the modulation frequency j, N _kj To unwrap the coefficient of the pixel k at the modulation frequency j, U _j And for the fuzzy distance corresponding to the modulation frequency j, j takes a value of 1 to n, and n is the number of the modulation frequencies.

for a modulation frequency j in the plurality of modulation frequencies, determining a target pixel with a depth value error larger than a fourth threshold corresponding to the modulation frequency j in pixels with the second resolution of the modulation frequency j, wherein j takes a value of 1 to n, and n is the number of the plurality of modulation frequencies;

Updating the unwrapping coefficient of the target pixel of the second resolution of the modulation frequency j using depth values of neighboring pixels of the target pixel in the depth map of the first resolution;

updating a depth value of the target pixel of the second resolution of the modulation frequency j using the updated unwrapping coefficient of the target pixel and the phase delay value of the target pixel of each of the modulation frequencies;

and combining the depth values of the pixels except the target pixel in the pixels with the second resolution of the modulation frequency j with the updated depth values of the target pixel to obtain a target depth map with the second resolution of the modulation frequency j, corresponding to the measured object.

In one possible implementation manner, the fourth threshold value corresponding to the modulation frequency j is: thD _j ＝U _j *ph _ith /(2π*2)；U _j For the fuzzy distance corresponding to the modulation frequency j, ph _ith And j is 1 to n for the minimum phase distance between adjacent line segments of the unwrapping area in the phase delay diagrams of the plurality of modulation frequencies, and n is the number of the plurality of modulation frequencies.

The division of the modules in the embodiments of the present application is schematically shown as only one logic function division, and another division manner may be adopted in actual implementation, and in addition, each functional module in each embodiment of the present application may be integrated in one processor, or may exist alone physically, or two or more modules may be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules.

The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or partly in the form of a software product or all or part of the technical solution, which is stored in a storage medium, and includes several instructions for causing a terminal device (which may be a personal computer, a mobile phone, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The embodiment of the application also provides a computing device for acquiring the depth map. Fig. 12 illustratively provides one possible architectural diagram of a computing device 1200.

The computing device 1200 includes a memory 1201, a processor 1202, a communication interface 1203, and a bus 1204. Wherein the memory 1201, the processor 1202 and the communication interface 1203 are communicatively coupled to each other via a bus 1204.

The memory 1201 may be a ROM, static storage device, dynamic storage device, or RAM. The memory 1201 may store a program, and the processor 1202 and the communication interface 1203 are configured to perform a method of acquiring a depth map when the program stored in the memory 1201 is executed by the processor 1202. The memory 1201 may also store raw data.

The processor 1202 may employ a general-purpose central processing unit (central processing unit, CPU), microprocessor, application specific integrated circuit (application specific integrated circuit, ASIC), graphics processor (graphics processing unit, GPU) or one or more integrated circuits.

The processor 1202 may also be an integrated circuit chip with signal processing capabilities. In implementation, some or all of the functionality of the depth map acquisition apparatus of the present application may be performed by integrated logic circuitry in hardware or instructions in software in the processor 1202. The processor 1202 described above may also be a general purpose processor, a digital signal processor (digital signal processing, DSP), an application specific integrated circuit, an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The methods, steps and logic blocks disclosed in the above embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in the memory 1201, and the processor 1202 reads the information in the memory 1201 and combines with its hardware to perform part of the functions of the apparatus for acquiring a depth map according to the embodiments of the present application.

The communication interface 1203 uses a transceiver module, such as, but not limited to, a transceiver, to enable communication between the computing device 1200 and other devices or communication networks. For example, the data set may be acquired through the communication interface 1203.

The bus 1204 may include a path to transfer information between various components of the computing device 1200 (e.g., the memory 1201, the processor 1202, the communication interface 1203).

The descriptions of the processes corresponding to the drawings have emphasis, and the descriptions of other processes may be referred to for the parts of a certain process that are not described in detail.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof, and when implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions which, when loaded and executed on a server or terminal, fully or partially produce a process or function in accordance with embodiments of the present application. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer readable storage medium may be any available medium that can be accessed by a server or terminal or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a digital video disk (digital videodisk, DVD), etc.), or a semiconductor medium (such as a solid state disk, etc.).

The foregoing description of the exemplary embodiments of the application is not intended to limit the application to the particular embodiments disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the application.

Claims

A method of acquiring a depth map, the method comprising:

performing downsampling processing on bare raw data of a plurality of modulation frequencies of a measured object to obtain a depth map with first resolution, wherein the raw data of each modulation frequency is obtained by exposing the measured object for a plurality of times under each modulation frequency when a time-of-flight TOF system ranges the distance of the measured object, the depth map with the first resolution is a depth map with pixels determined to have no abnormal unwrapping coefficient, the resolution of a pixel array in the TOF system is a second resolution, and the first resolution is lower than the second resolution;

calculating and obtaining an unwrapping coefficient of each pixel of the second resolution by using the depth map of the first resolution and the phase delay value of each pixel of the second resolution, wherein the phase delay value of each pixel of the second resolution is obtained by calculating by using raw data of the plurality of modulation frequencies;

And calculating to obtain a target depth map of the second resolution corresponding to the measured object by using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution.
The method of claim 1, wherein the downsampling the raw data of the measured object at the plurality of modulation frequencies to obtain the depth map with the first resolution comprises:

performing downsampling processing on raw data of a plurality of modulation frequencies of a measured object by using a first downsampling parameter to obtain a gray scale map of target resolution and a depth map of the target resolution, wherein the gray scale map corresponds to the raw data of each modulation frequency;

if no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, determining that the depth map with the target resolution is the depth map with the first resolution;

if at least one pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in the gray map, updating the first downsampling parameter, and downsampling the raw data of the modulation frequencies based on the updated downsampling parameter until a depth map with the first resolution is obtained.
The method of claim 1, wherein the downsampling the raw data of the measured object at the plurality of modulation frequencies to obtain the depth map with the first resolution comprises:

performing downsampling processing on raw data of a plurality of modulation frequencies of a measured object by using a first downsampling parameter to obtain a gray scale map of target resolution corresponding to the raw data of each modulation frequency;

if no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, downsampling the raw data of the plurality of modulation frequencies by using the first downsampling parameter to obtain a depth map with the first resolution;

if at least one pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in the gray map, updating the first downsampling parameter, and downsampling the raw data of the modulation frequencies based on the updated downsampling parameter until a depth map with the first resolution is obtained.
A method according to claim 2 or 3, characterized in that the first threshold value for each of the modulation frequencies is the lowest signal reception intensity of unwrapping in the corresponding distance noise curve obtained in the TOF system.
The method of claim 1, wherein the downsampling the raw data of the measured object at the plurality of modulation frequencies to obtain the depth map with the first resolution comprises:

obtaining unwrapping coefficients of each pixel in the initial depth map of the second resolution by using the raw data of the plurality of modulation frequencies and the phase delay maps of the plurality of modulation frequencies;

calculating to obtain a depth value of each pixel in the initial depth map by using the unwrapping coefficient of each pixel in the initial depth map and the phase delay value of each pixel of the second resolution;

and carrying out downsampling processing on the initial depth map to obtain the depth map with the first resolution.
The method of claim 5, wherein downsampling the initial depth map to obtain the depth map at the first resolution comprises:

performing downsampling processing on the initial depth map by using a second downsampling parameter to obtain a depth map with a third resolution;

determining a gradient map corresponding to the depth map with the third resolution;

if the gradient map meets the gradient condition, determining that the depth map with the third resolution is the depth map with the first resolution;

If the gradient map does not meet the gradient conditions, updating the second downsampling parameters, downsampling the initial depth map based on the updated downsampling parameters until a target gradient map meeting the gradient conditions is obtained, and determining the depth map of the target gradient map as the depth map of the second resolution;

wherein the gradient condition is that the absolute value of the gradient of the pixel with the distance interval smaller than the second threshold value is smaller than or equal to the third threshold value.
The method according to any one of claims 1 to 6, wherein calculating the unwrapping coefficient of each pixel of the second resolution using the depth map of the first resolution and the phase delay value of each pixel of the second resolution includes:

the unwrapping coefficient of the pixel k of the second resolution determining the modulation frequency j is:

wherein the modulation frequency j belongs to the plurality of modulation frequencies, round []To round-off operation, D _L For the depth value of the pixel k corresponding to the depth map of the first resolution, N _kj For the unwrap coefficient of the pixel k at the modulation frequency j,to the phase delay value of the pixel k at the modulation frequency j, U _j And the fuzzy distance corresponding to the modulation frequency j.
The method according to any one of claims 1 to 7, wherein calculating, using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution, a target depth map of the second resolution corresponding to the measured object includes:

calculating a depth value of each pixel of the second resolution for each of the modulation frequencies using an unwrapping coefficient of each pixel of the second resolution and a phase delay value of each pixel of the second resolution;

determining a depth value error for each pixel of the second resolution for each of the modulation frequencies using the depth map of the first resolution and the depth values for each pixel of the second resolution for each of the modulation frequencies;

and calculating and obtaining a target depth map of the second resolution corresponding to the measured object by using the depth value errors of the pixels of the first resolution of each modulation frequency.
The method of claim 8, wherein determining the depth value error for each pixel of the second resolution for each modulation frequency using the depth map of the first resolution and the depth values for each pixel of the second resolution for each modulation frequency comprises:

The depth value error of the pixel k of the second resolution determining the modulation frequency j is: e (E) _kj ＝|D _L -D _kj |；

Wherein D is _L For the corresponding depth value of the pixel k in the depth map of the first resolution,D _kj for the depth value of the pixel k of the second resolution of the modulation frequency j,to the phase delay value of the pixel k at the modulation frequency j, N _kj To unwrap the coefficient of the pixel k at the modulation frequency j, U _j And for the fuzzy distance corresponding to the modulation frequency j, j takes a value of 1 to n, and n is the number of the modulation frequencies.
The method according to claim 8 or 9, wherein calculating a target depth map of the second resolution corresponding to the object to be measured using the depth value errors of the pixels of the first resolution for each of the modulation frequencies, comprises:

for a modulation frequency j in the plurality of modulation frequencies, determining a target pixel with a depth value error larger than a fourth threshold corresponding to the modulation frequency j in pixels with the second resolution of the modulation frequency j, wherein j takes a value of 1 to n, and n is the number of the plurality of modulation frequencies;

updating the unwrapping coefficient of the target pixel of the second resolution of the modulation frequency j using depth values of neighboring pixels of the target pixel in the depth map of the first resolution;

Updating a depth value of the target pixel of the second resolution of the modulation frequency j using the updated unwrapping coefficient of the target pixel and the phase delay value of the target pixel of each of the modulation frequencies;

and combining the depth values of the pixels except the target pixel in the pixels with the second resolution of the modulation frequency j with the updated depth values of the target pixel to obtain a target depth map with the second resolution of the modulation frequency j, corresponding to the measured object.
The method of claim 10, wherein the fourth threshold corresponding to the modulation frequency j is: thD _j ＝U _j *ph _ith /(2π*2)；U _j For the fuzzy distance corresponding to the modulation frequency j, ph _ith And j is 1 to n for the minimum phase distance between adjacent line segments of the unwrapping area in the phase delay diagrams of the plurality of modulation frequencies, and n is the number of the plurality of modulation frequencies.
An apparatus for acquiring a depth map, the apparatus comprising:

the downsampling module is used for downsampling processing of bare raw data of a plurality of modulation frequencies of a measured object to obtain a depth map with first resolution, wherein the raw data of each modulation frequency is obtained by multiple exposure at each modulation frequency when a time-of-flight TOF system measures the distance of the measured object, the depth map with the first resolution is determined as a depth map with no abnormal unwrapping coefficient of pixels, the resolution of a pixel array in the TOF system is second resolution, and the first resolution is lower than the second resolution;

A unwrapping coefficient determining module, configured to calculate an unwrapping coefficient of each pixel of the second resolution using the depth map of the first resolution and a phase delay value of each pixel of the second resolution, where the phase delay value of each pixel of the second resolution is calculated using raw data of the plurality of modulation frequencies;

and the depth map determining module is used for calculating and obtaining a target depth map of the second resolution corresponding to the measured object by using the unwrapping coefficient of each pixel of the second resolution and the phase delay value of each pixel of the second resolution.
The apparatus of claim 12, wherein the downsampling module is configured to:

performing downsampling processing on raw data of a plurality of modulation frequencies of a measured object by using a first downsampling parameter to obtain a gray scale map of target resolution and a depth map of the target resolution, wherein the gray scale map corresponds to the raw data of each modulation frequency;

if no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, determining that the depth map with the target resolution is the depth map with the first resolution;

If at least one pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in the gray map, updating the first downsampling parameter, and downsampling the raw data of the modulation frequencies based on the updated downsampling parameter until a depth map with the first resolution is obtained.
The apparatus of claim 12, wherein the downsampling module is configured to:

performing downsampling processing on raw data of a plurality of modulation frequencies of a measured object by using a first downsampling parameter to obtain a gray scale map of target resolution corresponding to the raw data of each modulation frequency;

if no pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in each gray map, downsampling the raw data of the plurality of modulation frequencies by using the first downsampling parameter to obtain a depth map with the first resolution;

if at least one pixel with the gray value smaller than the first threshold corresponding to the modulation frequency exists in the gray map, updating the first downsampling parameter, and downsampling the raw data of the modulation frequencies based on the updated downsampling parameter until a depth map with the first resolution is obtained.
The apparatus of claim 13 or 14, wherein the first threshold value for each of the modulation frequencies is a lowest signal reception intensity of unwrapping in a corresponding distance noise curve obtained in the TOF system.
The apparatus of claim 12, wherein the downsampling module is configured to:

obtaining unwrapping coefficients of each pixel in the initial depth map of the second resolution by using the raw data of the plurality of modulation frequencies and the phase delay maps of the plurality of modulation frequencies;

calculating to obtain a depth value of each pixel in the initial depth map by using the unwrapping coefficient of each pixel in the initial depth map and the phase delay value of each pixel of the second resolution;

and carrying out downsampling processing on the initial depth map to obtain the depth map with the first resolution.
The apparatus of claim 16, wherein the downsampling module is configured to:

performing downsampling processing on the initial depth map by using a second downsampling parameter to obtain a depth map with a third resolution;

determining a gradient map corresponding to the depth map with the third resolution;

if the gradient map meets the gradient condition, determining that the depth map with the third resolution is the depth map with the first resolution;

If the gradient map does not meet the gradient conditions, updating the second downsampling parameters, downsampling the initial depth map based on the updated downsampling parameters until a target gradient map meeting the gradient conditions is obtained, and determining the depth map of the target gradient map as the depth map of the second resolution;

wherein the gradient condition is that the absolute value of the gradient of the pixel with the distance interval smaller than the second threshold value is smaller than or equal to the third threshold value.
The apparatus according to any one of claims 12 to 17, wherein the unwrapping coefficient determining module is configured to:

the unwrapping coefficient of the pixel k of the second resolution determining the modulation frequency j is:

wherein the modulation frequency j belongs to the plurality of modulation frequencies, round []To round-off operation, D _L For the depth value of the pixel k corresponding to the depth map of the first resolution, N _kj For the unwrap coefficient of the pixel k at the modulation frequency j,to the phase delay value of the pixel k at the modulation frequency j, U _j And the fuzzy distance corresponding to the modulation frequency j.
The apparatus according to any one of claims 12 to 18, wherein the depth map determination module is configured to:

Calculating a depth value of each pixel of the second resolution for each of the modulation frequencies using an unwrapping coefficient of each pixel of the second resolution and a phase delay value of each pixel of the second resolution;

determining a depth value error for each pixel of the second resolution for each of the modulation frequencies using the depth map of the first resolution and the depth values for each pixel of the second resolution for each of the modulation frequencies;

and calculating and obtaining a target depth map of the second resolution corresponding to the measured object by using the depth value errors of the pixels of the first resolution of each modulation frequency.
The apparatus of claim 19, wherein the depth map determination module is configured to:

the depth value error of the pixel k of the second resolution determining the modulation frequency j is: e (E) _kj ＝|D _L -D _kj |；

Wherein D is _L For the corresponding depth value of the pixel k in the depth map of the first resolution,D _kj for the depth value of the pixel k of the second resolution of the modulation frequency j,to the phase delay value of the pixel k at the modulation frequency j, N _kj To unwrap the coefficient of the pixel k at the modulation frequency j, U _j And for the fuzzy distance corresponding to the modulation frequency j, j takes a value of 1 to n, and n is the number of the modulation frequencies.
The apparatus of claim 19 or 20, wherein the depth map determination module is configured to:

for a modulation frequency j in the plurality of modulation frequencies, determining a target pixel with a depth value error larger than a fourth threshold corresponding to the modulation frequency j in pixels with the second resolution of the modulation frequency j, wherein j takes a value of 1 to n, and n is the number of the plurality of modulation frequencies;

updating the unwrapping coefficient of the target pixel of the second resolution of the modulation frequency j using depth values of neighboring pixels of the target pixel in the depth map of the first resolution;

updating a depth value of the target pixel of the second resolution of the modulation frequency j using the updated unwrapping coefficient of the target pixel and the phase delay value of the target pixel of each of the modulation frequencies;

and combining the depth values of the pixels except the target pixel in the pixels with the second resolution of the modulation frequency j with the updated depth values of the target pixel to obtain a target depth map with the second resolution of the modulation frequency j, corresponding to the measured object.
The apparatus of claim 21, wherein the fourth threshold value corresponding to the modulation frequency j is: thD _j ＝U _j *ph _ith /(2π*2)；U _j For the fuzzy distance corresponding to the modulation frequency j, ph _ith And j is 1 to n for the minimum phase distance between adjacent line segments of the unwrapping area in the phase delay diagrams of the plurality of modulation frequencies, and n is the number of the plurality of modulation frequencies.
A computing device that obtains a depth map, the computing device comprising a processor and a memory, wherein:

the memory stores computer instructions;

the processor executing the computer instructions to cause the computing device to perform the method of any of claims 1 to 11.
A computer readable storage medium storing computer instructions which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 11.