WO2020223980A1

WO2020223980A1 - Time-of-flight depth camera, and distance measurement method employing single-frequency modulation/demodulation

Info

Publication number: WO2020223980A1
Application number: PCT/CN2019/086293
Authority: WO
Inventors: 胡小龙; 朱亮
Original assignee: 深圳奥比中光科技有限公司
Priority date: 2019-05-09
Filing date: 2019-05-09
Publication date: 2020-11-12

Abstract

A time-of-flight depth camera (10), and a distance measurement method employing single-frequency modulation/demodulation. The time-of-flight depth camera (10) comprises: an emission module (11) having a light source (111) and used to emit a pulsed light beam toward an object under measurement (20); an acquisition module (12) comprising an image sensor (121) consisting of at least one pixel, each pixel comprising at least three taps and used to acquire a charge signal generated from the pulsed light beam reflected by the object under measurement (20) or a charge signal of background light; and a processing circuit (13) used to receive data of the charge signal acquired by the at least three taps, to assess the data of the charge signal so as to determine whether the data of the charge signal contains a charge signal of the reflected pulsed light beam, and to calculate, according to an assessment result, a time-of-flight of the pulsed light beam and/or a distance from the object under measurement (20). In the invention, each acquisition process of a frame of depth information only requires exposure and output of signal volumes of three taps, thereby significantly reducing overall power consumption and increasing a frame frequency for measurement.

Description

Time-of-flight depth camera and single-frequency modulation and demodulation distance measurement method

Technical field

The invention relates to the field of optical measurement, in particular to a time-of-flight depth camera and a single-frequency modulation and demodulation distance measurement method.

Background technique

The full name of ToF is Time-of-Flight, that is, time of flight. ToF ranging method is a technology that achieves precise ranging by measuring the round-trip flight time of light pulses between the transmitting/receiving device and the target object. In the ToF technology, the technology of directly measuring the optical time of flight is called dToF (direct-TOF); the emitted light signal is periodically modulated, and the phase delay of the reflected light signal relative to the emitted light signal is measured. The measurement technique that calculates the time of flight phase delay is called iToF (Indirect-TOF) technique. According to the different types of modulation and demodulation, it can be divided into continuous wave (CW) modulation and demodulation and pulse modulation (Pulse Modulated, PM) modulation and demodulation.

At present, CW-iToF technology is mainly used in measurement systems based on two-tap sensors. The core measurement algorithm is a four-phase modulation and demodulation method, which requires at least two exposures (in order to ensure measurement accuracy, four exposures are usually required. ) Can complete the collection of four phase data and output one frame of depth image, so it is difficult to obtain a higher frame rate. PM-iToF modulation technology is mainly used in four-tap sensors (three taps are used for signal acquisition and output, and one tap is used to release invalid electrons). The measurement distance of this measurement method is currently limited by the modulation and demodulation signal Pulse width. When remote measurement is required, the pulse width of the modem signal needs to be extended. However, the extension of the pulse width of the modem signal will increase the power consumption and decrease the measurement accuracy, and thus cannot meet the market demand. Aiming at the shortcomings of the two current modulation and demodulation methods, a new modulation and demodulation method is proposed to optimize the iToF technical solution.

Summary of the invention

In order to solve the existing problems, the present invention provides a time-of-flight depth camera and a single-frequency modulation and demodulation distance measurement method.

In order to solve the above problems, the technical solutions adopted by the present invention are as follows:

A time-of-flight depth camera, including: a transmitting module, including a light source, for emitting pulsed beams to an object to be measured; and an acquisition module, including an image sensor composed of at least one pixel, each of which includes at least 3 taps , The tap is used to collect the charge signal generated by the reflected pulse beam reflected by the object under test or the charge signal of the background light; the processing circuit is used to receive the charge signal data of the at least 3 taps; The data of the charge signal is judged to determine whether the charge signal of the reflected pulse beam is included in the data of the charge signal; the flight time of the pulse beam and/or the to-be-measured is calculated according to the judgment result The distance between objects.

In an embodiment of the present invention, the processing circuit calculates the flight time of the pulse beam according to the following formula:

Wherein, QA is the charge amount of the charge signal including the reflected pulse beam collected by the first tap acquired after the judgment; QB is the second tap acquired after the judgment The collected charge amount of the charge signal containing the reflected pulse beam; QO is the charge amount of the charge signal collected by the tap only containing the background light; m=n-1, where n is the QA The sequence number of the corresponding tap; Th is the pulse width of the pulse acquisition signal of each tap. The judgment includes a single-tap maximum method, that is, the first tap with the largest amount of charge of the charge signal among the at least three taps is sequentially searched and acquired, if the second tap before the first tap is greater than the first tap The charge amount of the charge signal of the third tap after one tap is large, and the charge amount of the charge signal collected by the second tap and the first tap is the QA and the QB respectively; if the first tap before the first tap The second tap is smaller than the charge signal of the third tap after the first tap, and the charge amount of the charge signal collected by the first tap and the third tap is the QA and the QB. The judgment includes the adjacent tap and maximum value method, that is, after sequentially calculating the charge amounts of the charge signals of adjacent taps and searching for the maximum value, the maximum value corresponds to the value of the charge signals collected by the two taps. The amount of charge is the QA and the QB respectively. The QO is obtained by at least one of the following ways: taking the charge amount of the collected charge signal of a tap behind the tap corresponding to the QB; or, taking the QA corresponding to the front of the tap The charge quantity of the charge signal collected by one tap; or, take the average value of the charge quantity of the charge signals collected by all the taps except the tap corresponding to the QA and the QB; or, take the QA , The tap corresponding to the QB and the average value of the charge quantities of the charge signals collected by all the taps except one of the taps after the tap corresponding to the QB.

The present invention also provides a single-frequency modulation and demodulation distance measurement method, which includes: using a light source to emit a pulsed light beam to an object to be measured; using an image sensor composed of at least one pixel to collect the reflection of the object to be measured Reflecting the charge signal of the pulse beam, each of the pixels includes at least 3 taps, the taps are used to collect the charge signal or the charge signal of the background light; receiving the charge signal data of the at least 3 taps; Judge the data of the charge signal to determine whether the charge signal of the reflected pulse beam is included in the charge signal data; calculate the flight time of the pulse beam and/or the waiting time according to the judgment result The distance of the measured object.

In an embodiment of the present invention, the flight time is calculated according to the following formula:

Wherein, QA is the charge amount of the charge signal including the reflected pulse beam collected by the first tap acquired after the judgment; QB is the second tap acquired after the judgment The collected charge amount of the charge signal containing the reflected pulse beam; QO is the charge amount of the charge signal collected by the tap only containing the background light; m=n-1, where n is the QA The sequence number of the corresponding tap; Th is the pulse width of the pulse acquisition signal of each tap. The judgment includes a single-tap maximum method, that is, the first tap with the largest amount of charge of the charge signal among the at least three taps is sequentially searched and acquired, if the second tap before the first tap is greater than the first tap The charge amount of the charge signal of the third tap after one tap is large, and the charge amount of the charge signal collected by the second tap and the first tap is the QA and the QB respectively; if the first tap before the first tap The charge amount of the charge signal of the second tap is smaller than that of the third tap after the first tap, and the charge amount of the charge signal collected by the first tap and the third tap is the QA and the QB, respectively. The judgment includes the adjacent tap and maximum value method, that is, after sequentially calculating the charge amounts of the charge signals of adjacent taps and searching for the maximum value, the maximum value corresponds to the value of the charge signals collected by the two taps. The amount of charge is the QA and the QB in the order of the tap number. The QO is obtained by at least one of the following ways: taking the charge amount of the collected charge signal of a tap behind the tap corresponding to the QB; or, taking the QA corresponding to the front of the tap The charge quantity of the charge signal collected by one tap; or, take the average value of the charge quantity of the charge signals collected by all the taps except the tap corresponding to the QA and the QB; or, take the QA , The tap corresponding to the QB and the average value of the charge quantities of the charge signals collected by all the taps except one of the taps after the tap corresponding to the QB.

The beneficial effects of the present invention are: provide a time-of-flight depth camera and a single-frequency modulation and demodulation distance measurement method, which expands the measurement distance under the same pulse width compared to the existing PM measurement scheme; compared to CW-iToF The measurement scheme requires only one exposure to output the signal of three taps to obtain one frame of depth information, thus significantly reducing the overall measurement power consumption and increasing the measurement frame rate.

Description of the drawings

Fig. 1 is a schematic diagram of the principle of a time-of-flight camera according to an embodiment of the present invention.

2 is a schematic diagram of a method for transmitting and collecting optical signals of a time-of-flight camera according to an embodiment of the present invention.

Fig. 3 is a schematic diagram of light signal emission and collection of a time-of-flight camera according to another embodiment of the present invention.

Detailed ways

In order to make the technical problems, technical solutions, and beneficial effects to be solved by the embodiments of the present invention clearer and clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention.

It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be used for fixing or circuit connection.

It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top" , "Bottom", "Inner", "Outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying. The device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present invention, "a plurality of" means two or more than two, unless otherwise specifically defined.

Fig. 1 is a schematic diagram of a time-of-flight camera according to an embodiment of the present invention. The time-of-flight camera 10 includes a transmitting module 11, a collecting module 12, and a processing circuit 13. The transmitting module 11 provides a transmitted light beam 30 to the target space to illuminate an object 20 in the space, and at least part of the transmitted light beam 30 is reflected by the object 20 Afterwards, a reflected light beam 40 is formed. At least part of the reflected light beam 40 is collected by the collection module 12. The processing circuit 13 is respectively connected with the transmission module 11 and the collection module 12 to synchronize the trigger signals of the transmission module 11 and the collection module 12 to calculate The time required for the light beam to be emitted by the transmitter module 11 and received by the collection module 12, that is, the flight time t between the transmitted light beam 30 and the reflected light beam 40, further, the total light flight distance D of the corresponding point on the object can be determined by Calculate:

D=c·t (1)

Among them, c is the speed of light.

The transmitting module 11 includes a light source 111, a beam modulator 112, a light source driver (not shown in the figure), and the like. The light source 111 can be a light source such as a light emitting diode (LED), an edge emitting laser (EEL), a vertical cavity surface emitting laser (VCSEL), or a light source array composed of multiple light sources. The light beam emitted by the light source can be visible light or infrared light. , UV light, etc. The light source 111 emits light beams outward under the control of the light source driver (which can be further controlled by the processing circuit 13). For example, in one embodiment, the light source 111 emits a pulsed light beam at a certain frequency under control, which can be used in the direct time flight method. In (Direct TOF) measurement, the frequency is set according to the measurement distance, for example, it can be set to 1MHz～100MHz, and the measurement distance is from several meters to several hundred meters; in one embodiment, the light source 111 is controlled to modulate the beam amplitude. It can be used in indirect time-of-flight (Indirect TOF) measurement by emitting pulsed beam, square wave beam, sine wave beam and other beams. It is understandable that a part of the processing circuit 13 or a sub-circuit independent of the processing circuit 13 can be used to control the light source 111 to emit related light beams, such as a pulse signal generator.

The beam modulator 112 receives the light beam from the light source 111 and emits a spatially modulated beam, such as a flood beam with uniform intensity distribution or a patterned beam with uneven intensity distribution. It is understandable that the uniform distribution here is a relative concept, not absolute uniformity. Generally, a slightly lower beam intensity at the edge of the field of view is allowed, and the intensity of the imaging area in the middle can also be within a certain threshold. Changes, for example, can allow for intensity changes not exceeding 15% or 10%. In some embodiments, the beam modulator 112 is also used to expand the received beam to expand the angle of view.

The acquisition module 12 includes an image sensor 121, a lens unit 122, and may also include a filter (not shown in the figure). The lens unit 122 receives and reflects at least part of the spatially modulated light beams reflected by the object and images at least part of the image. On the sensor 121, the filter needs to select a narrow-band filter that matches the wavelength of the light source to suppress the background light noise in the other wavelength bands. The image sensor 121 may be an image sensor composed of charge coupled device (CCD), complementary metal oxide semiconductor (CMOS), avalanche diode (AD), single photon avalanche diode (SPAD), etc. The size of the array represents the resolution of the depth camera , Such as 320x240, etc. Generally, connected to the image sensor 121 also includes a readout circuit composed of one or more of a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (ADC) and other devices (not shown in the figure). ).

Generally, the image sensor 121 includes at least one pixel, and each pixel includes multiple taps (tap, used to store and read or discharge the charge signal generated by incident photons under the control of the corresponding electrode), for example, including 3 taps , To read the charge signal data.

In some embodiments, the time-of-flight depth camera 10 may also include a drive circuit, a power supply, a color camera, an infrared camera, an IMU and other devices, which are not shown in the figure. The combination with these devices can achieve richer functions, such as 3D texture modeling, infrared face recognition, SLAM and other functions. The time-of-flight depth camera 10 can be embedded in electronic products such as mobile phones, tablet computers, and computers.

The processing circuit 13 can be an independent dedicated circuit, such as a dedicated SOC chip, FPGA chip, ASIC chip, etc. composed of CPU, memory, bus, etc., or a general processing circuit, such as when the depth camera is integrated into a mobile phone, In smart terminals such as televisions and computers, the processing circuit in the terminal can be used as at least a part of the processing circuit 13. In some embodiments, the processing circuit 13 is used to provide a modulation signal (transmission signal) required when the light source 111 emits laser light, and the light source emits a pulsed beam to the object under the control of the modulation signal; in addition, the processing circuit 13 also provides an image sensor 121 The demodulated signal (collection signal) of the tap in each pixel. The tap collects the charge signal generated by the beam containing the reflected pulse beam reflected by the object under the control of the demodulated signal. Generally, except for the reflection of the object In addition to the reflected pulse beam, there are some background light, interference light and other light beams; the processing circuit 13 can also provide auxiliary monitoring signals, such as temperature sensing, overcurrent, overvoltage protection, fall protection, etc.; the processing circuit 13 can also It is used to save the raw data collected by each tap in the image sensor 121 and perform corresponding processing to obtain the specific location information of the object to be measured. The modulation and demodulation method, control, processing and other functions performed by the processing circuit 13 will be described in detail in the embodiments of FIG. 2 to FIG. 3. For ease of explanation, the PM-iTOF modulation and demodulation method is used as an example for description.

2 is a schematic diagram of a method for transmitting and collecting optical signals of a time-of-flight camera according to an embodiment of the present invention. Figure 2 exemplarily shows the timing diagram of the laser emission signal (modulation signal), received signal, and acquisition signal (demodulation signal) within two frame periods T. The meaning of each signal is: Sp represents the pulse emission of the light source Signal, each pulse emission signal represents a pulse beam; Sr represents the reflected light signal of the pulsed light reflected back by the object, and each reflected light signal represents the corresponding pulse beam reflected back by the object to be measured, which is in the time line (Figure The horizontal axis) has a certain delay relative to the pulse emission signal. The delay time t is the flight time of the pulse beam to be calculated; S1 represents the pulse collection signal of the first tap of the pixel, and S2 represents the pulse collection signal of the second tap of the pixel , S3 represents the pulse collection signal of the third tap of the pixel, each pulse collection signal represents the charge signal (electron) generated by the pixel during the time period corresponding to the signal is collected by the tap; Tp=N×Th, where N is the electron of the participating pixel The number of taps collected.

The entire frame period T is divided into two time periods Ta and Tb, where Ta represents the time period during which each tap of the pixel performs charge collection and storage, and Tb represents the time period during which the charge signal is read out. In the charge collection and storage period Ta, there is a phase delay time (n-1)×Th between the collected signal pulse of the n-th tap and the laser emission signal pulse. When the reflected light signal is reflected by the object back to the pixel, each The tap collects the electrons generated on the pixel during its pulse period. In this embodiment, the collection signal of the first tap is triggered synchronously with the laser emission signal. When the reflected light signal is reflected back to the pixel by the object, the first tap, the second tap, and the third tap respectively perform charge collection and storage in sequence. The charges q1, q2, and q3 are obtained respectively, and a pulse period Tp is thus completed. For the case of 3 taps, Tp=3Th. In the embodiment shown in FIG. 2, there are two pulse periods Tp in a single frame period, and a total of 2 laser pulse signals are emitted. Therefore, the total amount of charge collected and read out by each tap in the Tb period is the total charge collected twice The light signal corresponds to the sum of the electric charges. It can be understood that in a single frame period, the pulse period Tp or the number of laser pulse signal transmissions can be K times, and K is not less than 1, but can be as high as tens of thousands, or even higher. The number is determined according to actual requirements. In addition, the number of pulses in different frame periods can also vary.

Therefore, the total amount of charge collected and read out by each tap in the Tb period is the sum of the corresponding charges of the optical signal collected by each tap multiple times in the entire frame period T. The total charge amount of each tap in a single frame period can be expressed as follows:

Qi=∑qi,i=1,2,3 (2)

According to formula (2), the total charge in the single frame period of the first tap, second tap, and third tap is Q1, Q2, and Q3.

In the traditional modulation and demodulation method, the measurement range is limited to a single pulse width time Th, that is, it is assumed that the reflected light signal is collected by the first tap and the second tap (the first tap and the second tap will also collect ambient light at the same time). Signal), the third tap is used to collect the ambient light signal, so based on the total charge collected by each tap, the processing unit can calculate the total light flight distance of the pulsed light signal from emission to reflection to the pixel according to the following formula :

Furthermore, the spatial coordinates of the target can be calculated according to the optical and structural parameters of the camera.

The advantage of the traditional modulation and demodulation method is that the calculation is simple, but the disadvantage is that the measurement range is limited, the measured flight time is limited to within Th, and the corresponding maximum flight distance measurement range is limited to c×Th.

In order to increase the measurement distance, the present invention provides a new modulation and demodulation method. Fig. 2 is a schematic diagram of optical signal transmission and collection according to an embodiment of the present invention. At this time, the reflected optical signal can not only fall on the first tap and the second tap, but can also fall on the second tap and the third tap. , It is even allowed to fall on the third tap and the first tap within the next pulse period Tp (for at least two pulse periods Tp or more). "Falling on the tap" mentioned here means that it can be tapped. Since the total charges read in the time period Tb are Q1, Q2, and Q3, unlike the traditional modulation and demodulation method, the present invention does not limit the tap or even the period of receiving the reflected light signal.

Considering that the amount of charge collected by the tap that receives the reflected light signal is greater than that of the tap containing only the background light signal, the processing circuit will judge the three total charges Q1, Q2, and Q3 acquired to determine that the capture contains reflection The light signal excites the electronic taps and/or acquires the taps containing only the background signal. In actual use, there may be electronic crosstalk between the taps. For example, some reflected light signals may enter into the taps originally used to obtain the background signal. Errors will be allowed and are also within the scope of protection of this program. Assuming that after the judgment, the two total charges including the reflected light signal in turn (receiving the reflected light signal in chronological order) are respectively marked as QA and QB, and the total charge including only the background light signal is recorded as QO, then for the three-tap For image sensors, there are three possibilities:

(1) QA=Q1, QB=Q2, QO=Q3;

(2) QA=Q2, QB=Q3, QO=Q1;

(3) QA=Q3, QB=Q1 (the next pulse period Tp), QO=Q2;

Subsequently, the processing circuit can calculate the flight time of the optical signal according to the following formula:

The m in the formula reflects the delay of the tap where the reflected light signal falls for the first time relative to the first tap. For the above three cases, m is 0, 1, 2 respectively. That is, if the reflected light signal falls into the n-th tap first, then m=n-1. n refers to the sequence number of the tap corresponding to the QA, the phase delay time of the tap with the sequence number n relative to the transmitted light pulse signal is (n-1)×Th; j refers to the reflected pulse beam being pulsed first Tap collection in the j-th pulse period after the beam is emitted (the pulse period of the emission pulse is the 0th pulse period after the emission pulse beam); Th is the pulse width of the pulse collection signal of each tap; Tp is the pulse period, Tp= N×Th, where N is the number of taps participating in pixel electronic collection.

Comparing formula (4) with formula (3), it can be clearly seen that the measurement distance has been extended, and the maximum measurement flight distance is expanded from c·Th in the traditional method to c×Tp=c×N×Th in this application, where N is the number of taps participating in the electronic collection of pixels, and its value is 3 in this example. Therefore, compared with the traditional modulation and demodulation method, the above method achieves 3 times the measurement distance of the traditional method through the judgment mechanism.

The key to the above modulation and demodulation method is how to determine the tap into which the reflected light signal falls. This application provides the following judgment methods:

(1) Single tap maximum value method. Find the tap (denoted as Node _x ) with the largest output signal (total charge) from tap 1 to tap N (N=3 in the above embodiment), and then follow Node ₁ →Node ₂ →…→Node _N →Node ₁ →… referred to a tap sequence before Node _x is Node _w; note after a tap for the Node _x Node _y. If the total charge of Node _w and Node _y Q _w ≥ Q _y , then Node _{w is} tap A; if Q _w <Q _y , Node _{x is} tap A.

(2) Adjacent tap and maximum value method. First, calculate the sum of the total charges of adjacent taps in the order of Node ₁ →Node ₂ →…→Node _N →Node ₁ →…, that is, Sum ₁ =Q ₁ +Q ₂ ,Sum ₂ =Q ₂ +Q ₃ ,... , Sum _N = Q _N + Q ₁ , search for the maximum item Sum _n among them, then tap n is tap A, and the tap after tap n is tap B.

After completing the confirmation of taps A and B, there are at least the following four calculation methods for the background signal:

(2) Background behind B; that is, the semaphore of one tap after the B tap is taken as the background semaphore.

(3) A front background; that is, the semaphore of one tap before the A tap is taken as the background semaphore.

(4) Average background; that is, the average value of all the taps except the A and B taps is taken as the background signal.

(5) Subtract one average background; that is, take the average value of the signal quantities of all the taps except one tap after the A and B taps as the background signal quantity.

It should be noted that when N=3, that is, there are only 3 taps, method (4) is not desirable, methods (1)～(3) are equivalent; when k=4, methods (3) and (4) are equivalent , In order to minimize the crosstalk of the signal volume, you can give priority to method (3). When k>4, method (4) can be selected first.

The foregoing embodiment introduces the modulation and demodulation method based on 3-tap pixels. It can be understood that this modulation and demodulation method is also applicable to pixels with more taps, that is, N>3. For example, for a 4-tap pixel, the maximum 4Th measurement distance, for 5-tap pixels, a maximum measurement distance of 5Th can be achieved. Compared with the traditional PM-iTOF measurement scheme, this measurement method extends the farthest measurement flight time from the pulse width time Th to the entire pulse period Tp, which is called a single-frequency full-period measurement scheme here.

Although the above-mentioned modulation and demodulation method achieves a measurement distance increase of N-1 times, it still cannot satisfy the measurement of a longer distance. For example, based on the modulation and demodulation method of 3-tap pixels, when the flight time corresponding to the object distance exceeds 3Th, the reflected light signal in a certain pulse period Tp will first fall on the tap in the subsequent pulse period. Neither formula (3) nor formula (4) can accurately measure flight time or distance. For example, when the reflected light signal in a certain pulse period Tp first falls on the n-th tap in the subsequent j-th pulse period, the flight time of the light signal corresponding to the real object is shown in the following equation:

Where m=n-1, n is the sequence number of the tap corresponding to QA. Since the total charge of each tap integrates the accumulated charge in the pulse period, the specific value of j cannot be distinguished only from the total charge of each tap output, which causes confusion in distance measurement.

Fig. 3 is a schematic diagram of light signal emission and collection of a time-of-flight camera according to another embodiment of the present invention, which can be used to solve the above-mentioned confusion problem. The difference from the embodiment shown in FIG. 2 is that this embodiment adopts a multi-frequency modulation and demodulation method, that is, different modulation and demodulation frequencies are used in adjacent frames. In this embodiment, for ease of explanation, two adjacent frame periods are taken as an example for description. In adjacent frame periods, the number of pulse transmissions K=2 (it can also be multiple times, and the number of times in different frames can also be different). The number of pixel taps N=3, the pulse period TPi is Tp1, Tp2, the pulse width Thi is Th1, Th2, and the accumulated charge of each pulse of the three taps is q11, q12, q21, q22, q31, q32, according to the formula (2) The total charge available is Q11, Q12, Q21, Q22, Q31, Q32.

It is assumed that the distance of the object in the period of adjacent frames (or consecutive multiple frames) is unchanged, so t in the period of adjacent frames is the same. After the processing circuit receives the total charge of each tap, it uses the modulation and demodulation method shown in Figure 2 to measure the distance d (or t) in each frame period, and calculates the distance d (or t) in each frame period through the above judgment method. QAi, QBi and QOi, i represent the i-th frame period, i=1, 2 in this embodiment. In order to expand the measurement range, allow the reflected light signal to fall on the tap in the subsequent pulse period. It is assumed that the reflected light signal on a certain pixel in the i-th frame period first falls on the jith one after the pulse period where the light pulse is emitted. At the mi-th tap in the pulse period (the pulse period of the transmitted pulse is the 0th pulse period after the transmitted pulse beam is emitted), the corresponding flight time can be expressed as follows according to equation (5):

Considering that the distance between objects in adjacent frame periods is unchanged, the following formula holds for the case of two consecutive frames in this embodiment:

(x1+m1)Th1+j1·Tp1=(x2+m2)Th2+j2·Tp2 (7)

among them,

For continuous multiple frames (assuming continuous w frames, that is, i=1, 2,...,w), the following formula holds:

(x1+m1)Th1+j1·Tp1=(x2+m2)Th2+j2·Tp2

=...=xw+mwThw+jw·Tpw (8)

It is understandable that when w=1, it corresponds to the single-frequency full-period measurement scheme described above. When w>1, according to the remainder theorem or by traversing various ji combinations within the maximum measurement distance, find a group of ji combinations with the smallest ti variance at each modulation and demodulation frequency as the solution value to complete the solution for ji; The final flight time or measured distance is obtained by weighted average of the flight time or measured distance calculated under each group of frequencies. Using multi-frequency modulation and demodulation methods,

The maximum measurement flight time is expanded to:

t _max =LCM(Tp ₁ ,Tp ₂ ,...,Tp _w ) (9)

The maximum measurement flight distance is expanded to:

D _max =LCM(D _max1 ,D _max2 ,...,D _maxw ) (10)

Among them, Dmax _i = C·Tpi, LCM (Lowest Common Multiple) means to take the “least common multiple” (here, the “least common multiple” is a generalized expansion of the least common multiple of the integer domain, and LCM(a,b) is defined as The smallest real number divisible by real numbers a and b).

Assuming that in the embodiment shown in FIG. 3, T _p = 15 ns, the maximum measured flying distance is 4.5 m; if T _p = 20 ns, the maximum measured flying distance is 6 m. If a multi-frequency modulation and demodulation method is used, for example, in one embodiment, T _p1 =15ns, T _p2 =20ns, the least common multiple of 15ns and 20ns is 60ns, and the maximum measurement distance corresponding to 60ns is 18m, which corresponds to the farthest measurement target The distance can reach 9m.

It can be understood that, although in the embodiment shown in FIG. 3, the distance of the object is calculated through at least two frames of data, in one embodiment, the forward and backward frames can be extended so as not to reduce the number of collected frames. For example, in the case of dual-frequency modulation and demodulation method to obtain a single flight time measurement through the front and rear frames, the first flight time is calculated from

frames

1 and 2, and the second flight time is calculated from

frames

2, 3, and so on. , So as not to reduce the measurement frame rate.

It is understandable that in the above-mentioned multi-frequency modulation and demodulation method, different frequency combinations can be used to meet the requirements of different measurement scenarios. For example, by increasing the number of measurement frequencies, the accuracy of the final distance resolution can be improved. In order to be able to dynamically meet the measurement requirements in different measurement scenarios, in an embodiment of the present invention, the processing circuit will adaptively adjust the number of modulation and demodulation frequencies and specific frequency combinations through result feedback to meet the requirements in different measurement scenarios as much as possible. . Specifically, in one embodiment, the processing circuit calculates the target distance after calculating the current distance (or flight time) of the object, and when most of the measured target distances are relatively close, a smaller frequency can be used to measure In order to ensure a higher frame rate and reduce the impact of target motion on the measurement result, when there are more distant targets in the measurement target, the number of measurement frequencies can be appropriately increased or the combination of measurement frequencies can be adjusted to ensure measurement accuracy.

In addition, for the method of the present invention and the content described in the embodiments, it should be noted that any multi-frequency long-range and single-frequency full-period measurement scheme based on sensors with more than three taps, regardless of the waveform of the modulation and demodulation signal Whether it is continuous or intermittent within the exposure time range, the measurement sequence of different frequency modulation and demodulation signals and the fine adjustment of the modulation frequency within the same exposure time should be within the scope of protection of this patent, which is done to explain the principle of this patent. The example description or analysis algorithm is only an example description of this patent, and should not be regarded as a limitation to the content of this patent. For those skilled in the art to which the present invention belongs, without departing from the concept of the present invention, several equivalent substitutions or obvious modifications can be made, and the same performance or use should be regarded as belonging to the protection scope of the present invention.

For the time-of-flight depth camera in each of the above embodiments, since it is based on iToF technology, active light emission is required. When multiple iToF depth cameras work at a relatively close distance at the same time, the device's acquisition module will not only receive object reflections The light signal from the light-emitting unit of the device itself will also receive the emitted light or reflected light from other devices. These light signals from other devices will interfere with the amount of electrons collected between each tap, and then affect the final The accuracy and precision of the target distance measurement are adversely affected. In view of this problem, the present invention provides the following ways to eliminate related interference between multiple devices.

(1) Frequency conversion scheme. The so-called frequency conversion scheme means that in the actual measurement process, when the frequency of the modulation and demodulation signal is set to f _m0 , the actually used frequency of the modulation and demodulation signal is f _m =f _m0 +Δf. Where Δf is a random frequency offset. In this way, there can be at least one random deviation in the operating frequency of each single machine, thereby significantly reducing the mutual interference between the devices.

(2) Random exposure time. Relative to the entire working time, the camera's exposure time is relatively limited. Taking dual frequency as an example, acquiring each depth frame data requires only 2 exposures at most. When the single exposure time is 1ms and the depth frame rate is 30fps, the exposure time in the entire working time accounts for only 6%. The selection of exposure time is usually evenly distributed throughout the working hours. In order to reduce the mutual interference between devices, a random offset can be added to the uniform distribution of exposure time, so that the exposure and imaging time between different devices can be exhausted. May be staggered to avoid mutual interference. In order to ensure that the time intervals for acquiring images are as the same as possible, you can choose to use the same time offset in a relatively long working time slice (such as 1s) to ensure that the image time intervals in this time slice are the same.

The beneficial effect achieved by the present invention is to get rid of the contradiction that the pulse width is directly proportional to the measurement distance and power consumption in the current PM-iToF measurement scheme, but is negatively related to the measurement accuracy; the expansion of the measurement distance is no longer limited to the pulse width, Thereby, it can maintain lower measurement power consumption and higher measurement accuracy even with a longer measurement distance. Compared with the CW-iToF measurement scheme, the single set of modulation and demodulation frequency in this scheme only needs one exposure to output three-tap signal to obtain one frame of depth information, thus significantly reducing the overall measurement power consumption and increasing Measure the frame rate. Therefore, this solution has obvious advantages over the existing iToF technical solutions.

The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be considered that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art to which the present invention belongs, without departing from the concept of the present invention, several equivalent substitutions or obvious modifications can be made, and the same performance or use should be regarded as belonging to the protection scope of the present invention.

Claims

A time-of-flight depth camera, characterized in that it comprises:

The transmitting module, including a light source, is used to emit a pulsed beam to the object to be measured;

The acquisition module includes an image sensor composed of at least one pixel, each of the pixels includes at least 3 taps, and the taps are used to collect the charge signal or background generated by the reflected pulse beam reflected by the object under test Light charge signal;

A processing circuit for receiving the charge signal data of the at least 3 taps; judging the charge signal data to determine whether the charge signal data includes the charge signal of the reflected pulse beam; according to The judgment result calculates the flight time of the pulse beam and/or the distance of the object to be measured.
The time-of-flight camera of claim 1, wherein the processing circuit calculates the flight time of the pulse beam according to the following formula:

Wherein, QA is the charge amount of the charge signal including the reflected pulse beam collected by the first tap acquired after the judgment; QB is the second tap acquired after the judgment The collected charge amount of the charge signal containing the reflected pulse beam; QO is the charge amount of the charge signal collected by the tap only containing the background light; m=n-1, where n is the QA The sequence number of the corresponding tap; Th is the pulse width of the pulse acquisition signal of each tap.
The time-of-flight camera according to claim 2, wherein the judgment includes a single-tap maximum method, that is, the first tap with the largest charge amount of the charge signal among the at least three taps is sequentially searched and obtained If the charge amount of the charge signal of the second tap before the first tap is greater than the charge amount of the charge signal of the third tap after the first tap, the charge amounts of the charge signals collected by the second tap and the first tap are respectively The QA, the QB; if the charge amount of the charge signal of the second tap before the first tap is smaller than the charge signal of the third tap after the first tap, then the charge signals collected by the first tap and the third tap According to the sequence of the tap sequence number, the charge amount is the QA and the QB respectively.
The time-of-flight camera according to claim 2, wherein the judgment includes the adjacent tap and the maximum value method, that is, after sequentially calculating the charge amount of the charge signals of the adjacent taps and searching for the maximum value, The charge amounts of the charge signals collected by the two taps corresponding to the maximum value term are the QA and the QB respectively.
The time-of-flight depth camera of claim 2, wherein the QO is obtained by at least one of the following methods:

Take the charge amount of the charge signal collected by the one tap behind the tap corresponding to the QB; or, take the charge amount of the charge signal collected by the first tap of the QA corresponding to the tap; or, take The QA, the QB corresponding to the tap and the QB corresponding to the tap and the QB corresponding to the average value of the charge amount of the charge signals collected by all the taps except the tap; The average value of the charge amount of the charge signals collected by all the taps except one of the taps behind the tap.
A distance measurement method for single-frequency modulation and demodulation, which is characterized in that it includes:

Use a light source to emit pulsed beams to the object to be measured;

An image sensor consisting of at least one pixel is used to collect the charge signal of the reflected pulse beam reflected by the object to be measured. Each pixel includes at least 3 taps, and the taps are used to collect the charge signal or background. Light charge signal;

Receiving the data of the charge signal of the at least 3 taps;

Judging the data of the charge signal to determine whether the data of the charge signal includes the charge signal of the reflected pulse beam;

Calculate the flight time of the pulse beam and/or the distance of the object to be measured according to the judgment result.
The distance measurement method for single-frequency modulation and demodulation according to claim 6, wherein the flight time is calculated according to the following formula:

Wherein, QA is the charge amount of the charge signal including the reflected pulse beam collected by the first tap acquired after the judgment; QB is the second tap acquired after the judgment The collected charge amount of the charge signal containing the reflected pulse beam; QO is the charge amount of the charge signal collected by the tap only containing the background light; m=n-1, where n is the QA The sequence number of the corresponding tap; Th is the pulse width of the pulse acquisition signal of each tap.
The single-frequency modulation and demodulation distance measurement method according to claim 7, wherein the judgment comprises a single-tap maximum method, that is, the charge amount of the charge signal in the at least three taps is sequentially searched and obtained. The largest first tap. If the charge amount of the charge signal of the second tap before the first tap is greater than the charge signal of the third tap after the first tap, then the charge signal collected by the second tap and the first tap is The charge amounts are the QA and QB respectively; if the charge amount of the charge signal of the second tap before the first tap is smaller than the charge signal of the third tap after the first tap, then the first tap and the third tap The charge amount of the collected charge signal is the QA and the QB respectively.
7. The single-frequency modulation and demodulation distance measurement method according to claim 7, wherein the judgment includes adjacent taps and the maximum value method, that is, the charge amount of the charge signals of the adjacent taps is calculated in sequence and searched for. The maximum value item of the maximum value item, the charge amount of the charge signal collected by the two taps corresponding to the maximum value item is the QA and the QB in the order of the tap sequence numbers.
8. The single-frequency modulation and demodulation distance measurement method according to claim 7, wherein the QO is obtained in at least one of the following ways:

Take the charge amount of the charge signal collected by the one tap behind the tap corresponding to the QB; or, take the charge amount of the charge signal collected by the first tap of the QA corresponding to the tap; or, take The QA, the QB corresponding to the tap and the QB corresponding to the tap and the QB corresponding to the average value of the charge amount of the charge signals collected by all the taps except the tap; The average value of the charge amount of the charge signals collected by all the taps except one of the taps behind the tap.