CN110361751B

CN110361751B - Time flight depth camera and distance measuring method for reducing noise of single-frequency modulation and demodulation

Info

Publication number: CN110361751B
Application number: CN201910518104.4A
Authority: CN
Inventors: 王飞; 朱亮; 胡小龙
Original assignee: Orbbec Inc
Current assignee: Orbbec Inc
Priority date: 2019-06-14
Filing date: 2019-06-14
Publication date: 2021-04-30
Anticipated expiration: 2039-06-14
Also published as: WO2020248334A1; CN110361751A

Abstract

The invention provides a time flight depth camera and a distance measuring method for reducing noise of single-frequency modulation and demodulation, wherein the time flight depth camera comprises the following steps: the transmitting module comprises a light source and is used for transmitting a pulse light beam to the object to be detected; the acquisition module comprises an image sensor consisting of at least one pixel, each pixel comprises at least 3 taps, and the taps are used for acquiring charge signals generated by reflected pulse beams reflected by the object to be measured and/or charge signals of background light; and the processing circuit is used for controlling at least 3 taps to alternately acquire the charge signals among at least 3 frame periods of the macrocycle and receiving data of the charge signals to calculate the flight time of the pulse light beam and/or the distance of the object to be measured. The extension of the measurement distance is not limited by the pulse width any more, and in addition, fixed noise caused by mismatch between taps or between readout circuits due to process manufacturing errors and the like is reduced or eliminated by a tap rotation acquisition method.

Description

Time flight depth camera and distance measuring method for reducing noise of single-frequency modulation and demodulation

Technical Field

The invention relates to the technical field of optical measurement, in particular to a time flight depth camera and a distance measurement method for reducing noise of a single-frequency modulation solution.

Background

ToF is known collectively as Time-of-Flight, and ToF ranging is a technique for achieving accurate ranging by measuring the round-trip Time of Flight of an optical pulse between a transmitting/receiving device and a target object. The technique of directly measuring the optical time of flight in the ToF technique is called dtofs (direct-ToF); the measurement technique of periodically modulating the emitted light signal, measuring the phase delay of the reflected light signal relative to the emitted light signal, and calculating the time of flight from the phase delay is called the iToF (index-TOF) technique. According to the difference of the modulation and demodulation type, the modulation and demodulation method can be divided into a Continuous Wave (CW) modulation and demodulation method and a Pulse Modulated (PM) modulation and demodulation method.

At present, the CW-iToF technology is mainly applied to a measurement system constructed based on a two-tap sensor, a core measurement algorithm is a four-phase modulation and demodulation mode, and at least two exposures (usually four exposures are required to ensure measurement accuracy) are required to complete the acquisition of four-phase data and output a frame depth image, so that a higher frame frequency is difficult to obtain. The PM-iToF modulation technology is mainly applied to a four-tap pixel sensor (three taps are used for signal acquisition and output, and one tap is used for ineffective electron release), the measurement distance of the measurement means is limited by the pulse width of a modem signal at present, when remote measurement is required, the pulse width of the modem signal needs to be prolonged, and the prolonging of the pulse width of the modem signal can cause the increase of power consumption and the reduction of measurement accuracy.

In addition, for a multi-tap pixel sensor, mismatch between taps or between readout circuits due to process manufacturing errors and the like is often faced, so that Fixed-Pattern Noise (FPN) is introduced, and measurement accuracy is further affected.

Disclosure of Invention

The invention provides a time flight depth camera and a distance measuring method for reducing noise of single-frequency modulation and demodulation, aiming at solving the problems in the prior art.

The technical problem of the invention is solved by the following technical scheme:

a time-of-flight depth camera, comprising: the transmitting module comprises a light source and is used for transmitting a pulse light beam to the object to be detected; the acquisition module comprises an image sensor consisting of at least one pixel, each pixel comprises at least 3 taps, and the taps are used for acquiring charge signals generated by reflected pulse beams reflected by the object to be measured and/or charge signals of background light; and the processing circuit is used for controlling the at least 3 taps to acquire the charge signals in turn between at least 3 frame periods of a macrocycle and receiving data of the charge signals to calculate the flight time of the pulse light beam and/or the distance of the object to be measured.

In one embodiment of the invention, the processing circuitry calculates the time of flight of the pulsed light beam according to:

wherein Q is₁₁、Q₂₁、Q₃₁、Q₁₂、Q₂₂、Q₃₂、Q₁₃、Q₂₃、Q₃₃Respectively representing the signals acquired by 3 taps within 3 consecutive frame periods of the macrocycle. The processing circuit controls the acquisition time sequence of the at least 3 taps to change constantly or controls the time delay of the light source for emitting the pulse light beam so as to realize that the at least 3 taps acquire charge signals by rotation. The time delay between successive frame periods is regularly increasing, regularly decreasing or irregularly varying; the difference in time delay between successive said frame periods is an integer multiple of the pulse width. The processing circuit is further configured to determine data of the charge signal to determine whether the data of the charge signal includes the charge signal of the reflected pulse beam, and then calculate the flight time of the pulse beam and/or the distance between the object to be measured according to a determination result.

The invention also provides a distance measuring method for reducing noise of single-frequency modulation and demodulation, which comprises the following steps: s1: emitting a pulse light beam to an object to be measured by using a light source; s2: collecting charge signals of reflected pulse light beams reflected by the object to be measured by using an image sensor consisting of at least one pixel, wherein each pixel comprises at least 3 taps, and the taps are used for collecting the charge signals and/or the charge signals of background light; s3: and controlling the at least 3 taps to alternately acquire charge signals among at least 3 frame periods of a macrocycle, and receiving data of the charge signals to calculate the flight time of the pulse light beam and/or the distance of the object to be measured.

In one embodiment of the invention, the time of flight of the pulsed light beam is calculated according to the following equation:

wherein Q is₁₁、Q₂₁、Q₃₁、Q₁₂、Q₂₂、Q₃₂、Q₁₃、Q₂₃、Q₃₃Respectively representing the signals acquired by said 3 taps during consecutive 3 frame periods. The processing circuit controls the acquisition time sequence of the at least 3 taps to change constantly or controls the time delay of the light source for emitting the pulse light beam so as to realize that the at least 3 taps acquire charge signals by rotation. The time delay between successive frame periods is regularly increasing, regularly decreasing or irregularly varying; the difference in time delay between successive said frame periods is an integer multiple of the pulse width. The method further comprises the steps of judging the data of the charge signals to determine whether the data of the charge signals contain the charge signals of the reflected pulse beams or not, and calculating the flight time of the pulse beams and/or the distance of the object to be measured according to the judgment result.

The invention has the beneficial effects that: the time flight depth camera and the distance measurement method for reducing noise of single-frequency modulation and demodulation are provided, and the contradiction that pulse width is directly proportional to measurement distance and power consumption and is negatively related to measurement precision in the existing PM-iToF measurement scheme is eliminated; the extension of the measuring distance is not limited by the pulse width any more, so that the lower measuring power consumption and higher measuring precision can be maintained under the condition of longer measuring distance, and in addition, Fixed-Noise (FPN) caused by mismatch between taps or between reading circuits due to process manufacturing errors and the like is reduced or eliminated by a tap-rotating acquisition method. Compared with the CW-iToF measurement scheme, in the scheme, only one set of modulation and demodulation frequency needs to expose and output the signal quantity of three taps once to obtain one frame of depth information, so that the overall measurement power consumption is obviously reduced, and the measurement frame frequency is improved. Therefore, the scheme has obvious advantages compared with the existing iToF technical scheme.

Drawings

FIG. 1 is a schematic diagram of a temporal depth of flight camera according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of a method for transmitting and collecting optical signals of a time-of-flight depth camera according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of a method for emitting and collecting optical signals of a time-of-flight depth camera with reduced noise according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of another noise-reducing time-of-flight depth camera optical signal emission and collection method according to an embodiment of the present invention.

Fig. 5 is a schematic diagram of a noise-reduced distance measurement method of single-frequency modem according to an embodiment of the present invention.

FIG. 6 is a schematic diagram of optical signal emission and collection for another time-of-flight depth camera according to an embodiment of the present invention.

Fig. 7 is a method for forward-backward frame acquisition according to an embodiment of the present invention.

Fig. 8(a) is a schematic diagram of another sequential frame sequential acquisition method according to an embodiment of the present invention.

Fig. 8(b) is a schematic diagram of another sequential frame sequential acquisition method according to an embodiment of the present invention.

Fig. 9 is a schematic diagram of a noise-reduced distance measurement method of multi-frequency modulation and demodulation according to an embodiment of the present invention.

Detailed Description

In order to make the technical problems, technical solutions and advantageous effects to be solved by the embodiments of the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and the embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

It will be understood that when an element is referred to as being "secured to" or "disposed on" another element, it can be directly on the other element or be indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or be indirectly connected to the other element. The connection may be for fixation or for circuit connection.

It is to be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like are used in an orientation or positional relationship indicated in the drawings for convenience in describing the embodiments of the present invention and to simplify the description, and are not intended to indicate or imply that the referenced device or element must have a particular orientation, be constructed in a particular orientation, and be in any way limiting of the present invention.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of the present invention, "a plurality" means two or more unless specifically limited otherwise.

FIG. 1 is a schematic view of a temporal depth of flight camera according to one embodiment of the invention. Time flight depth camera 10 includes emission module 11, gather module 12 and processing circuit 13, wherein emission module 11 provides emission beam 30 to object 20 in the space with the illumination space in the target space, at least partial emission beam 30 forms reflected light beam 40 after object 20 reflects, reflected light beam 40 is at least partly gathered by gathering module 12, processing circuit 13 is connected with emission module 11 and gathering module 12 respectively, synchronous emission module 11 and gather module 12's trigger signal is sent out and is gathered module 12 by gathering by emission module 11 with the calculation beam and receive required time, namely emission beam 30 and reflected light beam 40's time of flight t, furthermore, total light flying distance D of corresponding point on the object can be calculated by the following formula:

D＝c·t (1)

where c is the speed of light.

The emitting module 11 includes a light source 111, a light beam modulator 112, a light source driver (not shown), and the like. The light source 111 may be a light source such as a Light Emitting Diode (LED), an Edge Emitting Laser (EEL), a Vertical Cavity Surface Emitting Laser (VCSEL), or a light source array composed of a plurality of light sources, and light beams emitted by the light sources may be visible light, infrared light, ultraviolet light, or the like. The light source 111 emits a light beam outwards under the control of a light source driver (which may be further controlled by the processing circuit 13), for example, in one embodiment, the light source 111 emits a pulsed light beam under the control at a certain frequency, which may be set according to a measurement distance, for example, 1MHz to 100MHz, which may be set in a range of several meters to several hundred meters, and may be used in Direct time of flight (TOF) measurement; in one embodiment, the light source 111 emits light beams whose amplitude is modulated under control to emit pulsed light beams, square wave light beams, sine wave light beams, and the like, which can be used in Indirect time of flight (inductively TOF) measurements. It will be appreciated that the light source 111 may be controlled to emit the associated light beam by means of a part of the processing circuitry 13 or a sub-circuit present independently of the processing circuitry 13, such as a pulse signal generator.

The beam modulator 112 receives the light beam from the light source 111 and emits a spatially modulated light beam, such as a floodlight beam with a uniform intensity distribution or a patterned light beam with a non-uniform intensity distribution, to the outside. It is to be understood that the distribution uniformity is a relative concept and is not absolutely uniform, and generally the light beam intensity at the edge of the field of view is allowed to be slightly lower, and in addition, the intensity for the imaging area in the middle can be changed within a certain threshold, for example, the intensity change can be allowed to be not more than 15% or 10%. In some embodiments, the beam modulator 112 is also configured to expand the received beam to expand the field of view.

The collecting module 12 includes an image sensor 121, a lens unit 122, and may further include a filter (not shown in the figure), the lens unit 122 receives and images at least part of the spatially modulated light beam reflected by the object on at least part of the image sensor 121, and the filter needs to select a narrow band filter matched with the wavelength of the light source to suppress background light noise in the rest of the wavelength bands. The image sensor 121 may be a Charge Coupled Device (CCD), Complementary Metal Oxide Semiconductor (CMOS), Avalanche Diode (AD), Single Photon Avalanche Diode (SPAD), etc., with an array size representing the resolution of the depth camera, such as 320x240, etc. Generally, a readout circuit (not shown in the figure) composed of one or more of a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (ADC), and the like is also included in connection with the image sensor 121.

Generally, the image sensor 121 comprises at least one pixel, each pixel then comprising a plurality of taps (for storing and reading or discharging charge signals generated by incident photons under control of the respective electrodes), for example 3 taps, for reading charge signal data.

In some embodiments, the time-of-flight depth camera 10 may further include a driving circuit, a power supply, a color camera, an infrared camera, an IMU, and so on, which are not shown in the drawings, and a combination of these devices may realize more abundant functions, such as 3D texture modeling, infrared face recognition, SLAM, and so on. Time-of-flight depth camera 10 may be embedded in a cell phone, tablet, computer, or other electronic product.

The processing circuit 13 may be a stand-alone dedicated circuit, such as a dedicated SOC chip, an FPGA chip, an ASIC chip, etc. including a CPU, a memory, a bus, etc., or may include a general-purpose processing circuit, such as when the depth camera is integrated into an intelligent terminal, such as a mobile phone, a television, a computer, etc., and the processing circuit in the terminal may be at least a part of the processing circuit 13. In some embodiments, the processing circuit 13 is configured to provide a modulation signal (emission signal) required when the light source 111 emits laser light, and the light source emits a pulse light beam to the object to be measured under the control of the modulation signal; the processing circuit 13 further provides a demodulation signal (acquisition signal) of a tap in each pixel of the image sensor 121, and the tap acquires a charge signal generated by a light beam including a reflected pulse light beam reflected back by the object under the control of the demodulation signal, and generally, there are some light beams such as background light, interference light, and the like in addition to the reflected pulse light beam reflected back by the object; the processing circuit 13 may also provide auxiliary monitoring signals such as temperature sensing, over-current, over-voltage protection, dropout protection, etc.; the processing circuit 13 may also be configured to store and perform corresponding processing on raw data acquired by each tap in the image sensor 121 to obtain specific position information of the object to be measured. The functions of the modulation and demodulation method, control, processing, etc. performed by the processing circuit 13 will be described in detail in the embodiments of fig. 2 to 8, and for convenience of explanation, the PM-ietf modulation and demodulation method is taken as an example for explanation.

Fig. 2 is a schematic diagram of a method for transmitting and collecting optical signals of a time-of-flight depth camera according to an embodiment of the present invention. Fig. 2 shows an exemplary timing diagram of a laser emission signal (modulation signal), a reception signal, and a collection signal (demodulation signal) in two frame periods T, where the meaning of each signal is: sp represents a pulsed emission signal of the light source, each pulsed emission signal representing a pulsed light beam; sr represents a reflected light signal of the pulse light reflected by the object, each reflected light signal represents a corresponding pulse light beam reflected by the object to be measured, a certain delay is arranged on a time line (horizontal axis in the figure) relative to a pulse emission signal, and the delayed time t is the flight time of the pulse light beam to be calculated; s1 represents a pulse pickup signal of a first tap of the pixel, S2 represents a pulse pickup signal of a second tap of the pixel, and S3 represents a pulse pickup signal of a third tap of the pixel, each pulse pickup signal representing that the tap picks up a charge signal (electrons) generated by the pixel in a period corresponding to the signal; tp is N × Th, where N is the number of taps participating in pixel electron collection, and N is 3 in the embodiment shown in fig. 2.

The entire frame period T is divided into two periods Ta and Tb, where Ta denotes a period during which charge collection and storage is performed by each tap of the pixel, and Tb denotes a period during which the charge signal is read out. In the charge collection and storage period Ta, the collected signal pulse of the nth tap has a phase delay time of (n-1) × Th with respect to the laser emission signal pulse, respectively, and each tap collects electrons generated on the pixel during its pulse period when the reflected light signal is reflected back to the pixel by the object. In the present embodiment, the first tap is triggered in synchronization with the laser emission signal, and when the reflected light signal is reflected back to the pixel by the object, the first tap, the second tap, and the third tap respectively and sequentially perform charge collection and storage to obtain the charge amounts q1, q2, and q3, respectively, so as to complete a pulse period Tp, which is 3Th for the case of 3 taps. In the embodiment shown in fig. 2, two pulse periods Tp are included in a single frame period, and 2 laser pulse signals are emitted in total, so that the total charge amount collected and read by each tap in the Tb period is the sum of the charge amounts corresponding to the two collected optical signals, it can be understood that, in a single frame period, the pulse period Tp or the number of times of emitting the laser pulse signals may be K times, K is not less than 1, and may be as high as several tens of thousands, or even higher, and the specific number is determined according to actual requirements, and in addition, the number of pulses in different frame periods may also be different.

Therefore, the total charge amount collected and read out by each tap in the Tb period is the sum of the charge amounts corresponding to the optical signals collected by each tap for a plurality of times in the whole frame period T, and the total charge amount of each tap in a single frame period can be represented as follows:

Qi＝∑qi，i＝1，2，3 (2)

the total charge amount in a single frame period of the first tap, the second tap and the third tap can be obtained as Q1, Q2 and Q3 according to the formula (2).

In the conventional modulation and demodulation method, the measurement range is limited to a single pulse width time Th, that is, assuming that the reflected light signal is collected by the first tap and the second tap (the first tap and the second tap will also collect the ambient light signal), the third tap is used for collecting the ambient light signal, so that based on the total charge quantity collected by each tap, the processing unit can calculate the total light flying distance of the pulsed light signal from the light emitted to the pixel reflected by the pixel according to the following formula:

further spatial coordinates of the target can be calculated based on the optical and structural parameters of the camera.

The conventional modem method has the advantages of simple calculation, but has the disadvantages that the measurement range is limited, the measured flight time is limited within Th, and the corresponding maximum flight distance measurement range is limited within c × Th.

In order to improve the measurement distance, the invention provides a novel modulation and demodulation method. Fig. 2 is a schematic diagram of optical signal emission and collection according to an embodiment of the present invention, in which the reflected optical signal can fall not only on the first tap and the second tap, but also on the second tap and the third tap, and even on the third tap and the first tap within the next pulse period Tp (for the case of at least two pulse periods Tp). The term "falling on the tap" as used herein means that the tap can be picked up. Since the total charge amount read in the period Tb is Q1, Q2, and Q3, unlike the conventional modulation and demodulation method, the present invention does not limit the tap or even the period of receiving the reflected light signal.

Considering that the amount of charge collected by the tap receiving the reflected light signal is larger than that of the tap containing only the background light signal, the processing circuit determines the three total amounts of charge Q1, Q2 and Q3 to determine to obtain the tap containing the excitation electrons of the reflected light signal and/or to obtain the tap containing only the background signal, and there may be electronic crosstalk between the taps in actual use, for example, some reflected light signals may enter the tap originally used for obtaining only the background signal, and these errors will be allowed and are within the protection scope of the present scheme. Assuming that after the determination, two total charge amounts including the reflected light signal are respectively denoted as QA and QB and a total charge amount including only the background light signal is denoted as QO in sequence (the reflected light signal is received in time sequence), there are three possibilities for the three-tap image sensor:

(1)QA＝Q1，QB＝Q2，QO＝Q3；

(2)QA＝Q2，QB＝Q3，QO＝Q1；

(3) QA-Q3, QB-Q1 (next pulse period Tp), and QO-Q2;

the processing circuitry may then calculate the time of flight of the optical signal according to:

m in the equation reflects the delay of the tap into which the reflected optical signal first falls relative to the first tap, and m is 0, 1,2 for the three cases described above, respectively. That is, if the reflected light signal first falls into the nth tap, m is equal to n-1. n is the serial number of the tap corresponding to the QA, and the phase delay time of the tap with the serial number n relative to the emitted light pulse signal is (n-1) multiplied by Th; th is the pulse width of the pulse acquisition signal of each tap; tp is the pulse period, Tp-N × Th, where N is the number of taps participating in pixel electron collection.

Comparing equation (4) with equation (3), it is clear that the measured distance is extended, and the maximum measured flying distance is extended from c · Th in the conventional method to c × Tp ═ c × N × Th in the present application, where N is the number of taps participating in pixel electron collection, which in the present example is 3, so that the above method achieves a measured distance 3 times that of the conventional method by the judgment mechanism, i.e., with respect to the conventional modem method.

The key to the above modem method is how to determine the tap into which the reflected optical signal falls. For this purpose, the present application provides the following methods of determination:

(1) single tap maxima method. The tap (denoted as Node) with the largest output signal (total charge amount) among the taps 1 to N (N is 3 in the above embodiment) is searched_x) Then according to Node₁→Node₂→…→Node_N→Node₁Sequential nodel recording of → …_xThe previous tap is Node_w(ii) a Node recording_xThe latter tap of (2) is Node_y. If Node_wAnd Node_yTotal amount of electric charge Q of_w≥Q_yThen Node_wNamely a tap A; if Q_w<Q_yThen Node_xNamely tap a.

(2) Adjacent taps and the maximum method. According to Node first₁→Node₂→…→Node_N→Node₁The sequence of → … calculates the Sum of the total charge of adjacent taps, Sum₁＝Q₁+Q₂,Sum₂＝Q₂+Q₃,…,Sum_N＝Q_N+Q₁Finding the maximum term Sum therein_nThen tap n is tap a and the tap after tap n is tap B.

After the taps a and B are confirmed, the background signal amount is calculated in at least four ways:

(2) b, background; that is, the signal amount of one tap after the B tap is taken as the background signal amount.

(3) A, a front background; that is, the signal amount of the tap before the a tap is taken as the background signal amount.

(4) Average background; i.e. taking the average of all tap signal quantities except the A, B taps as the background signal quantity.

(5) Subtracting an average background; i.e. the average of A, B taps and the signal quantity of all taps except one tap after the B tap is taken as the background signal quantity.

Note that, when N is 3, that is, there are only 3 taps, the method (4) is not preferable, and the methods (1) to (3) are equivalent; when k is 4, the methods (3) and (4) are equivalent, and the method (3) may be selected preferentially in order to reduce crosstalk of the signal amount as much as possible. Method (4) may be preferentially selected when k > 4.

While a 3-tap pixel based modem method has been described in the above embodiments, it will be appreciated that this modem method is equally applicable to more-tap pixels, i.e., N >3, such as a maximum 4Th measured distance for a 4-tap pixel and a maximum 5Th measured distance for a 5-tap pixel. This measurement method extends the farthest measured flight time from the pulse width time Th to the entire pulse period Tp, relative to conventional PM-ietf measurement schemes, referred to herein as single frequency full period measurement schemes.

In the analysis of the above embodiment, the charge amount collected by each tap and the calculation formula of the time of flight are both ideal, however, in actual situations, the mismatch of pixels due to process manufacturing errors or the mismatch between ADCs (analog-to-digital converters) of the taps may cause FPN (Fixed-pattern Noise), which is a problem that the gains of the taps are different or the offsets (offsets) of the circuits such as the ADCs are different, and finally cause measurement errors.

To solve this problem, the present invention provides a measurement method that can reduce noise. FIG. 3 is a schematic diagram of a method for emitting and collecting optical signals of a time-of-flight depth camera with reduced noise according to an embodiment of the present invention. Fig. 3 schematically shows the modem signal in three consecutive frame periods T1, T2, T3, which are taken as a unit of a macrocycle in the present scheme, i.e. the modem signal will continuously cycle with macrocycles of T1, T2, T3, T1, T2, T3, T1 … in time sequence. In three continuous frame periods Ti (i ═ 1,2 and 3) of a single macrocycle unit, the processing circuit controls the acquisition timing (acquisition phase) of each tap to be changed continuously so that the three taps can alternately acquire charge signals. For example, in the embodiment shown in fig. 3, in the T1 period, three taps sequentially acquire charge signals in time periods of 0 to 1/3Tp (0 to 120 °), 1/3Tp to 2/3Tp (120 to 240 °), and 2/3Tp to Tp (240 to 360 °) in the order of S1 to S2 to S3 in each pulse period Tp; in a T2 period, three taps sequentially acquire charge signals in time periods of 0-1/3 Tp (0-120 degrees), 1/3 Tp-2/3 Tp (120-240 degrees) and 2/3 Tp-Tp (240-360 degrees) in the sequence of S3-S1-S2 in each pulse period Tp; in a T3 period, charge signals in time periods of 0-1/3 Tp (0-120 DEG), 1/3 Tp-2/3 Tp (120-240 DEG) and 2/3 Tp-Tp (240-360 DEG) are sequentially acquired by three taps in the order of S2-S3-S1 in each pulse period Tp.

It can be understood that, in each frame period, the variation manner of the tap acquisition time sequence is not limited to the sequential order rotation method in the above example, and any variation manner may be used as long as the acquisition time sequence of each tap can realize the rotation acquisition.

Generally, for a pixel with N taps, a single macrocycle unit will contain at least N frame periods, thus ensuring that a full rotation acquisition can be achieved for each tap. For example, as for the 3-tap pixel in the embodiment shown in fig. 3, a single macro-period unit contains 3 frame periods, and it is understood that a single macro-period unit may also contain more frame periods, for example, in an embodiment, 3n frame periods, that is, integer multiples of tap data, and of course, any other frame periods may also be contained according to actual requirements. In addition, the N frame periods in the unit of macrocycle are not necessarily consecutive in time sequence, for example, in one embodiment, the frame periods included in two macrocycles or a plurality of macrocycles may intersect with each other.

Suppose that the charge signals respectively collected along three taps in time sequence under ideal conditions are respectively Q_O、Q₁₂₀、Q₂₄₀In fact, due to the existence of FPN, the signal collected by each tap in three consecutive frame periods is Q₁₁、Q₂₁、Q₃₁、Q₁₂、Q₂₂、Q₃₂、Q₁₃、Q₂₃、Q₃₃Wherein Q is_ij＝∑q_ijI denotes a tap and i is 1,2,3, j denotes a period, and j is 1,2, 3. Further, Q is GQ + O, where G, O represents the gain and offset (offset) of the corresponding tap, for example, for the T1 period in fig. 3, there are:

Q₁₁＝G₁Q_O+O₁,Q₂₁＝G₂Q₁₂₀+O₂,Q₃₁＝G₃Q₂₄₀+O₃ (5)

for period T2 in fig. 3, there are:

Q₁₂＝G₁Q₁₂₀+O₁,Q₂₂＝G₂Q₂₄₀+O₂,Q₃₂＝G₃Q_O+O₃ (6)

for period T3 in fig. 3, there are:

Q₁₃＝G₁Q₂₄₀+O₁,Q₂₃＝G₂Q_O+O₂,Q₃₃＝G₃Q₁₂₀+O₃ (7)

in order to reduce the FPN, the present scheme calculates a single-frame time-of-flight value (or depth value) by using charge signals collected from three consecutive frames, and for the convenience of analysis, it is assumed that the reflected light signals fall into taps of corresponding time periods of 0 to 1/3Tp (0 to O °), 1/3Tp to 2/3Tp (O ° to 120 °), and the calculation formula is as follows:

if the single-frequency full-period measurement scheme shown in fig. 2 is considered, the calculation formula is as follows:

taking the case corresponding to the formula (8) as an example for analysis, substituting the formulas (5) to (7) into the formula (8):

as can be seen from equation (10), the flight time calculated by concatenating 3 frames of data is not affected by the gain G and the offset O, and the error caused by FPN is theoretically eliminated.

FIG. 4 is a schematic diagram of a method for emitting and collecting optical signals of a time-of-flight depth camera with reduced noise according to another embodiment of the present invention. In order to reduce noise, in the embodiment shown in fig. 3, a manner of changing the acquisition timing of the taps in each frame period in a macro-period unit to realize alternate acquisition is adopted, however, since changing the acquisition timing of the taps continuously is relatively difficult to realize in practical application, in order to overcome this problem, a manner of controlling the pulse transmission time is adopted in the present embodiment. Also taking 3 taps as an example for illustration, a single macro-period includes 3 frame periods T1, T2, T3, in each frame period, the processing circuit controls the pulsed light beam to emit with a time delay of a certain time sequence to realize the alternate collection of the charge signal by each tap, in this embodiment, in the frame periods T1, T2, T3, the pulsed light beam is respectively emitted with Δ T₁、Δt₂、Δt₃Is transmitted with a time delay of Δ t_i＝(i-1)T_h(i ═ 1,2, 3). Since the lowest time delay Δ t1 is 0, it is not shown in the figure. It will be appreciated that, in other embodiments of the invention,the lowest delay may not be 0.

In fig. 4, during the T3 frame period, the reflected pulse signal enters the second pulse period Tp, resulting in the charge signal being collected by only a single tap in the first pulse period, but the error is negligible due to the fact that there are thousands to tens of thousands of pulse periods.

It will be appreciated that the time delay of the pulsed light beam in successive frame periods of a single macrocycle may not be in the form of a regular increment as in the embodiment of fig. 4, for example, it may be in the form of a regular decrement or an irregular increment, the lowest time delay may not be 0, and the difference between the time delays may not be a single pulse width, and may be an integer multiple of the pulse width, for example, 2 pulse widths.

As can be seen from fig. 4, by applying a time delay to the pulsed light beam, under the premise of not changing the acquisition timing of each tap, the rotation acquisition of the charge signal by each tap in each frame period of a single macrocycle is also realized, the calculation formula of the flight time is also formulas (5) - (10), and the FPN noise is also reduced.

While the embodiment shown in fig. 3 and 4 describes a 3-tap pixel-based modulation and demodulation method for reducing noise, it can be understood that this modulation and demodulation method is also applicable to pixels with more taps, i.e. N >3, for example, for a 4-tap pixel, a single macrocycle unit contains 4 consecutive frame periods, and in each period, the processing circuit controls the acquisition timing of each tap to be changed continuously or controls the pulsed light beam to be emitted with a time delay of a certain timing so that each tap can rotate to acquire a charge signal, thereby reducing noise.

The proposed single-frequency full-period measurement scheme in the embodiment shown in fig. 2 is also applicable to the noise reduction measurement schemes shown in fig. 3 and 4, that is, the charge signals measured by the respective taps are determined to determine whether the charge signal data of the reflected pulsed light beam includes the charge signal of the reflected pulsed light beam, so as to confirm the value of each charge amount Q in formula (9), and then the flight time is calculated based on formula (9).

As shown in fig. 5, a schematic diagram of a distance measurement method for reducing noise in single-frequency modulation and demodulation specifically includes the following steps:

s1: emitting a pulse light beam to an object to be measured by using a light source;

s2: collecting charge signals of reflected pulse light beams reflected by the object to be measured by using an image sensor consisting of at least one pixel, wherein each pixel comprises at least 3 taps, and the taps are used for collecting the charge signals and/or the charge signals of background light;

s3: and controlling the at least 3 taps to alternately acquire charge signals among at least 3 frame periods of a macrocycle, and receiving data of the charge signals to calculate the flight time of the pulse light beam and/or the distance of the object to be measured.

The single frequency full period measurement scheme increases the measurement distance to some extent, but still cannot satisfy the measurement of longer distance. For example, in a modulation and demodulation method based on 3-tap pixels, when the flight time corresponding to the object distance exceeds 3Th, the reflected light signal in a certain pulse period Tp will fall onto the tap in the subsequent pulse period first, and at this time, the flight time or distance cannot be accurately measured by using either formula (3) or formula (4). For example, when the reflected light signal in a certain pulse period Tp first falls into the nth tap in the subsequent jth pulse period, the flight time of the light signal corresponding to the real object is as follows:

where m is n-1, and n is the serial number of the tap corresponding to QA. Since the total charge of each tap is an integral of the charge accumulated during the pulse period of interest, the specific value of j cannot be discerned from the total charge of each tap output alone, which causes confusion in distance measurement.

FIG. 6 is a schematic diagram of optical signal emission and collection for a time-of-flight depth camera according to another embodiment of the present invention, which can be used to solve the above confusion problem. Unlike the embodiment shown in fig. 2, this embodiment employs a multi-frequency modem method, i.e., adjacent frames are controlled by the processing circuit to employ different modem frequencies. For convenience of explanation in this embodiment, two adjacent frame periods are taken as an example, in the adjacent frame periods, the number of pulse emission times K is 2 (may be multiple times, and different frame times may be different), the number of taps N of the pixel is 3, the pulse period TPi is Tp1 and Tp2, the pulse width Thi is Th1 and Th2, the pulse frequency or modulation and demodulation frequency is f1 and f2, the accumulated charges of the three taps per pulse are Q11, Q12, Q21, Q22, Q31 and Q32, respectively, and the total charge amount is Q11, Q12, Q21, Q22, Q31 and Q32 according to the formula (2).

It is assumed that the distance of the object in the period of the adjacent frame (or continuous frames) is not changed, so t in the period of the adjacent frame is the same. After receiving the total charge amount of each tap, the processing circuit measures the distance d (or t) in each frame period by using the modulation and demodulation method shown in fig. 2, and calculates QAi, QBi, and QOi in each frame period by using the above-described determination method, where i represents the i-th frame period, and in this embodiment, i is 1 or 2. In order to expand the measurement range and allow the reflected light signal to fall on the tap in the subsequent pulse period, assuming that the reflected light signal on a certain pixel in the ith frame period firstly falls on the mi-th tap in the ji-th pulse period after the pulse period of the transmitted light pulse (the pulse period of the transmitted light pulse is the 0 th pulse period after the transmitted light pulse beam is emitted), the corresponding flight time can be expressed as follows according to the formula (11):

considering that the object distance in the adjacent frame period is not changed, the following equation holds for the case of two consecutive frames in this embodiment:

(x1+m1)Th1+j1·Tp1＝(x2+m2)Th2+j2·Tp2 (13)

wherein the content of the first and second substances,

i is 1, 2. For theThe following holds for the case of consecutive multiframes (assuming consecutive w frames, i.e., 1,2, …, w):

(x1+m1)Th1+j1·Tp1＝(x3+m2)Th2+j2·Tp2

＝…＝xw+mwThw+jw·Tpw (14)

it is to be understood that when w is 1, this corresponds to the single frequency full period measurement scheme set forth above. When w is greater than 1, the processing circuit can find out a group of ji combinations with the minimum ti variance under each modulation and demodulation frequency as a solution value according to the remainder theorem or by traversing various ji combinations in the maximum measurement distance, and then complete the solution of ji; and then carrying out weighted average on the flight time or the measured distance solved under each group of frequencies to obtain the final flight time or the measured distance. With the multi-frequency modulation-demodulation method, the maximum measurement flight time is extended to:

t_max＝LCM(Tp₁，Tp₂，…，Tp_w) (15)

the maximum measured flight distance is extended to:

D_max＝LCM(D_max1,D_max2,…,D_maxw) (16)

wherein Dmax_iLCM (lowest Common multiple) denotes taking the 'least Common multiple' (where 'least Common multiple' is a generalized extension of the least Common multiple of the integer domain, LCM (a, b) being defined as the smallest real number divisible by the real numbers a, b).

Suppose that in the embodiment shown in FIG. 6, T_p15ns, the maximum measured flight distance is 4.5 m; if T_pThe maximum measured flight distance is 6m, 20 ns. If a multi-frequency modulation-demodulation method is used, for example, in one embodiment, T_p1＝15ns，T_p2The least common multiple of 15ns and 20ns is 60ns, 60ns corresponds to a maximum measuring distance of 18m, and the corresponding farthest measuring target distance can reach 9 m.

It is understood that, although in the embodiment shown in fig. 6, the distance of the object is calculated by at least two frames of data, in one embodiment, the number of the acquired frames may not be reduced by a forward and backward frame sequential manner, in the forward and backward frame sequential acquisition method according to an embodiment of the present invention shown in fig. 7, that is, in the case of obtaining a single time-of-flight measurement by forward and backward frames in the dual-frequency modem method, the first time-of-flight is calculated by 1 and 2 frames, the second time-of-flight is calculated by 2 and 3 frames, and so on, the frame rate of the time-of-flight is only 1 frame less than the frame period, so that the frame rate of the measurement is not reduced.

The multi-frequency modulation-demodulation scheme is also applicable to the noise-reduced time-of-flight measurement schemes shown in fig. 3 and 4. Fig. 8(a) and 8(b) are schematic diagrams illustrating a noise-reduced pmdm time-of-flight measurement method according to an embodiment of the present invention. Taking 3 taps as an example for explanation, a single macro-period comprises 3 frame periods, and the processing circuit controls the acquisition timing of each tap to be changed constantly or controls the pulse light beam to be emitted with a certain time delay in each frame period so that each tap can rotate to acquire charge signals, thereby reducing noise.

In order to increase the measurement distance, different modulation and demodulation frequencies are adopted in two adjacent macrocycles, such as f1 and f2 shown in fig. 8(a), and the flight time of the pulse beam and/or the distance of the object to be measured are calculated by combining the charge signal data collected in the two macrocycles, and the principle of the flight time calculation method is similar to the formulas (12) and (13), which is not repeated herein.

In some embodiments, in order for a time-of-flight depth camera to have more application scope, it is often necessary to satisfy multiple modem functions. For example, the high frame rate measurement may be implemented by using the modulation and demodulation method shown in fig. 2, or the high accuracy measurement may be implemented by using the modulation and demodulation method shown in fig. 3 or 4, which correspond to the high frame rate measurement mode and the high accuracy measurement mode, respectively. A further measuring range, i.e. a large-range measuring mode, can also be realized by multi-frequency modulation on the basis of the two modes. It can be understood that the frequency modulation needs to be implemented by a specific modulation driving circuit, and the multi-frequency modulation scheme shown in fig. 7 corresponds to a different modulation driving circuit from the multi-frequency modulation scheme shown in fig. 8(a), which means that at least two independent modulation driving circuits need to be set for control when the depth camera is required to satisfy the modulation scheme, which undoubtedly increases the design difficulty and cost. For this reason, as shown in fig. 8(b), high-precision measurement can be achieved also with the frequency modulation scheme shown in fig. 7. The macrocycle may be regarded as the composition of the (n), (n +2) and (n +4) th frames, for example, from the 1 st frame, the 1 st, 3th and 5th frames constitute one macrocycle, and the 2 nd, 4th and 5th frames constitute another adjacent macrocycle, and by combining the charge signal data collected in the macrocycles of the two different modulation and demodulation frequencies, the flight time of the pulsed light beam and/or the distance to the object to be measured can be calculated.

Similarly, in order not to reduce the frame rate, a front-to-back frame sequential manner may be adopted, for example, as shown in fig. 8, the first flight time is calculated and obtained from the 1 st to 6 th frames of the acquired signal data, the second flight time is calculated and obtained from the 2 nd to 7 th frames of the acquired signal data, and so on, the frame rate of the flight time is only 5 frames less than the frame period, and the measurement frame rate is not reduced.

It is understood that in the above-mentioned multi-frequency modulation and demodulation method, different measurement scenario requirements can be satisfied by adopting different frequency combinations, for example, the accuracy of the final distance resolution can be improved by increasing the number of measurement frequencies. In order to dynamically meet the measurement requirements under different measurement scenarios, in an embodiment of the present invention, the processing circuit adaptively adjusts the number of frequencies modulated and demodulated and the specific frequency combination through result feedback to meet the requirements under different measurement scenarios as much as possible. Specifically, in one embodiment, after calculating the current distance (or flight time) of the object, the processing circuit performs statistics on the target distance, and when most of the measurement targets are closer, the processing circuit may use a smaller number of frequencies to perform measurement to ensure a higher frame rate, and reduce the influence of target motion on the measurement result, and when there are more distant targets in the measurement targets, the processing circuit may appropriately increase the number of frequencies to be measured or adjust the combination of the measurement frequencies to ensure the measurement accuracy.

As shown in fig. 9, a schematic diagram of the distance measuring method for reducing noise in multi-frequency modulation and demodulation specifically includes the following steps:

t1: using a light source to emit a pulse light beam to an object to be measured;

t2: collecting charge signals of reflected pulse light beams reflected by the object to be measured by using an image sensor consisting of at least one pixel, wherein each pixel comprises at least 3 taps, and the taps are used for collecting the charge signals and/or the charge signals of background light;

t3: and controlling the at least 3 taps to alternately acquire charge signals between at least 3 frame periods of a macro period, adopting different modulation and demodulation frequencies in two adjacent macro periods, and receiving data of the charge signals received in the two adjacent macro periods to calculate the flight time of the pulse light beam and/or the distance of the object to be measured.

In addition, with respect to the method and the contents described in the embodiments of the present invention, it should be noted that any single-frequency full-period measurement scheme, noise reduction measurement scheme, and multi-frequency long-distance measurement scheme based on three or more taps sensors, whether the waveform of the modem signal is continuous or discontinuous in the exposure time range, or the measurement sequence of different frequency modem signals and the fine adjustment of the modulation frequency in the same exposure time, etc. are all within the protection scope of the present patent, and the exemplary description or analysis algorithm performed to explain the principles of the present patent is only one exemplary description of the present patent and should not be considered as a limitation to the present patent content. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.

The invention has the advantages that the contradiction that the pulse width is directly proportional to the measurement distance and the power consumption and is negatively related to the measurement precision in the existing PM-iToF measurement scheme is eliminated; the extension of the measuring distance is not limited by the pulse width any more, so that the lower measuring power consumption and higher measuring precision can be maintained under the condition of longer measuring distance, and in addition, Fixed-Noise (FPN) caused by mismatch between taps or between reading circuits due to process manufacturing errors and the like is reduced or eliminated by a tap-rotating acquisition method. Compared with the CW-iToF measurement scheme, in the scheme, only one set of modulation and demodulation frequency needs to expose and output the signal quantity of three taps once to obtain one frame of depth information, so that the overall measurement power consumption is obviously reduced, and the measurement frame frequency is improved. Therefore, the scheme has obvious advantages compared with the existing iToF technical scheme.

All or part of the flow of the method of the embodiments may be implemented by a computer program, which may be stored in a computer readable storage medium and executed by a processor, to instruct related hardware to implement the steps of the embodiments of the methods. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.

The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.

Claims

1. A time-of-flight depth camera, comprising:

the transmitting module comprises a light source and is used for transmitting a pulse light beam to the object to be detected;

the acquisition module comprises an image sensor consisting of at least one pixel, each pixel comprises at least 3 taps, and the taps are used for acquiring charge signals generated by reflected pulse beams reflected by the object to be measured and/or charge signals of background light in a rotating manner by continuously changing the acquisition time sequence among at least 3 frame periods of a macro period; the tap and the period for receiving the transmitted pulse beam are not limited;

and the processing circuit is used for controlling the acquisition time sequence of the at least 3 taps to continuously change between at least 3 frame periods of a macro period so as to alternately acquire the charge signals, and receiving data of the charge signals to calculate the flight time of the pulse light beam and/or the distance of the object to be measured.

2. The time depth of flight camera of claim 1,

the processing circuitry calculates the time of flight of the pulsed light beam according to:

wherein Q is₁₁、Q₂₁、Q₃₁、Q₁₂、Q₂₂、Q₃₂、Q₁₃、Q₂₃、Q₃₃Respectively representing the signals acquired by 3 taps within 3 consecutive frame periods of the macrocycle.

3. The time-of-flight depth camera of claim 1, wherein the processing circuitry alternates the acquisition of the charge signal for the at least 3 taps by controlling the acquisition timing for the at least 3 taps to be constantly varied or controlling the time delay for the light source to emit the pulsed light beam.

4. The time depth of flight camera of claim 3, in which the time delay between successive frame periods is regularly increasing, regularly decreasing or irregularly varying; the difference in time delay between successive said frame periods is an integer multiple of the pulse width.

5. The time-of-flight depth camera of claim 1, wherein the processing circuit is further configured to determine the data of the charge signal to determine whether the data of the charge signal includes the charge signal of the reflected pulsed light beam, and calculate the flight time of the pulsed light beam and/or the distance to the object according to the determination result.

6. A noise-reducing distance measuring method for single-frequency modulation and demodulation is characterized by comprising the following steps:

s2: collecting charge signals of reflected pulse light beams reflected by the object to be measured by using an image sensor consisting of at least one pixel, wherein each pixel comprises at least 3 taps, and the taps are used for collecting the charge signals of which the time sequence is continuously changed between at least 3 frame periods of a macro period so as to alternately collect the charge signals and/or the background light; the tap and the period for receiving the transmitted pulse beam are not limited;

7. A single frequency modem noise reduction distance measuring method according to claim 6 wherein said pulsed beam time of flight is calculated according to the formula:

wherein Q is₁₁、Q₂₁、Q₃₁、Q₁₂、Q₂₂、Q₃₂、Q₁₃、Q₂₃、Q₃₃Respectively representing the signals acquired by said 3 taps during consecutive 3 frame periods.

8. The single frequency modem noise reduction distance measuring method of claim 6, wherein the processing circuit alternately collects the charge signals by controlling the collection timing of the at least 3 taps to be continuously varied or controlling the time delay of the light source emitting the pulse beam.

9. A single frequency modem noise reduction distance measuring method according to claim 8, wherein said time delay between successive said frame periods is regularly increasing, regularly decreasing or irregularly varying; the difference in time delay between successive said frame periods is an integer multiple of the pulse width.

10. The single frequency modem noise reduction distance measuring method according to claim 6, further comprising determining the data of the charge signal to determine whether the data of the charge signal contains the charge signal of the reflected pulse beam, and calculating the time of flight of the pulse beam and/or the distance to the object according to the determination result.