CN110221273B - Time flight depth camera and distance measuring method of single-frequency modulation and demodulation - Google Patents
Time flight depth camera and distance measuring method of single-frequency modulation and demodulation Download PDFInfo
- Publication number
- CN110221273B CN110221273B CN201910385779.6A CN201910385779A CN110221273B CN 110221273 B CN110221273 B CN 110221273B CN 201910385779 A CN201910385779 A CN 201910385779A CN 110221273 B CN110221273 B CN 110221273B
- Authority
- CN
- China
- Prior art keywords
- tap
- charge
- taps
- signal
- collected
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
- G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
- G01S17/26—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves wherein the transmitted pulses use a frequency-modulated or phase-modulated carrier wave, e.g. for pulse compression of received signals
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/483—Details of pulse systems
- G01S7/486—Receivers
- G01S7/4865—Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak
- G01S7/4866—Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak by fitting a model or function to the received signal
Abstract
The invention provides a time flight depth camera and a distance measuring method of single-frequency modulation and demodulation, wherein the time flight depth camera comprises an emitting module, a light source and a distance measuring module, wherein the emitting module comprises a light source used for emitting a pulse light beam to an object to be measured; the acquisition module comprises an image sensor consisting of at least one pixel, each pixel comprises at least 3 taps, and the taps are used for acquiring charge signals generated by reflected pulse beams reflected by an object to be measured or charge signals of background light; a processing circuit for receiving data of the charge signals of at least 3 taps; judging the data of the charge signal to determine whether the data of the charge signal contains the charge signal of the reflected pulse light beam; and calculating the flight time of the pulse light beam and/or the distance of the object to be measured according to the judgment result. One frame of depth information can be obtained only by outputting the signal quantity of three taps in one exposure, so that the overall measurement power consumption is obviously reduced, and the measurement frame frequency is improved.
Description
Technical Field
The invention relates to the field of optical measurement, in particular to a time flight depth camera and a distance measurement method of single-frequency modulation and demodulation.
Background
ToF is known collectively as Time-of-Flight, and ToF ranging is a technique for achieving accurate ranging by measuring the round-trip Time of Flight of an optical pulse between a transmitting/receiving device and a target object. The technique of directly measuring the optical time of flight in the ToF technique is called dtofs (direct-ToF); the measurement technique of periodically modulating the emitted light signal, measuring the phase delay of the reflected light signal relative to the emitted light signal, and calculating the time of flight from the phase delay is called the iToF (index-TOF) technique. According to the difference of the modulation and demodulation type, the modulation and demodulation method can be divided into a Continuous Wave (CW) modulation and demodulation method and a Pulse Modulated (PM) modulation and demodulation method.
At present, the CW-iToF technology is mainly applied to a measurement system constructed based on a two-tap sensor, a core measurement algorithm is a four-phase modulation and demodulation mode, and at least two exposures (usually four exposures are required to ensure measurement accuracy) are required to complete the acquisition of four-phase data and output a frame depth image, so that a higher frame frequency is difficult to obtain. The PM-iToF modulation technology is mainly applied to a four-tap sensor (three taps are used for signal acquisition and output, and one tap is used for releasing invalid electrons), the measurement distance of the measurement means is limited by the pulse width of a modem signal at present, when remote measurement is needed, the pulse width of the modem signal needs to be extended, and the extension of the pulse width of the modem signal causes the increase of power consumption and the reduction of measurement accuracy, so that the market demand cannot be met. Aiming at the defects of the two current modulation and demodulation modes, a new modulation and demodulation mode is provided for optimizing the iToF technical scheme.
Disclosure of Invention
The invention provides a time flight depth camera and a distance measuring method of single-frequency modulation and demodulation, aiming at solving the existing problems.
In order to solve the above problems, the technical solution adopted by the present invention is as follows:
a time-of-flight depth camera, comprising: the transmitting module comprises a light source and is used for transmitting a pulse light beam to the object to be detected; the acquisition module comprises an image sensor consisting of at least one pixel, each pixel comprises at least 3 taps, and the taps are used for acquiring charge signals generated by reflected pulse beams reflected by the object to be measured or charge signals of background light; a processing circuit for receiving data of the charge signals of the at least 3 taps; judging the data of the charge signal to determine whether the charge signal of the reflected pulse light beam is contained in the data of the charge signal; and calculating the flight time of the pulse light beam and/or the distance of the object to be detected according to the judgment result.
In one embodiment of the invention, the processing circuitry calculates the time of flight of the pulsed light beam according to:
wherein QA is an amount of charge of the charge signal containing the reflected pulsed light beam collected by a first one of the taps acquired after the judgment; QB is the amount of charge of the charge signal containing the reflected pulse beam collected by the second one of the taps acquired after the judgment; QO is the charge amount of the charge signal collected by the tap and only containing the background light; m is n-1, wherein n is the serial number of the tap corresponding to the QA; th is the pulse width of the pulse acquisition signal of each tap. The judgment comprises a single tap maximum value method, namely a first tap with the largest charge amount of charge signals in the at least 3 taps is searched and obtained in sequence, and if the charge amount of a charge signal of a second tap before the first tap is larger than that of a charge signal of a third tap after the first tap, the charge amounts of the charge signals collected by the second tap and the first tap are QA and QB respectively; and if the charge amount of the charge signal of the second tap before the first tap is smaller than that of the charge signal of the third tap after the first tap, the charge amounts of the charge signals collected by the first tap and the third tap are respectively QA and QB according to the sequence of tap serial numbers. The judgment comprises adjacent taps and a maximum value method, namely, after the electric charge quantity of the charge signals of the adjacent taps is calculated in sequence, the maximum value item is searched, and the electric charge quantities of the charge signals collected by the two taps corresponding to the maximum value item are respectively QA and QB. The QO is obtained by at least one of the following modes: taking the charge amount of the collected charge signal of a tap behind the tap corresponding to the QB; or, the charge quantity of the charge signal collected by the previous tap corresponding to the tap by QA is taken; or, taking the average value of the charge quantity of the charge signals collected by all the taps except the tap corresponding to the QA and the QB; or, taking the average value of the charge quantity of the charge signals collected by the QA, the tap corresponding to the QB and all the taps except the tap behind the tap corresponding to the QB.
The invention also provides a distance measuring method for single-frequency modulation and demodulation, which comprises the following steps: using a light source to emit a pulse light beam to an object to be measured; collecting charge signals of reflected pulse light beams reflected by the object to be measured by using an image sensor consisting of at least one pixel, wherein each pixel comprises at least 3 taps, and the taps are used for collecting the charge signals or charge signals of background light; receiving data of the charge signals of the at least 3 taps; judging the data of the charge signal to determine whether the charge signal of the reflected pulse light beam is contained in the data of the charge signal; and calculating the flight time of the pulse light beam and/or the distance of the object to be detected according to the judgment result.
In one embodiment of the invention, the time of flight is calculated according to the following equation:
wherein QA is an amount of charge of the charge signal containing the reflected pulsed light beam collected by a first one of the taps acquired after the judgment; QB is the amount of charge of the charge signal containing the reflected pulse beam collected by the second one of the taps acquired after the judgment; QO is the charge amount of the charge signal collected by the tap and only containing the background light; m is n-1, wherein n is the serial number of the tap corresponding to the QA; th is the pulse width of the pulse acquisition signal of each tap. The judgment comprises a single tap maximum value method, namely a first tap with the largest charge amount of charge signals in the at least 3 taps is searched and obtained in sequence, and if the charge amount of a charge signal of a second tap before the first tap is larger than that of a charge signal of a third tap after the first tap, the charge amounts of the charge signals collected by the second tap and the first tap are QA and QB respectively; and if the charge quantity of the charge signal of a second tap before the first tap is smaller than that of a third tap after the first tap, the charge quantities of the charge signals collected by the first tap and the third tap are respectively QA and QB. The judgment comprises adjacent taps and a maximum value method, namely, the electric charge quantity of the charge signals of the adjacent taps is calculated in sequence, then the maximum value item is searched, and the electric charge quantities of the charge signals collected by the two taps corresponding to the maximum value item are respectively QA and QB according to the sequence of tap serial numbers. The QO is obtained by at least one of the following modes: taking the charge amount of the collected charge signal of a tap behind the tap corresponding to the QB; or, the charge quantity of the charge signal collected by the previous tap corresponding to the tap by QA is taken; or, taking the average value of the charge quantity of the charge signals collected by all the taps except the tap corresponding to the QA and the QB; or, taking the average value of the charge quantity of the charge signals collected by the QA, the tap corresponding to the QB and all the taps except the tap behind the tap corresponding to the QB.
The invention has the beneficial effects that: the time flight depth camera and the single-frequency modulation and demodulation distance measuring method are provided, and compared with the existing PM measuring scheme, the measuring distance is expanded under the condition of the same pulse width; compared with the CW-iToF measurement scheme, the depth information of one frame can be obtained only by exposing and outputting the semaphore of three taps once, thereby obviously reducing the overall measurement power consumption and improving the measurement frame frequency.
Drawings
FIG. 1 is a schematic diagram of a temporal depth of flight camera according to an embodiment of the present invention.
FIG. 2 is a schematic diagram of a method for transmitting and collecting optical signals of a time-of-flight depth camera according to an embodiment of the present invention.
FIG. 3 is a schematic diagram of optical signal emission and collection for a time-of-flight depth camera according to another embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects to be solved by the embodiments of the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and the embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
It will be understood that when an element is referred to as being "secured to" or "disposed on" another element, it can be directly on the other element or be indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or be indirectly connected to the other element. The connection may be for fixation or for circuit connection.
It is to be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like are used in an orientation or positional relationship indicated in the drawings for convenience in describing the embodiments of the present invention and to simplify the description, and are not intended to indicate or imply that the referenced device or element must have a particular orientation, be constructed in a particular orientation, and be in any way limiting of the present invention.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of the present invention, "a plurality" means two or more unless specifically limited otherwise.
FIG. 1 is a schematic view of a temporal depth of flight camera according to one embodiment of the invention. Time flight depth camera 10 includes emission module 11, gather module 12 and processing circuit 13, wherein emission module 11 provides emission beam 30 to object 20 in the space with the illumination space in the target space, at least partial emission beam 30 forms reflected light beam 40 after object 20 reflects, reflected light beam 40 is at least partly gathered by gathering module 12, processing circuit 13 is connected with emission module 11 and gathering module 12 respectively, synchronous emission module 11 and gather module 12's trigger signal is sent out and is gathered the required time of module 12 receipt by gathering by emission module 11 with the calculation beam, namely, the flight time t between emission beam 30 and the reflected light beam 40, furthermore, total light flying distance D of corresponding point on the object can be calculated by the following formula:
D＝c·t (1)
where c is the speed of light.
The emitting module 11 includes a light source 111, a light beam modulator 112, a light source driver (not shown), and the like. The light source 111 may be a light source such as a Light Emitting Diode (LED), an Edge Emitting Laser (EEL), a Vertical Cavity Surface Emitting Laser (VCSEL), or a light source array composed of a plurality of light sources, and light beams emitted by the light sources may be visible light, infrared light, ultraviolet light, or the like. The light source 111 emits a light beam outwards under the control of a light source driver (which may be further controlled by the processing circuit 13), for example, in one embodiment, the light source 111 emits a pulsed light beam under the control at a certain frequency, which may be set according to a measurement distance, for example, 1MHz to 100MHz, which may be set in a range of several meters to several hundred meters, and may be used in Direct time of flight (TOF) measurement; in one embodiment, the light source 111 emits light beams whose amplitude is modulated under control to emit pulsed light beams, square wave light beams, sine wave light beams, and the like, which can be used in Indirect time of flight (inductively TOF) measurements. It will be appreciated that the light source 111 may be controlled to emit the associated light beam by means of a part of the processing circuitry 13 or a sub-circuit present independently of the processing circuitry 13, such as a pulse signal generator.
The beam modulator 112 receives the light beam from the light source 111 and emits a spatially modulated light beam, such as a floodlight beam with a uniform intensity distribution or a patterned light beam with a non-uniform intensity distribution, to the outside. It is to be understood that the distribution uniformity is a relative concept and is not absolutely uniform, and generally the light beam intensity at the edge of the field of view is allowed to be slightly lower, and the intensity for the imaging area in the middle can be changed within a certain threshold, for example, the intensity change can be allowed to be not more than 15% or 10%. In some embodiments, the beam modulator 112 is also configured to expand the received beam to expand the field of view.
The collecting module 12 includes an image sensor 121, a lens unit 122, and may further include a filter (not shown in the figure), the lens unit 122 receives and images at least part of the spatially modulated light beam reflected by the object on at least part of the image sensor 121, and the filter needs to select a narrow band filter matched with the wavelength of the light source to suppress background light noise in the rest of the wavelength bands. The image sensor 121 may be a Charge Coupled Device (CCD), Complementary Metal Oxide Semiconductor (CMOS), Avalanche Diode (AD), Single Photon Avalanche Diode (SPAD), etc., with an array size representing the resolution of the depth camera, such as 320x240, etc. Generally, a readout circuit (not shown in the figure) composed of one or more of a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (ADC), and the like is also included in connection with the image sensor 121.
Generally, the image sensor 121 comprises at least one pixel, each pixel then comprising a plurality of taps (for storing and reading or discharging charge signals generated by incident photons under control of the respective electrodes), for example 3 taps, for reading charge signal data.
In some embodiments, the time-of-flight depth camera 10 may further include a driving circuit, a power supply, a color camera, an infrared camera, an IMU, and so on, which are not shown in the drawings, and a combination of these devices may realize more abundant functions, such as 3D texture modeling, infrared face recognition, SLAM, and so on. Time-of-flight depth camera 10 may be embedded in a cell phone, tablet, computer, or other electronic product.
The processing circuit 13 may be a stand-alone dedicated circuit, such as a dedicated SOC chip, an FPGA chip, an ASIC chip, etc. including a CPU, a memory, a bus, etc., or may include a general-purpose processing circuit, such as when the depth camera is integrated into an intelligent terminal, such as a mobile phone, a television, a computer, etc., and the processing circuit in the terminal may be at least a part of the processing circuit 13. In some embodiments, the processing circuit 13 is configured to provide a modulation signal (emission signal) required when the light source 111 emits laser light, and the light source emits a pulse light beam to the object to be measured under the control of the modulation signal; the processing circuit 13 further provides a demodulation signal (acquisition signal) of a tap in each pixel of the image sensor 121, and the tap acquires a charge signal generated by a light beam including a reflected pulse light beam reflected back by the object under the control of the demodulation signal, and generally, there are some light beams such as background light, interference light, and the like in addition to the reflected pulse light beam reflected back by the object; the processing circuit 13 may also provide auxiliary monitoring signals such as temperature sensing, over-current, over-voltage protection, dropout protection, etc.; the processing circuit 13 may also be configured to store and perform corresponding processing on raw data acquired by each tap in the image sensor 121 to obtain specific position information of the object to be measured. The functions of the modulation and demodulation method, control, processing, etc. performed by the processing circuit 13 will be described in detail in the embodiments of fig. 2-3, and for convenience of explanation, the PM-ietf modulation and demodulation method is taken as an example for explanation.
Fig. 2 is a schematic diagram of a method for transmitting and collecting optical signals of a time-of-flight depth camera according to an embodiment of the present invention. Fig. 2 shows an exemplary timing diagram of a laser emission signal (modulation signal), a reception signal, and a collection signal (demodulation signal) in two frame periods T, where the meaning of each signal is: sp represents a pulsed emission signal of the light source, each pulsed emission signal representing a pulsed light beam; sr represents a reflected light signal of the pulse light reflected by the object, each reflected light signal represents a corresponding pulse light beam reflected by the object to be measured, a certain delay is arranged on a time line (horizontal axis in the figure) relative to a pulse emission signal, and the delayed time t is the flight time of the pulse light beam to be calculated; s1 represents a pulse pickup signal of a first tap of the pixel, S2 represents a pulse pickup signal of a second tap of the pixel, and S3 represents a pulse pickup signal of a third tap of the pixel, each pulse pickup signal representing that the tap picks up a charge signal (electrons) generated by the pixel in a period corresponding to the signal; tp is N × Th, where N is the number of taps participating in pixel electron collection.
The entire frame period T is divided into two periods Ta and Tb, where Ta denotes a period during which charge collection and storage is performed by each tap of the pixel, and Tb denotes a period during which the charge signal is read out. In the charge collection and storage period Ta, the collected signal pulse of the nth tap has a phase delay time of (n-1) × Th with respect to the laser emission signal pulse, respectively, and when the reflected light signal is reflected back to the pixel by the object, each tap collects electrons generated at the pixel during its pulse period. In the present embodiment, the first tap is triggered in synchronization with the laser emission signal, and when the reflected light signal is reflected back to the pixel by the object, the first tap, the second tap, and the third tap respectively and sequentially perform charge collection and storage to obtain the charge amounts q1, q2, and q3, respectively, so as to complete a pulse period Tp, which is 3Th for the case of 3 taps. In the embodiment shown in fig. 2, two pulse periods Tp are included in a single frame period, and 2 laser pulse signals are emitted in total, so that the total charge amount collected and read by each tap in the Tb period is the sum of the charge amounts corresponding to the two collected optical signals, it can be understood that, in a single frame period, the pulse period Tp or the number of times of emitting the laser pulse signals may be K times, K is not less than 1, and may be as high as several tens of thousands, or even higher, and the specific number is determined according to actual requirements, and in addition, the number of pulses in different frame periods may also be different.
Therefore, the total charge amount collected and read out by each tap in the Tb period is the sum of the charge amounts corresponding to the optical signals collected by each tap for a plurality of times in the whole frame period T, and the total charge amount of each tap in a single frame period can be represented as follows:
Qi＝∑qi,i＝1,2,3 (2)
the total charge amount in a single frame period of the first tap, the second tap and the third tap can be obtained as Q1, Q2 and Q3 according to the formula (2).
In the conventional modulation and demodulation method, the measurement range is limited to a single pulse width time Th, that is, assuming that the reflected light signal is collected by the first tap and the second tap (the first tap and the second tap will also collect the ambient light signal), the third tap is used for collecting the ambient light signal, so that based on the total charge quantity collected by each tap, the processing unit can calculate the total light flying distance of the pulsed light signal from the light emitted to the pixel reflected by the pixel according to the following formula:
further spatial coordinates of the target can be calculated based on the optical and structural parameters of the camera.
The conventional modem method has the advantages of simple calculation, but has the disadvantages that the measurement range is limited, the measured flight time is limited within Th, and the corresponding maximum flight distance measurement range is limited within c × Th.
In order to improve the measurement distance, the invention provides a novel modulation and demodulation method. Fig. 2 is a schematic diagram of optical signal emission and collection according to an embodiment of the present invention, in which the reflected optical signal can fall not only on the first tap and the second tap, but also on the second tap and the third tap, and even on the third tap and the first tap within the next pulse period Tp (for the case of at least two pulse periods Tp). The term "falling on the tap" as used herein means that the tap can be picked up. Since the total charge amount read in the period Tb is Q1, Q2, and Q3, unlike the conventional modulation and demodulation method, the present invention does not limit the tap or even the period of receiving the reflected light signal.
Considering that the amount of charge collected by the tap receiving the reflected light signal is larger than that of the tap containing only the background light signal, the processing circuit determines the three total amounts of charge Q1, Q2 and Q3 to determine to obtain the tap containing the excitation electrons of the reflected light signal and/or to obtain the tap containing only the background signal, and in actual use, electronic crosstalk may occur between the taps, for example, some reflected light signals may enter the tap originally used for obtaining only the background signal, and these errors are allowed and are also within the protection scope of the present scheme. Assuming that after the determination, two total charge amounts including the reflected light signal are respectively denoted as QA and QB and a total charge amount including only the background light signal is denoted as QO in sequence (the reflected light signal is received in time sequence), there are three possibilities for the three-tap image sensor:
(1)QA＝Q1,QB＝Q2,QO＝Q3；
(2)QA＝Q2,QB＝Q3,QO＝Q1；
(3) QA-Q3, QB-Q1 (next pulse period Tp), and QO-Q2;
the processing circuitry may then calculate the time of flight of the optical signal according to:
m in the equation reflects the delay of the tap into which the reflected optical signal first falls relative to the first tap, and m is 0, 1,2 for the three cases described above, respectively. That is, if the reflected light signal first falls into the nth tap, m is equal to n-1. n is the serial number of the tap corresponding to the QA, and the phase delay time of the tap with the serial number n relative to the emitted light pulse signal is (n-1) multiplied by Th; j refers to the tap collection in the jth pulse period after the reflected pulse beam is firstly emitted by the emission pulse beam (the pulse period in which the emission pulse is located is the 0 th pulse period after the emission pulse beam is emitted); th is the pulse width of the pulse acquisition signal of each tap; tp is the pulse period, Tp-N × Th, where N is the number of taps participating in pixel electron collection.
Comparing equation (4) with equation (3), it is clear that the measured distance is extended, and the maximum measured flying distance is extended from c · Th in the conventional method to c × Tp ═ c × N × Th in the present application, where N is the number of taps participating in pixel electron collection, which in the present example is 3, so that the above method achieves a measured distance 3 times that of the conventional method by the judgment mechanism, i.e., with respect to the conventional modem method.
The key to the above modem method is how to determine the tap into which the reflected optical signal falls. For this purpose, the present application provides the following methods of determination:
(1) single tap maxima method. The tap (denoted as Node) with the largest output signal (total charge amount) among the taps 1 to N (N is 3 in the above embodiment) is searchedx) Then according to Node1→Node2→…→NodeN→Node1Sequential nodel recording of → …xThe previous tap is Nodew(ii) a Node recordingxThe latter tap of (2) is Nodey. If NodewAnd NodeyTotal amount of electric charge Q ofw≥QyThen NodewNamely a tap A; if Qw<QyThen, thenNodexNamely tap a.
(2) Adjacent taps and the maximum method. According to Node first1→Node2→…→NodeN→Node1The sequence of → … calculates the Sum of the total charge of adjacent taps, Sum1＝Q1+Q2,Sum2＝Q2+Q3,…,SumN＝QN+Q1Finding the maximum term Sum thereinnThen tap n is tap a and the tap after tap n is tap B.
After the taps a and B are confirmed, the background signal amount is calculated in at least four ways:
(2) b, background; that is, the signal amount of one tap after the B tap is taken as the background signal amount.
(3) A, a front background; that is, the signal amount of the tap before the a tap is taken as the background signal amount.
(4) Average background; i.e. taking the average of all tap signal quantities except the A, B taps as the background signal quantity.
(5) Subtracting an average background; i.e. the average of A, B taps and the signal quantity of all taps except one tap after the B tap is taken as the background signal quantity.
Note that, when N is 3, that is, there are only 3 taps, the method (4) is not preferable, and the methods (1) to (3) are equivalent; when k is 4, the methods (3) and (4) are equivalent, and the method (3) may be selected preferentially in order to reduce crosstalk of the signal amount as much as possible. Method (4) may be preferentially selected when k > 4.
While a 3-tap pixel based modem method has been described in the above embodiments, it will be appreciated that this modem method is equally applicable to more-tap pixels, i.e., N >3, such as a maximum 4Th measured distance for a 4-tap pixel and a maximum 5Th measured distance for a 5-tap pixel. This measurement method extends the farthest measured flight time from the pulse width time Th to the entire pulse period Tp, relative to conventional PM-ietf measurement schemes, referred to herein as single frequency full period measurement schemes.
Although the modulation and demodulation method realizes the increase of the measurement distance by N-1 times, the measurement of a longer distance cannot be satisfied. For example, in a modulation and demodulation method based on 3-tap pixels, when the flight time corresponding to the object distance exceeds 3Th, the reflected light signal in a certain pulse period Tp will fall onto the tap in the subsequent pulse period first, and at this time, the flight time or distance cannot be accurately measured by using either formula (3) or formula (4). For example, when the reflected light signal in a certain pulse period Tp first falls into the nth tap in the subsequent jth pulse period, the flight time of the light signal corresponding to the real object is as follows:
where m is n-1, and n is the serial number of the tap corresponding to QA. Since the total charge of each tap is integrated with the charge accumulated during the pulse period, the specific value of j cannot be distinguished from the total charge of each tap outputted, which causes confusion in distance measurement.
FIG. 3 is a schematic diagram of optical signal emission and collection for a time-of-flight depth camera according to another embodiment of the present invention, which can be used to solve the above confusion problem. Unlike the embodiment shown in fig. 2, the present embodiment employs a multi-frequency modulation and demodulation method, that is, different modulation and demodulation frequencies are used in adjacent frames. For convenience of explanation in this embodiment, two adjacent frame periods are taken as an example, in the adjacent frame periods, the pulse emission frequency K is 2 (may be multiple times, and may be different for different frame frequencies), the number of taps N of the pixel is 3, the pulse period TPi is Tp1 and Tp2, the pulse width Thi is Th1 and Th2, the three taps accumulate charges Q11, Q12, Q21, Q22, Q31 and Q32 for each pulse, and the total charge amount is Q11, Q12, Q21, Q22, Q31 and Q32 according to formula (2).
It is assumed that the distance of the object in the period of the adjacent frame (or continuous frames) is not changed, so t in the period of the adjacent frame is the same. After receiving the total charge amount of each tap, the processing circuit measures the distance d (or t) in each frame period by using the modulation and demodulation method shown in fig. 2, and calculates QAi, QBi, and QOi in each frame period by using the above-described determination method, where i represents the i-th frame period, and in this embodiment, i is 1 or 2. In order to expand the measurement range and allow the reflected light signal to fall on the tap in the subsequent pulse period, assuming that the reflected light signal on a certain pixel in the ith frame period firstly falls on the mi-th tap in the ji-th pulse period after the pulse period of the transmitted light pulse (the pulse period of the transmitted pulse is the 0 th pulse period after the transmitted pulse beam is emitted), the corresponding flight time can be expressed as follows according to the formula (5):
considering that the object distance in the adjacent frame period is not changed, the following equation holds for the case of two consecutive frames in this embodiment:
(x1+m1)Th1+j1·Tp1＝(x2+m2)Th2+j2·Tp2 (7)
wherein the content of the first and second substances,
for the case of consecutive multiframes (assuming consecutive w frames, i.e., i ═ 1,2, …, w), the following holds:
(x1+m1)Th1+j1·Tp1＝(x2+m2)Th2+j2·Tp2
＝…＝xw+mwThw+jw·Tpw (8)
it is to be understood that when w is 1, this corresponds to the single frequency full period measurement scheme set forth above. When w is greater than 1, a group of ji combinations with the minimum ti variance under each modulation and demodulation frequency can be found out as solving values according to the remainder theorem or by traversing various ji combinations in the maximum measurement distance, and the solution of ji is completed; and then carrying out weighted average on the flight time or the measured distance solved under each group of frequencies to obtain the final flight time or the measured distance. By using the multi-frequency modulation-demodulation method,
the maximum measured time of flight extends to:
tmax＝LCM(Tp1,Tp2,…,Tpw) (9)
the maximum measured flight distance is extended to:
Dmax＝LCM(Dmax1,Dmax2,…,Dmaxw) (10)
wherein DmaxiLCM (lowest Common multiple) denotes taking the 'least Common multiple' (where 'least Common multiple' is a generalized extension of the least Common multiple of the integer domain, LCM (a, b) being defined as the smallest real number divisible by the real numbers a, b).
Suppose that in the embodiment shown in FIG. 3, Tp15ns, the maximum measured flight distance is 4.5 m; if TpThe maximum measured flight distance is 6m, 20 ns. If a multi-frequency modulation-demodulation method is used, for example, in one embodiment, Tp1＝15ns，Tp2The least common multiple of 15ns and 20ns is 60ns, 60ns corresponds to a maximum measuring distance of 18m, and the corresponding farthest measuring target distance can reach 9 m.
It will be appreciated that although in the embodiment shown in fig. 3, the distance of the object is calculated by at least two frames, in one embodiment, the number of the acquired frames may not be reduced by sequential frames, for example, in the case of obtaining a single time-of-flight measurement by sequential frames in the dual-frequency modem method, the first time-of-flight is calculated by 1 and 2 frames, the second time-of-flight is calculated by 2 and 3 frames, and so on, so as not to reduce the measurement frame rate.
It is understood that in the above-mentioned multi-frequency modulation and demodulation method, different measurement scenario requirements can be satisfied by adopting different frequency combinations, for example, the accuracy of the final distance resolution can be improved by increasing the number of measurement frequencies. In order to dynamically meet the measurement requirements under different measurement scenarios, in an embodiment of the present invention, the processing circuit adaptively adjusts the number of frequencies modulated and demodulated and the specific frequency combination through result feedback to meet the requirements under different measurement scenarios as much as possible. Specifically, in one embodiment, after calculating the current distance (or flight time) of the object, the processing circuit performs statistics on the target distance, and when most of the measurement targets are closer, the processing circuit may use a smaller number of frequencies to perform measurement to ensure a higher frame rate, and reduce the influence of target motion on the measurement result, and when there are more distant targets in the measurement targets, the processing circuit may appropriately increase the number of frequencies to be measured or adjust the combination of the measurement frequencies to ensure the measurement accuracy.
In addition, with respect to the method and the contents described in the embodiments of the present invention, it should be noted that, in any multi-frequency long-distance and single-frequency full-period measurement scheme based on three or more taps, no matter whether the waveform of the modem signal is continuous or discontinuous in the exposure time range, the measurement sequence of the different-frequency modem signals and the fine adjustment of the modulation frequency in the same exposure time should be within the protection scope of the present patent, and the exemplary description or the analysis algorithm performed to explain the principle of the present patent is only one exemplary description of the present patent and should not be considered as a limitation to the present patent content. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.
For the time flight depth cameras in the above embodiments, active light emission is required due to the fact that the time flight depth cameras are based on the iToF technology, when a plurality of iToF depth cameras work at a short distance at the same time, the acquisition module of the device receives not only the light signal from the light emitting unit of the device reflected by the object, but also the emitted light or reflected light from other devices, and the light signal from the other devices interferes with the amount of electrons collected between the taps, and further adversely affects the accuracy and precision of the final target distance measurement. The invention provides the following ways to eliminate the correlated interference among multiple devices.
(1) And (4) frequency conversion scheme. The frequency conversion scheme is that in the actual measurement process, when the frequency of the modulation and demodulation signal is set to be fm0When the modulation and demodulation signal frequency actually used is fm＝fm0+ Δ f. Where Δ f is a random frequency offset. In this way, the working frequency between the single units can be adjusted toAnd a random deviation is less, so that the mutual interference among the devices is remarkably reduced.
(2) Random exposure time. The exposure time of the camera is relatively limited with respect to the overall operating time. Taking the double frequency as an example, the exposure time for obtaining each depth frame data only needs 2 times at most, and when the single exposure time is 1ms and the depth frame rate is 30fps, the exposure time ratio in the whole working time is only 6%. The selection of the exposure time is generally uniformly distributed in the whole working time, and in order to reduce the mutual interference among the devices, a random offset can be added on the basis of uniform distribution of the exposure time, so that the exposure imaging time among different devices can be staggered as much as possible, and the mutual interference can be avoided. In order to ensure that the time intervals between the acquisition of the images are as uniform as possible, it may be chosen to use the same time offset over a relatively long operating time slice (e.g. 1s) to ensure that the image time intervals within that time slice are the same.
The invention has the advantages that the contradiction that the pulse width is directly proportional to the measurement distance and the power consumption and is negatively related to the measurement precision in the existing PM-iToF measurement scheme is eliminated; the extension of the measuring distance is not limited by the pulse width, so that the lower measuring power consumption and higher measuring precision can be still maintained under the condition of longer measuring distance. Compared with the CW-iToF measurement scheme, in the scheme, only one set of modulation and demodulation frequency needs to expose and output the signal quantity of three taps once to obtain one frame of depth information, so that the overall measurement power consumption is obviously reduced, and the measurement frame frequency is improved. Therefore, the scheme has obvious advantages compared with the existing iToF technical scheme.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.
Claims (10)
1. A time-of-flight depth camera, comprising:
the transmitting module comprises a light source and is used for transmitting a pulse light beam to the object to be detected;
the acquisition module comprises an image sensor consisting of at least one pixel, each pixel comprises at least 3 taps, and the at least 3 taps are used for acquiring charge signals generated by reflected pulse beams reflected by the object to be measured or charge signals of background light in different pulse periods;
a processing circuit for receiving data of the charge signals of the at least 3 taps; judging the data of the charge signal to determine whether the charge signal of the reflected pulse light beam is contained in the data of the charge signal; and calculating the flight time of the pulse light beam and/or the distance of the object to be detected according to the judgment result.
2. The time-of-flight depth camera of claim 1, wherein the processing circuit calculates the time-of-flight of the pulsed light beam according to:
wherein QA is an amount of charge of the charge signal containing the reflected pulsed light beam collected by a first one of the taps acquired after the judgment; QB is the amount of charge of the charge signal containing the reflected pulse beam collected by the second one of the taps acquired after the judgment; QO is the charge amount of the charge signal collected by the tap and only containing the background light; m is n-1, wherein n is the serial number of the tap corresponding to the QA; th is the pulse width of the pulse acquisition signal of each tap.
3. The time-of-flight depth camera of claim 2, wherein the determining comprises a single-tap maximum value method, that is, by sequentially searching and acquiring a first tap with the largest charge amount of the charge signal in the at least 3 taps, if the charge amount of the charge signal of a second tap before the first tap is larger than that of a third tap after the first tap, the charge amount of the charge signal collected by the second tap and the first tap is QA and QB, respectively; and if the charge amount of the charge signal of the second tap before the first tap is smaller than that of the charge signal of the third tap after the first tap, the charge amounts of the charge signals collected by the first tap and the third tap are respectively QA and QB according to the sequence of tap serial numbers.
4. The time-of-flight depth camera of claim 2, wherein the determining comprises adjacent taps and a maximum value method, wherein the maximum value items are found after sequentially calculating the charge amounts of the charge signals of the adjacent taps, and the charge amounts of the charge signals collected by the two taps corresponding to the maximum value items are QA and QB, respectively.
5. The time depth of flight camera of claim 2, wherein the QO is obtained by at least one of:
taking the charge amount of the collected charge signal of a tap behind the tap corresponding to the QB; or, the charge quantity of the charge signal collected by the previous tap corresponding to the tap by QA is taken; or, taking the average value of the charge quantity of the charge signals collected by all the taps except the tap corresponding to the QA and the QB; or, taking the average value of the charge quantity of the charge signals collected by the QA, the tap corresponding to the QB and all the taps except the tap behind the tap corresponding to the QB.
6. A distance measuring method of single frequency modulation and demodulation is characterized by comprising the following steps:
using a light source to emit a pulse light beam to an object to be measured;
collecting charge signals of reflected pulse light beams reflected by the object to be measured by using an image sensor consisting of at least one pixel, wherein each pixel comprises at least 3 taps, and at least 3 taps are used for collecting the charge signals or the charge signals of background light in different pulse periods; receiving data of the charge signals of the at least 3 taps;
judging the data of the charge signal to determine whether the charge signal of the reflected pulse light beam is contained in the data of the charge signal;
and calculating the flight time of the pulse light beam and/or the distance of the object to be detected according to the judgment result.
7. A method of distance measurement for single frequency modem according to claim 6 wherein said time of flight is calculated according to the formula:
wherein QA is an amount of charge of the charge signal containing the reflected pulsed light beam collected by a first one of the taps acquired after the judgment; QB is the amount of charge of the charge signal containing the reflected pulse beam collected by the second one of the taps acquired after the judgment; QO is the charge amount of the charge signal collected by the tap and only containing the background light; m is n-1, wherein n is the serial number of the tap corresponding to the QA; th is the pulse width of the pulse acquisition signal of each tap.
8. The single-frequency modem distance measuring method according to claim 7, wherein said determining comprises a single-tap maximum value method, that is, by sequentially searching and obtaining a first tap with the largest charge amount of the charge signal among said at least 3 taps, if the charge amount of the charge signal of a second tap before said first tap is larger than that of a third tap after said first tap, the charge amount of the charge signal collected by the second tap and the first tap is said QA and said QB, respectively; and if the charge quantity of the charge signal of a second tap before the first tap is smaller than that of a third tap after the first tap, the charge quantities of the charge signals collected by the first tap and the third tap are respectively QA and QB.
9. The single-frequency modem distance measuring method according to claim 7, wherein said determining comprises calculating the charge amount of the charge signal of the adjacent tap and searching the maximum term therein, and the charge amounts of the charge signals collected by the two taps corresponding to the maximum term are QA and QB according to the tap serial number.
10. The method for distance measurement of single frequency modem of claim 7 wherein said QO is obtained by at least one of:
taking the charge amount of the collected charge signal of a tap behind the tap corresponding to the QB; or, the charge quantity of the charge signal collected by the previous tap corresponding to the tap by QA is taken; or, taking the average value of the charge quantity of the charge signals collected by all the taps except the tap corresponding to the QA and the QB; or, taking the average value of the charge quantity of the charge signals collected by the QA, the tap corresponding to the QB and all the taps except the tap behind the tap corresponding to the QB.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910385779.6A CN110221273B (en) | 2019-05-09 | 2019-05-09 | Time flight depth camera and distance measuring method of single-frequency modulation and demodulation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910385779.6A CN110221273B (en) | 2019-05-09 | 2019-05-09 | Time flight depth camera and distance measuring method of single-frequency modulation and demodulation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110221273A CN110221273A (en) | 2019-09-10 |
| CN110221273B true CN110221273B (en) | 2021-07-06 |
Family
ID=67820684
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910385779.6A Active CN110221273B (en) | 2019-05-09 | 2019-05-09 | Time flight depth camera and distance measuring method of single-frequency modulation and demodulation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110221273B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110361751B (en) * | 2019-06-14 | 2021-04-30 | 奥比中光科技集团股份有限公司 | Time flight depth camera and distance measuring method for reducing noise of single-frequency modulation and demodulation |
| CN111123285B (en) * | 2019-12-30 | 2022-05-13 | 宁波飞芯电子科技有限公司 | Signal receiving system and method based on array type sensor and array type sensor |
| CN111664798B (en) * | 2020-04-29 | 2022-08-02 | 奥比中光科技集团股份有限公司 | Depth imaging method and device and computer readable storage medium |
| CN112532970B (en) * | 2020-10-26 | 2022-03-04 | 奥比中光科技集团股份有限公司 | Tap non-uniformity correction method and device of multi-tap pixel sensor and TOF camera |
| CN115657053A (en) * | 2021-07-07 | 2023-01-31 | 奥比中光科技集团股份有限公司 | ITOF ranging system and method for shielding fuzzy distance value |
| CN115047436A (en) * | 2022-08-16 | 2022-09-13 | 武汉极动智能科技有限公司 | Anti-interference method, device, system and storage medium |
| CN116320667A (en) * | 2022-09-07 | 2023-06-23 | 奥比中光科技集团股份有限公司 | Depth camera and method for eliminating motion artifact |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101065683A (en) * | 2004-11-23 | 2007-10-31 | Iee国际电子工程股份公司 | Error compensation method for a 3D camera |
| CN101784911A (en) * | 2007-08-22 | 2010-07-21 | 浜松光子学株式会社 | Solid state imaging device and distance image measurement device |
| CN103155150A (en) * | 2010-10-12 | 2013-06-12 | 浜松光子学株式会社 | Range sensor and range image sensor |
| CN106471391A (en) * | 2014-05-08 | 2017-03-01 | 浜松光子学株式会社 | Range unit and the driving method of range unit |
| CN107533136A (en) * | 2015-06-24 | 2018-01-02 | 株式会社村田制作所 | Range sensor |
| CN109190469A (en) * | 2018-07-27 | 2019-01-11 | 阿里巴巴集团控股有限公司 | A kind of detection method and device, a kind of calculating equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101711061B1 (en) * | 2010-02-12 | 2017-02-28 | 삼성전자주식회사 | Method for estimating depth information using depth estimation device |
-
2019
- 2019-05-09 CN CN201910385779.6A patent/CN110221273B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101065683A (en) * | 2004-11-23 | 2007-10-31 | Iee国际电子工程股份公司 | Error compensation method for a 3D camera |
| CN101784911A (en) * | 2007-08-22 | 2010-07-21 | 浜松光子学株式会社 | Solid state imaging device and distance image measurement device |
| CN103155150A (en) * | 2010-10-12 | 2013-06-12 | 浜松光子学株式会社 | Range sensor and range image sensor |
| CN106471391A (en) * | 2014-05-08 | 2017-03-01 | 浜松光子学株式会社 | Range unit and the driving method of range unit |
| CN107533136A (en) * | 2015-06-24 | 2018-01-02 | 株式会社村田制作所 | Range sensor |
| CN109190469A (en) * | 2018-07-27 | 2019-01-11 | 阿里巴巴集团控股有限公司 | A kind of detection method and device, a kind of calculating equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110221273A (en) | 2019-09-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110221274B (en) | Time flight depth camera and multi-frequency modulation and demodulation distance measuring method | |
| CN110221273B (en) | Time flight depth camera and distance measuring method of single-frequency modulation and demodulation | |
| CN110320528B (en) | Time depth camera and noise reduction distance measurement method for multi-frequency modulation and demodulation | |
| CN110361751B (en) | Time flight depth camera and distance measuring method for reducing noise of single-frequency modulation and demodulation | |
| CN110596722B (en) | System and method for measuring flight time distance with adjustable histogram | |
| CN110221272B (en) | Time flight depth camera and anti-interference distance measurement method | |
| WO2021008209A1 (en) | Depth measurement apparatus and distance measurement method | |
| CN110596721B (en) | Flight time distance measuring system and method of double-shared TDC circuit | |
| CN110596725B (en) | Time-of-flight measurement method and system based on interpolation | |
| CN110596724B (en) | Method and system for measuring flight time distance during dynamic histogram drawing | |
| WO2020223981A1 (en) | Time flight depth camera and multi-frequency modulation and demodulation distance measuring method | |
| CN110596723B (en) | Dynamic histogram drawing flight time distance measuring method and measuring system | |
| CN110187355B (en) | Distance measurement method and depth camera | |
| CN113661407A (en) | Method for measuring optical crosstalk in a time-of-flight sensor and corresponding time-of-flight sensor | |
| TWI780462B (en) | Distance video camera device and distance video camera method | |
| JP2003510561A (en) | CMOS compatible 3D image sensor IC | |
| CN111885316B (en) | Image sensor pixel circuit, image sensor and depth camera | |
| WO2021103428A1 (en) | Depth measurement system and method | |
| US20210181316A1 (en) | Time-of-flight-based distance measurement system and method | |
| CN111045029A (en) | Fused depth measuring device and measuring method | |
| CN110632576A (en) | Time coding demodulation processing circuit and method | |
| WO2021113001A1 (en) | Configurable array of single-photon detectors | |
| WO2020223980A1 (en) | Time-of-flight depth camera, and distance measurement method employing single-frequency modulation/demodulation | |
| CN209746131U (en) | Three-dimensional real-time imaging laser radar system based on area array CCD | |
| WO2020223982A1 (en) | Time of flight depth camera and anti-interference distance measurement method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 11-13 / F, joint headquarters building, high tech Zone, 63 Xuefu Road, Yuehai street, Nanshan District, Shenzhen, Guangdong 518000 Applicant after: Obi Zhongguang Technology Group Co., Ltd Address before: 12 / F, joint headquarters building, high tech Zone, 63 Xuefu Road, Nanshan District, Shenzhen, Guangdong 518000 Applicant before: SHENZHEN ORBBEC Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |