US11885880B2 - Unwrapped phases for time-of-flight modulation light - Google Patents
Unwrapped phases for time-of-flight modulation light Download PDFInfo
- Publication number
- US11885880B2 US11885880B2 US16/528,472 US201916528472A US11885880B2 US 11885880 B2 US11885880 B2 US 11885880B2 US 201916528472 A US201916528472 A US 201916528472A US 11885880 B2 US11885880 B2 US 11885880B2
- Authority
- US
- United States
- Prior art keywords
- modulation light
- phase
- frequency
- frequencies
- unwrapping
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 claims description 73
- 239000013598 vector Substances 0.000 claims description 34
- 230000008569 process Effects 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 10
- 238000009472 formulation Methods 0.000 claims description 6
- 239000000203 mixture Substances 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 5
- 238000010408 sweeping Methods 0.000 claims description 4
- 230000003044 adaptive effect Effects 0.000 abstract description 4
- 238000003384 imaging method Methods 0.000 description 13
- 230000003287 optical effect Effects 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 6
- 230000002093 peripheral effect Effects 0.000 description 5
- 238000005259 measurement Methods 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 230000010363 phase shift Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000001276 controlling effect Effects 0.000 description 2
- 238000009795 derivation Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 101000822695 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C1 Proteins 0.000 description 1
- 101000655262 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C2 Proteins 0.000 description 1
- 101000655256 Paraclostridium bifermentans Small, acid-soluble spore protein alpha Proteins 0.000 description 1
- 101000655264 Paraclostridium bifermentans Small, acid-soluble spore protein beta Proteins 0.000 description 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005684 electric field Effects 0.000 description 1
- 230000005670 electromagnetic radiation Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000000819 phase cycle Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000009885 systemic effect Effects 0.000 description 1
- 230000026683 transduction Effects 0.000 description 1
- 238000010361 transduction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
- G01S17/32—Systems determining position data of a target for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
- G01S17/32—Systems determining position data of a target for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated
- G01S17/36—Systems determining position data of a target for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated with phase comparison between the received signal and the contemporaneously transmitted signal
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
- G01S17/894—3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/491—Details of non-pulse systems
- G01S7/4912—Receivers
- G01S7/4913—Circuits for detection, sampling, integration or read-out
- G01S7/4914—Circuits for detection, sampling, integration or read-out of detector arrays, e.g. charge-transfer gates
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/491—Details of non-pulse systems
- G01S7/4912—Receivers
- G01S7/4915—Time delay measurement, e.g. operational details for pixel components; Phase measurement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/141—Control of illumination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/143—Sensing or illuminating at different wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Definitions
- Time-of-flight (ToF) imaging systems may be used to produce a depth image of an environment, with each pixel of the depth image representing a distance to a corresponding point in the environment.
- the distance to a point on an imaged surface in the environment is determined based on the length of the time interval in which light emitted by the imaging system travels out to that point and then returns back to a sensor array in the imaging system (i.e., the ToF).
- An optical ToF camera measures this interval for many points on the surface and thereby assembles a depth image in which a depth coordinate for each pixel in the depth image is proportional to the ToF observed at that pixel.
- a scene is illuminated with modulation light having two or more frequencies that do not have a common integer denominator.
- the modulation light that is reflected off objects within the scene is received at a sensor array of the time-of-flight camera.
- the received modulation light is then processed to determine unwrapped phases for each of the two or more frequencies of modulation light.
- FIG. 1 shows aspects of example electronic devices having embedded or peripheral digital cameras.
- FIG. 2 shows aspects of an example digital camera.
- FIG. 3 shows an example method for determining unwrapped phases via a time-of-flight camera.
- FIG. 4 shows an example method for adjusting phase unwrapping parameters based on a noise condition.
- FIG. 5 A shows an example map of unwrapping passing rate.
- FIG. 5 B shows a scaled version of the example map of FIG. 5 A .
- FIG. 6 A shows an example plot of phase unwrapping up to a distance of 16.67 m for a noise condition.
- FIG. 6 B shows a plot of the example phase unwrapping of FIG. 6 A for a zero noise condition.
- FIG. 6 C shows an example plot of phase unwrapping up to a distance of 50 m for a noise condition.
- FIG. 6 D shows a plot of the example phase unwrapping of FIG. 6 C for a zero noise condition.
- FIG. 6 E shows a truncated version of the example plot of FIG. 6 C .
- FIG. 6 F shows a truncated version of the example plot of FIG. 6 D .
- FIG. 7 schematically shows a non-limiting embodiment of a computing system.
- Optical time-of-flight (ToF) imaging has become the predominant depth-imaging technology, owing to the development of high-speed, high-resolution optical sensor arrays.
- ‘Phase-based’ optical ToF imaging is an important variant of this technique, in which depth is computed based on the phase lag of modulated light reflected back from the subject.
- Devices employing this technique are increasingly found in both industrial and consumer applications, including device automation applications, gaming and virtual reality applications, biometric and facial recognition applications, etc.
- This disclosure is directed to an improvement to ToF imaging that allows for multi-frequency imaging using modulation light having two or more frequencies that do not have a common integer denominator. In turn, this results in the ability to adjust the frequencies utilized and thus improve depth measurements, improve power consumption, and/or provide other advantages.
- FIG. 1 shows aspects of four different examples of electronic devices ( 100 A-D) having embedded or peripheral digital cameras ( 102 A-D).
- Device 100 A is a smartphone that includes a camera 102 A.
- Device 100 B is a personal computer that includes a web camera 102 B.
- Device 100 C is a video game system that includes a peripheral 3D camera 102 C.
- Device 100 D is a virtual-reality headset that includes a 3D camera 102 D.
- the correction methods disclosed herein may be applicable to the output of these and other types of user-facing and/or world-facing digital cameras, in these and other device contexts.
- FIG. 2 shows aspects of an example imaging environment 200 including digital camera 202 in one implementation.
- Digital camera 202 includes a sensor array 204 comprising plural sensor elements 206 (e.g., pixels), a controller 208 , and an objective lens system 210 .
- Objective lens system 210 is configured to focus an image of at least one surface 220 of photographic subject 222 onto sensor array 204 .
- Controller 208 is configured to gather and process data from the various sensor elements 206 and thereby construct a digital image of the subject. Controller 208 may be implemented across one or more computing devices, examples of which are discussed herein and with regard to FIG. 7 .
- Digital images constructed by controller 208 may be represented as a numeric array with a value S j provided for each of a set of pixels (X, Y) j .
- the X, Y position of each pixel of the digital image is mapped to an associated sensor element 206 of sensor array 204 , and, via objective lens system 210 , to a corresponding associated locus 224 of surface 220 .
- the mapping of image pixels to sensor elements may be a 1:1 mapping, but other mappings may be used as well, such as 1:4, 4:1, among others.
- digital camera 202 may be configured to acquire a time-resolved sequence of digital images of subject 222 —i.e., video.
- S j may be a real- or integer-valued scalar that specifies the brightness of pixel (X, Y) j .
- S j may be a vector of real or integer values that specifies the color of pixel (X, Y) j —e.g., using scalar component values for the red, green, and blue color channels.
- S j may be preprocessed to include a complex value a+b ⁇ square root over ( ⁇ 1 ) ⁇ , where a and b are integers or real numbers.
- a complex value S j may be used to represent the signal response of the sensor elements of an optical ToF camera that employs continuous-wave (CW) modulation and phase discrimination to resolve radial distance.
- CW-ToF camera Such a camera is referred to herein as a ‘CW-ToF camera’.
- Digital camera 202 of FIG. 2 is a CW-ToF camera configured to resolve radial distance Z to a plurality of points j on surface 220 of photographic subject 222 .
- CW-ToF camera 202 includes a modulated light emitter 230 and a sensor array 204 with an analog and/or digitally modulated electronic shutter 232 .
- Modulated light emitter 230 may be configured to emit electromagnetic radiation having any frequency detectable by sensor elements 206 .
- Modulated light emitter 230 may be an infrared and/or near-infrared light-emitting diode (LED) or laser diode (LD), for example, and the sensor array may be a high-resolution array of complementary metal-oxide semiconductor (CMOS) sensor elements 206 .
- CMOS complementary metal-oxide semiconductor
- Other optical ToF cameras may include different optical, light-emitter and/or sensor-array variants—e.g., a charge-coupled device (CCD) sensor array or a microlens objective array.
- CCD charge-coupled device
- Electronic shutter 232 may refer to a controlled voltage signal, including any suitable modulation waveform, with a regulatory bias optionally applied concurrently to certain electrode structures of the various sensor elements 206 of sensor array 204 .
- the electrode structures receiving the controlled voltage signal with a bias may include current collectors that, depending on the level of the biased voltage signal, cause photoelectrons created within sensor elements 206 to drift to the current collectors and be measured as current or as voltage via a current to voltage convertor.
- the electrode structures receiving the controlled biased voltage signal may include gates that, depending on the level of the biased voltage signal, encourage the photoelectrons to drift towards the current collectors or current to voltage convertors.
- Controller 208 of CW-ToF camera 202 includes a modulation-control engine 240 configured to modulate light emitter 230 and synchronously modulate electronic shutter 232 of sensor array 204 .
- light emitter 230 and electronic shutter 232 are modulated at one or more predetermined frequencies, with a predetermined, angular phase offset ⁇ ′ controlling the retardance of the electronic-shutter modulation relative to the light-emitter modulation.
- ‘modulation’ refers to a sinusoidal or digitized quasisinusoidal waveform, and/or a basic harmonic of a rectangular waveform, which simplifies analysis. This feature is not strictly necessary, however, as modulation by other waveforms may be used instead.
- sensor array 204 images the component of the reflected light that lags the emitter modulation by each of a series of predetermined phase offsets ⁇ ′.
- Shutter-acquisition engine 242 of controller 208 is configured to interrogate sensor array 204 to retrieve a resulting signal value S j from each sensor element 206 .
- One digital image captured in this manner is called a ‘raw shutter.’
- a raw shutter 243 may be represented as a numeric array with a ⁇ ′-specific real intensity value S j provided for each sensor element 206 and associated with coordinates (X, Y) j that specify the position of that particular sensor element 206 in sensor array 204 .
- phase image is a numeric array with ⁇ j specified for each sensor element j and associated with coordinates (X, Y) j that specify the position of that particular sensor element 206 in sensor array 204 .
- each signal value S j is a complex number a+b ⁇ square root over ( ⁇ 1 ) ⁇ , where a is the signal component in phase with the emitter modulation, and bis the signal component that lags the emitter modulation by 90°.
- a given phase image may be processed by replacing each complex signal value S j by its modulus, or by the square of its modulus.
- An image of that kind is referred to herein as an ‘active-brightness’ image.
- the solution above is unique when the entire range of depth values Z j is no larger than half of the distance traveled by light in one modulation period, c/(2f), in which case N is a constant. Otherwise, the solution is underdetermined and periodic. In particular, surface points at depths that differ by any integer multiple of c/(2 f) are observed at the same phase lag (p. Depth-image data resolved only to that degree—e.g., data from a single phase image or corresponding triad of raw shutters—is said to be ‘aliased’ or ‘wrapped’.
- An unwrapped depth image, or ‘radial-distance map’ may be represented as a numeric array with a radial distance value Z j provided for each pixel and associated with coordinates (X, Y) j that specify the pixel position.
- modulation-control engine 240 may simultaneously output a modulation signal to a phase shifter.
- the phase shifting rf-signal may then be used to demodulate the signal at each pixel.
- the pixels of an unwrapped depth image may be classified into one or more segments according to a single-tier or multi-tier (i.e., hierarchical) classification scheme.
- Segmentation engine 246 of controller 208 may be configured to enact the classification.
- pixels may be classified as foreground or background.
- pixels classified as foreground may be further classified as a human subject.
- pixels classified as a human subject may be further classified as a ‘subject head’, ‘subject hand’, etc.
- a classified digital image may be represented as a numeric array with a signal value S and class value C j provided for each pixel and associated with coordinates (X, Y) j that specify the pixel position.
- model fitting may be applied to track the motion of the classified depth-image segments from frame to frame.
- classified depth-image segments corresponding to the hands may be segmented from the rest of the subject.
- the hand segments can then be tracked through the sequence of depth-image frames and/or fit to a kinematic model. Tracked hand segments can be used as input for virtual-reality video games or as gesture input for controlling a computer, for example. Tracking may be enacted in tracking engine 248 of controller 208 .
- the approach herein extends to various other segmentation and tracking tasks that may be performed on the output of a phase-based CW-ToF camera.
- Using a single higher radio frequency may increase depth resolution but does so with limited non-ambiguity (e.g., 0.75 m at 200 MHz).
- multiple higher radio frequencies are usually applied to extend the measurement distance following a phase unwrapping technique.
- Multiple depth maps of the same environment may be captured using different modulation frequencies of emitted light, improving accuracy for environments with varying reflectivity and/or partially occluded objects.
- phase domain phase vector domain
- the base frequency is relatively low, setting the unwrapping distance, and the applied frequency is relatively high. Without a common integer, the frequencies will still meet, but that could occur at infinity or some other irrational number.
- Example systems and methods are presented in this disclosure for phase unwrapping in the complex domain wherein vector phase unwrapping is converted into a scalar phase unwrapping process.
- Such an approach allows a 3D ToF camera to use any optimized set of frequencies without being constrained by common integer denominators, thus enabling adaptive systemic power efficiency.
- the computational power needed to perform unwrapping of scalars is less than phase vector-based methods for an equal or even higher number of frequencies used.
- the scalar unwrapping method in the complex domain may be weighted based on active brightness, therefore generating a system that is auto adaptive, as the system ages or unbalanced phase noise is encountered at each frequency.
- Such unwrapping methods may use the same generic formulation to handle any number of frequencies as well as any type of frequency combinations, allowing for adaptive, user friendly ToF cameras which may self-optimize for a given imaging environment. While this disclosure focuses on modulated light based amplitude continuous wave applications, such as infrared imaging, the technology disclosed herein may also be applicable to radar and other distance measurement techniques that rely upon phase detection of waveforms at different frequencies in order to determine distance.
- FIG. 3 shows a flow-chart for an example method 300 for phase unwrapping using a time-of-flight camera, such as digital camera 202 .
- method 300 includes illuminating a scene with modulation light (e.g., infrared light) having two or more frequencies that do not have a common integer denominator.
- the set of two or more frequencies may include integer frequencies and/or fractional frequencies.
- a modulation controller such as modulation controller 240 may be configured to generate radio-frequency modulation signals.
- a modulated light emitter, such as modulated light emitter 230 may be configured to receive radio-frequency modulation signals from the modulation controller and to illuminate a scene with modulation light.
- sets of example frequencies may include ⁇ 151, 167, 197 ⁇ , ⁇ 160.123, 168.456, 198.789 ⁇ , ⁇ 96.9, 102, 114, 141.3, 2016 ⁇ , and ⁇ 149, 168, 211, 225, 148 ⁇ .
- method 300 includes, at a sensor array, receiving the modulation light reflected off objects within the scene.
- the sensor array may include a plurality of sensor elements, subsets of which are tuned and/or filtered to receive infrared light at one or more of the emitted frequencies, as described for sensor array 204 .
- method 300 includes processing the received infrared light to determine wrapped phase vectors for each frequency of modulation light.
- CM m is a common mode of the capture at modulation frequency f m , representing a DC component of the signal.
- AB m is an active brightness at modulation frequency f m , which is related to the phase signal and contributed by the active light emitted from the modulated light emitter. This allows for differences in illumination power across frequencies to be accounted for in downstream equations.
- the phase shift steps may be the same at each frequency condition.
- N k represents the total number of phase shifting steps at each frequency, with the assumption that they are the same at each frequency.
- N m represents the total number of frequencies output by the modulated light emitter.
- Eq. 3 does not include high orders of harmonics, it may be applied to unwrapping the captured phase vectors. Eq. 3 may be converted to a normal equation and its accompanying derivations:
- the normal equation applies a least-squares method that effectively filters Gaussian noise from the phase vectors using a numeric filter in the complex domain.
- amplitude modulation across the different frequencies may be accounted for.
- phase error tends to be large, containing more phase noise.
- Accounting for active brightness allows for weighting of the different frequencies to reduce the error contributed to the entire system. Phase unwrapping thus becomes robust when phase error is unbalanced.
- the three system variables may be derived based on solving Equations 4a, 4b, and 4c for each specific frequency f n :
- CM m 1 ⁇ / ⁇ N k ⁇ ⁇ k ⁇ ( V m , k ) ;
- Eq . ⁇ 5 ⁇ b AB m 2 ⁇ / ⁇ N k ⁇ I ⁇ ( m ) 2 + R ⁇ ( m ) 2 , .
- I(m) ⁇ k [V m,k sin( ⁇ k )], and represents the imaginary part of a complex value
- R(m) ⁇ k [V m,k cos( ⁇ k )], and represents the real part of a complex value
- ⁇ d,m 2 ⁇ f m t dm , and represents the wrapped phase in a 2 ⁇ T module following the arctan operation expressed in Eq.5b
- t dm represents a wrapped time-of-flight at each frequency f m .
- method 300 includes determining unwrapped phases for each of the two or more frequencies of modulation light.
- an unwrapped phase represents a phase lag corresponding to time-of-flight of modulation light.
- Previously-used methods of phase unwrapping are accomplished via processing the phase vector set ⁇ d,m ⁇ to find the true time-of-flight.
- these methods are inflexible to frequency selection, and are increasingly computationally expensive as the number of frequencies utilized increases. As such, optimization of frequency sets using previously-used methods is challenging, resulting in non-optimal camera operation, which may be power-inefficient.
- method 300 includes converting the wrapped phase vectors to phase scalars in a complex domain. For example, by further solving Eq. 5c with all frequencies of f m together, an unwrapping formulation may be generated that is constrained by the complex value R(m) and I(m), unitarily in the complex domain:
- T d represents the unwrapped time-of-flight related to one single specific frequency f m0 , for example, the highest frequency.
- M is the wrapped period number of 2 T module at frequency f m0 .
- the integer M makes function U M of Eq.7 reach an extremum (e.g., a minimum value)
- the M value is the repeating number 2 ⁇ , and thus it the unwrapping process can be assumed to be resolving.
- This formulation yields phase scalars that are expressed with a single unknown integer (M) constrained in the complex domain, thus converting the vector phase unwrapping process into a scalar phase unwrapping process.
- method 300 includes determining unwrapped phases in the complex domain based on the phase scalars.
- a value for M may be determined based on a value that minimizes the value of U M , such that 0 ⁇ M ⁇ M max .
- Eq. 7 may be rewritten using Eq. 5b and Eq. 5c:
- Eq. 8a may be considered equivalent to Eq.6 and Eq.7.
- N k 2 may be dropped off, but is kept here for completion.
- the unwrapped phase at all other frequency f m may be expressed as integer numbers of multiples of 2 ⁇ in the ideal case or assumed ideal case.
- the unwrapped phase term may be expressed with an integer (N 0m ) and a residual error of the integer ⁇ N m , i.e.:
- AB m AB m ⁇ ⁇ 0 which is may be a constant for each sensor element regardless of the signal strength in the ideal case scenario.
- the residual integer number ⁇ N m can be expressed as a round operation with the relationship:
- the total integer number counting from 0 to M max is equal to the wrapped 2 ⁇ counts within a designed unwrapping distance at the specific frequency f m0 , for example at the highest frequency used by the system.
- a value for M max may be determined for the wrapped phase vectors within a determined maximum unwrapping distance. For example, at a designed unwrapping distance of 15 m,
- M may be set equal to 0, 1, 2 . . . (M max -1) e.g., 0, 1, 2 . . . 18, within Eq.6e or 6f to determine which one M 0 value minimizes U M .
- Unwrapped phases for each frequency of modulation light may then be determined based on at least M This yields all unwrapped phase ⁇ m expressed with M 0 together with wrapped phase ⁇ d,m as:
- Method 300 and accompanying formulations thus enable the use of multiple frequencies that are not constrained by integral relations with a common denominator among the frequencies, thus generating flexibility that may allow for switchable, adaptable ToF camera systems that remain economical.
- Method 300 allows a 3D ToF system to use whichever frequency configuration are determined to achieve accurate depth measurements with reduced power consumption within a range of environments.
- Frequency combinations may be selected based on the unwrapping distance, to improve power efficiency, to reduce noise related to depth jitter, etc.
- a scene may be illuminated with a first set of three or more frequencies of modulation light.
- the scene may be illuminated with a second set of three or more frequencies of modulation light, wherein at least one frequency of the second set is different than any frequency in the first set.
- This unwrapping method maps a multiple dimensional phase vector to a scalar with only one unknown integer (M) thus reducing the computational burden when the number of frequencies of modulation light is increased.
- the system robustness may be auto adapted or weighted by the system active brightness AB m of the received modulation light.
- Eq.8 reveals the robust unwrapping result at a Gaussian noise condition above a threshold.
- Eq. 10 is effective for determining unwrapped phases, but is not optimized for a significant Gaussian noise condition. As such, the unwrapping results may not be accurate when the phase noise exceeds a threshold. This can be described as unwrapping failure probability or unwrapping passing rate at noisy condition, which is defined as a ratio of:
- Method 400 of FIG. 4 shows an example method for adjusting phase unwrapping parameters based on a noise condition.
- a ToF camera employing a 3-frequency system ⁇ f1, f2, f3 ⁇ is used to describe the robust optimization work for increasing the unwrapping passing rate at a given noisy condition.
- similar methods may be applied to systems using smaller or larger frequency sets.
- method 400 includes determining a first frequency of light that is to be output by the light emitter.
- the first frequency of light may be the highest frequency of light output by the light emitter.
- the first frequency may be arbitrary, predetermined, or based on operating conditions.
- a highest frequency may be based on a desired depth resolution, and/or a desired range for the other, lower frequencies in the frequency set.
- f3 may be set to 198 MHz, but higher or lower frequencies may alternatively be used.
- method 400 includes determining a maximum unwrapping range for the scene to be imaged.
- the maximum unwrapping range may be empirically determined, estimated, predetermined, or arbitrary, and/or may be based on the determined first (e.g., highest) frequency.
- an unwrapping distance L may be set to 16 m, but higher or lower distances may alternatively be used.
- method 400 includes sweeping one or more other frequencies over the maximum unwrapping range. For example, if f3 is determined, a desired range for f1 and f2 may be determined, and ToF measurements taken at an interval resolution for each lower frequency while maintaining the higher frequency constant. As an example, f1 and f2 may be swept over 50-189 MHz for both frequencies, although greater or smaller ranges may be used. The ranges for f1 and f2 need not be identical or even overlapping.
- the frequency interval resolution may be any suitable value, for example, 1 MHz. In some examples, a lowest of frequencies may be predetermined, and only the middle frequency may be swept.
- the upper range of usable frequencies may be limited by the product of frequency (f) and the ToF pixel modulation efficiency ME(f), e.g., f ⁇ ME(f).
- the range of frequencies swept may include frequencies larger than this upper bound. This may enable later selection of the best frequency settings if characteristics of the device, system, and/or environment are altered in the future.
- method 400 includes adjusting the noise level for each captured image to within a predetermined phase noise standard deviation.
- a noisy perturbation such as Gaussian noise applied to Eq.3
- the noise level may be equalized.
- the noise level in Eq. 3 representing the capture may be adjusted to bring the phase noise within a standard deviation of 9.5 degrees, though any suitable value may be used. This may yield the noisy simulation result expressed in Eq.5b and 5c at the full unwrapping range L.
- method 400 includes determining unwrapped phases for each of the sets of frequencies of modulation light. For example, unwrapping may be performed in the complex domain based on Eq. 8e and Eq 9 . Each result may then be compared to that without accounting for the noise perturbation. Each frequency set may then be assigned a value, for example, using Eq. 9. For example, the value may indicate whether there are more comparative results within 360 degrees of each other (+/ ⁇ noise), or greater than 360 degrees (+/ ⁇ noise).
- method 400 includes indicating a frequency set having a highest unwrapping passing rate.
- FIG. 5 A is a map 500 of unwrapping passing rates at each phase noise standard deviation for a 9.5 degree phase noise condition, scaled from 50-100% as indicated on the right side the map. Visually, this scaling resolution may be too low to determine a desired frequency configuration.
- FIG. 5 B shows a map 520 indicating a portion of the unwrapping passing rate map 500 that is scaled from 99-100% for convenience of frequency selection. All the visible points on the map have a higher unwrapping probability or passing rate as described in Eq. 9.
- optimization of the unwrapping frequencies may by performed by sweeping all the frequencies, iteratively starting from lower numbers of frequencies to higher numbers of frequencies, at given noise conditions with the same procedure mentioned above, though it may not be feasible to visualize the results on a 2D map.
- FIG. 6 A and FIGS. 6 B- 6 C show two examples that demonstrate unwrapping results for 3-frequency and 5-frequency systems without using rational integer numbers among the frequencies.
- the unwrapping passing rate/probability is about 99.46% at a 1 standard deviation phase noise condition (approx. 9.3 degrees).
- FIG. 6 B shows a plot 610 indicating the example phase unwrapping result of plot 600 , but at a zero-noise (ideal) condition.
- the unwrapping passing rate/probability is about 99.66% at a 1 standard deviation phase noise condition (approx. 15 degrees). Increasing the number of frequencies number makes the unwrapping distanced larger and more robust.
- FIG. 6 D shows a plot 630 indicating the example phase unwrapping result of plot 620 , but at a zero noise condition.
- FIG. 6 E shows a plot 640 indicating a close-up portion of plot 620 showing values for distances around 10 m.
- FIG. 6 F shows a plot 650 indicating a close-up portion of plot 630 showing values for distances around 10 m at a zero-noise condition.
- the methods and processes described herein may be tied to a computing system of one or more computing devices.
- such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
- API application-programming interface
- FIG. 7 schematically shows a non-limiting embodiment of a computing system 700 that can enact one or more of the methods and processes described above.
- Computing system 700 is shown in simplified form.
- Computing system 700 may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), and/or other computing devices.
- Computing system 700 may be an example of controller 208 .
- Computing system 700 includes a logic machine 710 and a storage machine 720 .
- Computing system 700 may optionally include a display subsystem 730 , input subsystem 740 , communication subsystem 750 , and/or other components not shown in FIG. 7 .
- Logic machine 710 includes one or more physical devices configured to execute instructions.
- the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs.
- Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
- the logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic machine optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic machine may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.
- Storage machine 720 includes one or more physical devices configured to hold instructions executable by the logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage machine 720 may be transformed—e.g., to hold different data.
- Storage machine 720 may include removable and/or built-in devices.
- Storage machine 720 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others.
- Storage machine 720 may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.
- storage machine 720 includes one or more physical devices.
- aspects of the instructions described herein alternatively may be propagated by a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a finite duration.
- a communication medium e.g., an electromagnetic signal, an optical signal, etc.
- logic machine 710 and storage machine 720 may be integrated together into one or more hardware-logic components.
- Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
- FPGAs field-programmable gate arrays
- PASIC/ASICs program- and application-specific integrated circuits
- PSSP/ASSPs program- and application-specific standard products
- SOC system-on-a-chip
- CPLDs complex programmable logic devices
- module may be used to describe an aspect of computing system 700 implemented to perform a particular function.
- a module, program, or engine may be instantiated via logic machine 710 executing instructions held by storage machine 720 . It will be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc.
- module may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.
- display subsystem 730 may be used to present a visual representation of data held by storage machine 720 .
- This visual representation may take the form of a graphical user interface (GUI).
- GUI graphical user interface
- Display subsystem 730 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic machine 710 and/or storage machine 720 in a shared enclosure, or such display devices may be peripheral display devices.
- input subsystem 740 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller.
- the input subsystem may comprise or interface with selected natural user input (NUI) componentry.
- NUI natural user input
- Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board.
- NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity.
- communication subsystem 750 may be configured to communicatively couple computing system 700 with one or more other computing devices.
- Communication subsystem 750 may include wired and/or wireless communication devices compatible with one or more different communication protocols.
- the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network.
- the communication subsystem may allow computing system 700 to send and/or receive messages to and/or from other devices via a network such as the Internet.
- a method for a time-of-flight camera comprises illuminating a scene with modulation light having two or more frequencies that do not have a common integer denominator; at a sensor array, receiving the modulation light reflected off objects within the scene; and determining unwrapped phases for each of the two or more frequencies of modulation light.
- determining unwrapped phases for each of the two or more frequencies of modulation light additionally or alternatively includes: processing the received modulation light to determine wrapped phase vectors for each frequency of modulation light; converting the wrapped phase vectors to phase scalars in a complex domain; and determining unwrapped phases in the complex domain based on the phase scalars.
- the phase scalars are additionally or alternatively expressed with a single unknown integer M, wherein M is a wrapped period number of a 2 ⁇ module for a given frequency of modulation light.
- the method additionally or alternatively comprises determining a value for M max for the wrapped phase vectors within a determined maximum unwrapping distance; and determining a value for M based on a value that minimizes the value of U M , such that 0 ⁇ M ⁇ M max , wherein
- N k is a total phase shifting steps at each frequency
- AB m is an active brightness at each frequency f m
- t d,m is a wrapped time-of-flight at each frequency f m .
- unwrapped phases for a frequency of modulation light are additionally or alternatively determined based on at least M.
- Gaussian noise is additionally or alternatively filtered from the phase vectors using a numeric filter in the complex domain.
- a system robustness is additionally or alternatively weighted based on an active brightness of the received modulation light.
- a time-of-flight camera comprises a modulated light emitter configured to illuminate a scene with modulation light having two or more frequencies that do not have a common integer denominator; a sensor array configured to receive the modulation light reflected off objects within the scene; a controller configured to: process the received modulation light to determine wrapped phase vectors for each frequency of modulation light; and determine unwrapped phases for each of the two or more frequencies of modulation light.
- the controller is additionally or alternatively configured to: convert the wrapped phase vectors to phase scalars in a complex domain; and determine unwrapped phases in the complex domain based on the phase scalars.
- the phase scalars are additionally or alternatively expressed with a single unknown integer M, wherein M is a wrapped period number of a 2 ⁇ module for a given frequency of modulation light.
- the controller is additionally or alternatively configured to determine a value for M max for the wrapped phase vectors within a determined maximum unwrapping distance; and determine a value for M based on a value that minimizes the value of such that 0 ⁇ M ⁇ M max , wherein
- N k is a total phase shifting steps at each frequency
- AB m is an active brightness at each frequency f m
- t d,m is a wrapped time-of-flight at each frequency f m .
- unwrapped phases for a frequency of modulation light are additionally or alternatively determined based on at least the value of M that generates an extremum for the value of U M .
- the controller is additionally or alternatively configured to: filter Gaussian noise from the phase vectors using a numeric filter in the complex domain.
- the controller is additionally or alternatively configured to: weight a system robustness based on an active brightness of the received modulation light.
- a method for a time-of-flight camera comprises illuminating a scene with three or more frequencies of modulation light; at a sensor array, receiving the modulation light reflected off objects within the scene; processing the received modulation light to determine wrapped phase vectors for each frequency of modulation light; converting the phase vectors to phase scalars in a complex domain; and determining unwrapped phase in the complex domain based on the phase scalars.
- the phase scalars are additionally or alternatively expressed with a single unknown integer M, wherein M is a wrapped period number of a 2 ⁇ module for a given frequency of modulation light.
- the method additionally or alternatively comprises determining a value for M based on minimization of U M , such that 0 ⁇ M ⁇ M max , where
- the method additionally or alternatively comprises during a first condition, illuminating the scene with a first set of three or more frequencies of modulation light; and during a second condition, illuminating the scene with a second set of three or more frequencies of modulation light, wherein at least one frequency of the second set is different than any frequency in the first set.
- the three of more frequencies of modulation light are additionally or alternatively selected based on an optimization process comprising: determining a first emitted frequency of modulation light; determining a maximum unwrapping range for the scene; sweeping one or more additional frequencies of the three or more frequencies of modulation light over the maximum unwrapping range; adjusting a noise level for received modulation light to within a predetermined phase noise standard deviation; determining unwrapped phases for each set of frequencies of modulation light; and indicating a frequency set having a highest unwrapping passing rate.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Electromagnetism (AREA)
- Computer Networks & Wireless Communication (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Optical Radar Systems And Details Thereof (AREA)
Abstract
Description
S j =∥S j ∥e −iφ (Eq.1).
(φ/4π)+(N/2)=(Z j f/c), (Eq. 2),
where c is the velocity of light, f is the modulation frequency, and N is a non-negative integer.
V m,k =CM m +AB m·cos(φd,m+ψk) for k=1,2, . . . N k and m=1, 2, . . . N m (Eq. 3).
may be dropped off, but is kept here for completion. For the condition where fm=fm0, the
one term having the frequency fm0 is redundant and can be eliminated, yielding:
Thus Eq. 8b is now equivalent to:
U M=Σm≠m0 m-1 [AB M·(ΔN m)2]=0 (Eq.8e).
such that either Eq.8e or Eq.8f can use the instantly measured ABm or previously measured and averaged ratio of
which is may be a constant for each sensor element regardless of the signal strength in the ideal case scenario.
and may be rounded to an
and wherein Nk is a total phase shifting steps at each frequency, ABm is an active brightness at each frequency fm and td,m is a wrapped time-of-flight at each frequency fm. In any of the preceding examples, or any other example, unwrapped phases for a frequency of modulation light are additionally or alternatively determined based on at least M. In any of the preceding examples, or any other example, Gaussian noise is additionally or alternatively filtered from the phase vectors using a numeric filter in the complex domain. In any of the preceding examples, or any other example, a system robustness is additionally or alternatively weighted based on an active brightness of the received modulation light.
and wherein Nk is a total phase shifting steps at each frequency, ABm is an active brightness at each frequency fm and td,m is a wrapped time-of-flight at each frequency fm. In any of the preceding examples, or any other example, unwrapped phases for a frequency of modulation light are additionally or alternatively determined based on at least the value of M that generates an extremum for the value of UM. In any of the preceding examples, or any other example, the controller is additionally or alternatively configured to: filter Gaussian noise from the phase vectors using a numeric filter in the complex domain. In any of the preceding examples, or any other example, the controller is additionally or alternatively configured to: weight a system robustness based on an active brightness of the received modulation light.
In any of the preceding examples, or any other example, the method additionally or alternatively comprises determining a value for M based on minimization of UM, such that 0≤M≤Mmax, where
In any of the preceding examples, or any other example, the method additionally or alternatively comprises during a first condition, illuminating the scene with a first set of three or more frequencies of modulation light; and during a second condition, illuminating the scene with a second set of three or more frequencies of modulation light, wherein at least one frequency of the second set is different than any frequency in the first set. In any of the preceding examples, or any other example, the three of more frequencies of modulation light are additionally or alternatively selected based on an optimization process comprising: determining a first emitted frequency of modulation light; determining a maximum unwrapping range for the scene; sweeping one or more additional frequencies of the three or more frequencies of modulation light over the maximum unwrapping range; adjusting a noise level for received modulation light to within a predetermined phase noise standard deviation; determining unwrapped phases for each set of frequencies of modulation light; and indicating a frequency set having a highest unwrapping passing rate.
Claims (18)
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/528,472 US11885880B2 (en) | 2019-07-31 | 2019-07-31 | Unwrapped phases for time-of-flight modulation light |
EP20747531.0A EP4004590A1 (en) | 2019-07-31 | 2020-06-05 | Unwrapped phases for time-of-flight modulation light |
CN202080055900.XA CN114207474A (en) | 2019-07-31 | 2020-06-05 | Unwrapped phase of time-of-flight modulated light |
PCT/US2020/036225 WO2021021286A1 (en) | 2019-07-31 | 2020-06-05 | Unwrapped phases for time-of-flight modulation light |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/528,472 US11885880B2 (en) | 2019-07-31 | 2019-07-31 | Unwrapped phases for time-of-flight modulation light |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210033731A1 US20210033731A1 (en) | 2021-02-04 |
US11885880B2 true US11885880B2 (en) | 2024-01-30 |
Family
ID=71846476
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/528,472 Active 2042-07-02 US11885880B2 (en) | 2019-07-31 | 2019-07-31 | Unwrapped phases for time-of-flight modulation light |
Country Status (4)
Country | Link |
---|---|
US (1) | US11885880B2 (en) |
EP (1) | EP4004590A1 (en) |
CN (1) | CN114207474A (en) |
WO (1) | WO2021021286A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20230010289A (en) * | 2021-07-12 | 2023-01-19 | 에스케이하이닉스 주식회사 | Image acquisition method for tof camera |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8587771B2 (en) | 2010-07-16 | 2013-11-19 | Microsoft Corporation | Method and system for multi-phase dynamic calibration of three-dimensional (3D) sensors in a time-of-flight system |
US20140049767A1 (en) * | 2012-08-15 | 2014-02-20 | Microsoft Corporation | Methods and systems for geometric phase unwrapping in time of flight systems |
US20160116594A1 (en) | 2014-10-27 | 2016-04-28 | Microsoft Corporation | Time of flight camera |
US20180011195A1 (en) | 2016-07-07 | 2018-01-11 | Microsoft Technology Licensing, Llc | Multi-frequency unwrapping |
US20220078342A1 (en) * | 2018-12-21 | 2022-03-10 | Chronoptics Limited | Time of flight camera data processing system |
-
2019
- 2019-07-31 US US16/528,472 patent/US11885880B2/en active Active
-
2020
- 2020-06-05 CN CN202080055900.XA patent/CN114207474A/en active Pending
- 2020-06-05 EP EP20747531.0A patent/EP4004590A1/en active Pending
- 2020-06-05 WO PCT/US2020/036225 patent/WO2021021286A1/en unknown
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8587771B2 (en) | 2010-07-16 | 2013-11-19 | Microsoft Corporation | Method and system for multi-phase dynamic calibration of three-dimensional (3D) sensors in a time-of-flight system |
US20140049767A1 (en) * | 2012-08-15 | 2014-02-20 | Microsoft Corporation | Methods and systems for geometric phase unwrapping in time of flight systems |
US20160116594A1 (en) | 2014-10-27 | 2016-04-28 | Microsoft Corporation | Time of flight camera |
US9702976B2 (en) | 2014-10-27 | 2017-07-11 | Microsoft Technology Licensing, Llc | Time of flight camera |
US20180011195A1 (en) | 2016-07-07 | 2018-01-11 | Microsoft Technology Licensing, Llc | Multi-frequency unwrapping |
US20220078342A1 (en) * | 2018-12-21 | 2022-03-10 | Chronoptics Limited | Time of flight camera data processing system |
Non-Patent Citations (5)
Title |
---|
"International Search Report and Written Opinion Issued in PCT Application No. PCT/US20/036225", dated Sep. 10, 2020, 11 Pages. |
"Office Action Issued in European Patent Application No. 20747531.0", dated Oct. 19, 2023, 7 Pages. |
Kadambi, Achuta, Jamie Schiel, and Ramesh Raskar. "Macroscopic interferometry: Rethinking depth estimation with frequency-domain time-of-flight." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. (Year: 2016). * |
Loffeld, Otmar, et al. "Phase unwrapping for SAR interferometry—A data fusion approach by Kalman filtering." IEEE Transactions on Geoscience and Remote Sensing 46.1 (2007): 47-58. (Year: 2007). * |
Wang, Yimin, et al. "Two-dimensional phase unwrapping in Doppler Fourier domain optical coherence tomography." Optics express 24.23 (2016): 26129-26145. (Year: 2016). * |
Also Published As
Publication number | Publication date |
---|---|
US20210033731A1 (en) | 2021-02-04 |
EP4004590A1 (en) | 2022-06-01 |
CN114207474A (en) | 2022-03-18 |
WO2021021286A1 (en) | 2021-02-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Zennaro et al. | Performance evaluation of the 1st and 2nd generation Kinect for multimedia applications | |
US11619723B2 (en) | Multiple-mode frequency sharing for time-of-flight camera | |
JP5739174B2 (en) | Distance information extraction method and optical apparatus employing the method | |
US9729860B2 (en) | Indirect reflection suppression in depth imaging | |
KR101722641B1 (en) | 3D image acquisition apparatus and method of extractig depth information in the 3D image acquisition apparatus | |
US10178381B2 (en) | Depth-spatial frequency-response assessment | |
KR101955334B1 (en) | 3D image acquisition apparatus and method of extractig depth information in the 3D image acquisition apparatus | |
KR102194233B1 (en) | Apparatus and method for generating a depth image | |
CN110135232A (en) | System and method for enhancing depth transducer device | |
US10055881B2 (en) | Video imaging to assess specularity | |
US20160316193A1 (en) | Parametric online calibration and compensation in tof imaging | |
US10996335B2 (en) | Phase wrapping determination for time-of-flight camera | |
US11448503B2 (en) | Correction for pixel-to-pixel signal diffusion | |
US11941787B2 (en) | Denoising depth data of low-signal pixels | |
US11734801B2 (en) | Distributed depth data processing | |
CN110139004A (en) | System and method for enhancing optical sensor arrangement | |
US11885880B2 (en) | Unwrapped phases for time-of-flight modulation light | |
US20210341609A1 (en) | High resolution time-of-flight measurements | |
US11651503B2 (en) | Determining depth in a depth image | |
Schönlieb et al. | Stray-light mitigation for under-display time-of-flight imagers | |
US20240070886A1 (en) | Mixed-mode depth imaging | |
WO2022169603A1 (en) | DETERMINING PHASE ORDERS IN ToF IMAGING |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:XU, ZHANPING;REEL/FRAME:049923/0651 Effective date: 20190731 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP., ISSUE FEE NOT PAID |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |