Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/88—Lidar systems specially adapted for specific applications
  • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
  • G01S17/894—Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
  • G01S17/06—Systems determining position data of a target
  • G01S17/08—Systems determining position data of a target for measuring distance only
  • G01S17/32—Systems determining position data of a target for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated
  • G01S17/36—Systems determining position data of a target for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated with phase comparison between the received signal and the contemporaneously transmitted signal
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
  • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
  • G01S7/4802—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
  • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
  • G01S7/483—Details of pulse systems
  • G01S7/486—Receivers
  • G01S7/487—Extracting wanted echo signals, e.g. pulse detection
  • G01S7/4876—Extracting wanted echo signals, e.g. pulse detection by removing unwanted signals
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/11—Region-based segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

• the present disclosure generally pertains to an information processing device for a time-of-flight device, an information processing method for a time-of-flight device, a computer program and a computer-readable medium.
• ToF devices are known, which are used for determining a distance to objects in a scene or a depth map of (the objects in) the scene that is illuminated with light.
• ToF devices include an illumination device (e.g. an array of light emitting diodes (LED)), optical parts (e.g. lenses), a ToF sensor such as an image sensor (e.g. array of current assisted photonic demodulator (CAPD) pixels) with read-out circuitry and a control.
• an illumination device e.g. an array of light emitting diodes (LED)
• optical parts e.g. lenses
• a ToF sensor such as an image sensor (e.g. array of current assisted photonic demodulator (CAPD) pixels) with read-out circuitry and a control.
• image sensor e.g. array of current assisted photonic demodulator (CAPD) pixels
• the ToF device For capturing a depth image, the ToF device typically illuminates the scene with, for instance, a modulated light wave and images the backscattered/reflected light wave on the ToF sensor.
• a gain of the pixels of the ToF sensor for example, is modulated according to a demodulation signal which may be phase-shifted with respect to the modulation of the emitted light wave for generating ToF data (time-of-flight data) indicative for the distance to the objects in the scene.
• transparent or translucent objects may be difficult to detect and that it may be difficult to obtain a silhouette of such objects, in particular in front of a background (e.g., a white wall).
• a background e.g., a white wall.
• the phase and amplitude images are utilized to attempt to separate the foreground object from the background.
• transparent or translucent objects may be difficult to detect from phase images due to potential noise and low signal returning from the object, while amplitude images may suffer in object detection when there is limited change in a returned signal level.
• the disclosure provides an information processing device for a time- of-flight device, the time-of-flight device being configured to acquire time-of-flight data, comprising circuitry configured to: obtain first time-of-flight data representing a background; obtain, based on time-of-flight data acquired by the time-of-flight device, second time-of- flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.
• the disclosure provides an information processing method for a time-of-flight device, the time-of-flight device being configured to acquire time-of-flight data, comprising: obtaining first time-of-flight data representing a background; obtaining, based on time-of-flight data acquired by the time-of-flight device, second time- of-flight data in which an object in front of the background is represented; and separating data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.
• the disclosure provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the information processing method of according to the second aspect.
• the disclosure provides a computer-readable medium having stored thereon the computer program according to the third aspect.
• Fig. 1 schematically illustrates in a block diagram an embodiment of a time-of-flight device and an information processing device
• Fig. 2 schematically illustrates an embodiment of a light modulation signal, a received light signal and four demodulation signals
• Fig. 3 schematically illustrates an embodiment of a light modulation signal, a received light signal and integrated voltages of a two-tapped pixel
• Fig. 4 schematically illustrates an embodiment of data points in a quadrature-component-in- phase-component space
• Fig. 5 schematically illustrates an embodiment of a threshold in a quadrature-component-inphase-component space
• Fig. 6 schematically illustrates an embodiment of a threshold in a quadrature-component-in- phase-component space
• Fig. 7 schematically illustrates in a block diagram an embodiment of a time-of-flight device and an information processing device
• Fig. 8 schematically illustrates in a block diagram an embodiment of an object silhouette detection system
• Fig. 9 schematically illustrates an embodiment of a threshold in a quadrature-component-inphase-component space.
• Fig. 10 schematically illustrates in a flow diagram an embodiment of an information processing method.
• time-of-flight (ToF) devices which typically illuminate the scene with, for instance, a modulated light wave and images the backscattered/reflected light wave on a ToF sensor such as an image sensor (e.g. array of current assisted photonic demodulator (CAPD) pixels) with read-out circuitry.
• a gain of the pixels of the ToF sensor is modulated according to a demodulation signal which may be phase-shifted with respect to the modulation of the emitted light wave for generating ToF data (time-of-flight data) indicative for the distance to the objects in the scene.
• transparent or translucent objects may be difficult to detect and that it may be difficult to obtain a silhouette of such objects, in particular in front of a background (e.g., a white wall).
• a background e.g., a white wall.
• the phase and amplitude images are utilized to attempt to separate the foreground object from the background.
• transparent or translucent objects may be difficult to detect from phase images due to potential noise and low signal returning from the object, while amplitude images may suffer in object detection when there is limited change in a returned signal level.
• ToF device 1 For enhancing the general understanding of the present disclosure an embodiment of a ToF device 1 is discussed in the following under reference of Fig. 1, Fig. 2, Fig. 3, and Fig. 4, which also applies to other embodiments of the present disclosure.
• Fig. 1 schematically illustrates in a block diagram an embodiment of the ToF device 1 and an information processing device 8-1 or 8-2, which is discussed in the following.
• the ToF device 1 is an indirect ToF system (iToF system).
• iToF system indirect ToF system
• the ToF device 1 includes an illumination device 2, a ToF sensor 3, a lens portion 4, a control 5, a communication interface 6 and a data bus interface 7.
• the information processing device 8-1 is a computer with an application processor.
• the information processing device 8-2 includes the ToF device 1, for example, the information processing device 8-2 may be a smartphone or camera system with an application processor that processes ToF data obtained from the ToF device 1.
• the information processing device 8-1 can communicate and exchange data with the ToF device 1 via the data bus interface 7.
• the illumination device 2 includes a light source (e.g., an LED array or laser diode array).
• a light source e.g., an LED array or laser diode array.
• the ToF sensor 3 includes a plurality of pixels (not shown), which are two-tapped CAPD pixels.
• the illumination device 2 illuminates a scene with light modulated in accordance with a light modulation signal (e.g., LMS in Fig. 2 and Fig. 3) received from the control 5.
• a light modulation signal e.g., LMS in Fig. 2 and Fig. 3
• the scene includes a background 9 (e.g., a white wall) and a translucent object 10 (e.g., a plastic bottle) which has an opaque label 11.
• a background 9 e.g., a white wall
• a translucent object 10 e.g., a plastic bottle
• the background 9 and the label 11 reflect at least part of the light towards the ToF device 1 which the lens portion 4 gathers and images onto the ToF sensor 3 (received light signal (e.g., RLS in Fig. 2 and Fig 3)). Moreover, the rest of the translucent object 10 (except of the label 11) reflects and scatters part of the illumination light, however, it lets some illumination light pass through such that it is reflected at the background 9 which then passes again through the translucent object 10 until reaching the ToF sensor 3.
• the rest of the translucent object 10 except of the label 11 reflects and scatters part of the illumination light, however, it lets some illumination light pass through such that it is reflected at the background 9 which then passes again through the translucent object 10 until reaching the ToF sensor 3.
• iToF devices perform four correlation measurements - in contrast to intensity-based image sensors which typically measure variations in signal strength - where the modulation light signal is used to drive both the ToF sensor 3 (as a demodulation signal) and the light source, and the correlation between the received light signal and the ToF sensor response driven by the demodulation signal is output.
• the phase offset between the light modulation signal applied to the light source and the demodulation signal applied to the ToF sensor 3 is shifted between each correlation measurement (0°, 90°, 180°, 270°), where the correlation measurements with 0° and 180° represent the “in-phase-component” (I), and the correlation measurements with 90° and 270° represent the “quadrature-component” (Q).
• I- and Q-component are then used to compute the phase and amplitude of each pixel in the scene.
• control 5 basically controls the overall operation of the ToF device 1 and controls the ToF device 1 such that four correlation measurements are repeatedly performed for obtaining depth information about the scene.
• control 5 applies a demodulation signal to the ToF sensor 3, which has a predetermined phase-shift with respect to the light modulation signal applied to the light source for generating the ToF data (time-of-flight data).
• Fig. 2 schematically illustrates an embodiment of a light modulation signal (LMS), a received light signal (RLS) and four demodulation signals (DM 1-4), which is discussed in the following.
• LMS light modulation signal
• RLS received light signal
• DM 1-4 demodulation signals
• the light modulation signal LMS applied to the illumination device 2 of Fig. 1 is a rectangular modulation signal with a modulation period T.
• An intensity of emitted infrared light of the light source is then modulated in time according to the light modulation signal LMS.
• the emitted infrared light is at least partially reflected at the background 9 and the translucent object 10.
• the received light signal RLS is basically an intensity of the reflected illumination light at the ToF sensor 3 of Fig. 1, which is phase-shifted with respect to the light modulation signal LMS and varies according to the intensity-modulation of the emitted infrared light.
• the phase is proportional to distances to objects in the scene.
• the ToF sensor 3 performs four correlation measurements corresponding to the demodulation signals DM1, DM2, DM3 and DM4.
• the demodulation signal DM1 is phase-shifted by 0° with respect to the light modulation signal LMS.
• the ToF sensor 3 each of the plurality of two-tapped CP AD pixels
• the demodulation signal DM2 is phase-shifted by 90° with respect to the light modulation signal LMS.
• the ToF sensor 3 each of the plurality of two-tapped CP AD pixels
• the demodulation signal DM3 is phase-shifted by 180° with respect to the light modulation signal LMS.
• the ToF sensor 3 each of the plurality of two-tapped CP AD pixels
• the demodulation signal DM4 is phase-shifted by 270° with respect to the light modulation signal LMS.
• the ToF sensor 3 each of the plurality of two-tapped CP AD pixels
• the electrical charges ql, q2, q3 and q4, as generally known, are proportional to or representative for, e.g., a voltage signal (electric signal) of the respective pixel output by the ToF sensor 3 and, thus, representative for ToF data (time-of-flight data).
• the voltage signal represented by the electrical charge ql corresponds to an in-phase-component value and the voltage signal represented by the electrical charge q3 corresponds to an in-phasecomponent value.
• the voltage signal represented by the electrical charge q2 corresponds to a quadraturecomponent value and the voltage signal represented by the electrical charge q4 corresponds to a quadrature-component value.
• phase between the light modulation signal LMS and the received light signal RLS is given by:
• Q is the quadrature-component (of a pixel) calculated from the two quadrature-component values q2 and q4
• I is the in-phase-component (of a pixel) calculated from the two in-phasecomponent values ql and q3.
• Q and I are also known as IQ value.
• the amplitude is given by:
• VQ 2 + I 2 amplitude - - - .
• the amplitude typically represents a signal strength and a confidence level of the corresponding data point.
• the ToF data acquired by the ToF device 1 include the two in- phase-component values ql and q3 and the two quadrature-component values q2 and q4.
• the ToF data acquired by the ToF device 1 further include the phase and the amplitude for each pixel.
• the ToF data acquired by the ToF device 1 include only the phase and amplitude for each pixel.
• the ToF data acquired by the ToF device 1 include the quadrature-component (e.g., q4-q2) and/or the inphase-component (e.g., ql-q3) for each pixel of the ToF sensor 3.
• Fig. 3 schematically illustrates an embodiment of a light modulation signal (LMS), a received light signal (RLS) and integrated voltages (IV) of a two-tapped pixel, which is discussed in the following.
• LMS light modulation signal
• RLS received light signal
• IV integrated voltages
• the ToF sensor 3 includes a plurality of CAPD pixels and read-out circuitry, wherein each CAPD pixel has two taps: Tap A and Tap B.
• the upper graph schematically illustrates the light modulation signal (LMS) with which the light source of the illumination device 2 of the ToF device 1 is modulated such that the light source emits light to the scene (illuminates the scene) modulated in time according to the LMS.
• LMS light modulation signal
• the upper graph schematically illustrates only two modulation periods T1 and T2 for the sake of illustration and simplicity only.
• the background 9 and the translucent object 10 in front of the background 9 reflect (and scatter) at least part of the modulated illumination light back to the ToF device 1 which receives it and the lens portion 4 of the ToF device 1 images the received light signal (RLS) onto the ToF sensor 3.
• the middle graph schematically illustrates the RLS which includes a modulation light (ML) part and an ambient light (AL) part.
• the read-out circuitry applies a demodulation signal to Tap A and to Tap B, wherein the demodulation signal applied to Tap A is phase-shifted by 180° with respect to the demodulation signal applied to Tap B.
• a phase-shift between the LMS and the demodulation signal applied to Tap A is 0°.
• the lower graph schematically illustrates the integrated voltages (IV) at Tap A and Tap B (or at a capacitance connected with Tap A or Tap B respectively) during the modulation periods.
• the voltage of Tap A (dashed line) changes during sub-period Tl-1 due to photoelectric conversion of light corresponding to a part Al of the RLS.
• the voltage of Tap B (dashed dotted line) changes during sub-period Tl-2 due to photoelectric conversion of light corresponding to a part Bl of the RLS.
• the voltage of Tap A further changes during sub-period T2-1 due to photoelectric conversion of light corresponding to a part A2 of the RLS.
• the voltage of Tap B further changes during sub-period T2- 2 due to photoelectric conversion of light corresponding to a part B2 of the RLS.
• a signal contribution (SC) acquired by the pixel is given by the difference between the IV of Tap A and Tap B such that typically the ambient light (AL) part is cancelled out.
• the SC corresponds to the voltage change caused by the modulation light (ML) part of the RLS which thus includes distance information.
• Fig. 4 schematically illustrates an embodiment of data points 12 in a quadrature-component-in- phase-component space, which is discussed in the following under reference of Fig. 1 and Fig. 4.
• the ToF device 1 acquires ToF data which include the two in-phase-component values ql and q3 and the two quadrature-component values q2 and q4.
• the information processing device 8-1 obtains the ToF data acquired by the ToF device 1.
• the information processing device 8-1 obtains, based on the ToF data acquired by the ToF device 1, second ToF data by calculating, based on the two in-phase-component values ql and q3 and the two quadrature-component values q2 and q4, a phase and an amplitude for each pixel of the ToF sensor 3 of the ToF device 1.
• the ToF data obtained from the ToF device 1 include the phase and amplitude for each pixel such that the second ToF data may correspond to the ToF data acquired by the ToF device 1.
• the second ToF data include the data points 12.
• the data points 12 are thus an IQ value for each pixel of the ToF sensor 3 and are plotted in the quadrature-component-in-phase-component space (which may be referred to as IQ-plot), which is spanned by a quadrature-component (Q) axis and an in-phase-component (I) axis.
• IQ-plot quadrature-component-in-phase-component space
• the angle of a data point in the quadrature-component-in-phase-component space corresponds to its phase and the distance to the origin of the quadrature-component-in-phase-component space corresponds to its amplitude.
• the data points 12 have a large standard deviation and are scattered over a region of the quadrature-component-in-phase-component space, since the phase and the amplitude are typically different for the background 9, the label 11 and the rest of the translucent object 10.
• the data points should be separated based on a known background for extracting a silhouette or a contour of the foreground object, in particular in cases in which a single pair of phase/amplitude image may include limited information about the foreground object (e.g., for transparent or translucent objects).
• some embodiments pertain to an information processing device for a time-of-flight device, wherein the time-of-flight device is configured to acquire time-of-flight data
• the information processing device includes circuitry configured to: obtain first time-of-flight data representing a background; obtain, based on time-of-flight data acquired by the time-of-flight device, second time-of- flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.
• the information processing device may be a computer, a server, a smartphone, a camera (system or module) or the like.
• the circuitry may be based on or may include or may be implemented by typical electronic components configured to achieve the functionality as described herein.
• the circuitry may be based on or may include or may be implemented in parts by typical electronic components and integrated circuitry logic and in parts by software.
• the circuitry may include a data bus interface for transmitting (and receiving) data over a data bus.
• the data bus interface may be a Camera Serial Interface (CSI) in accordance with MIPI (Mobile Industry Processor Interface) specifications (e.g., MIPII CSI-2 or the like), an I 2 C (Inter- Integrated Circuit) interface, a Controller Area Network (CAN) bus interface, an FDP-link (Flat Panel Display link), a GSML (Gigabit Multimedia Serial Link), etc.
• CSI Camera Serial Interface
• MIPII Mobile Industry Processor Interface
• I 2 C Inter- Integrated Circuit
• CAN Controller Area Network
• FDP-link Flat Panel Display link
• GSML Gigabit Multimedia Serial Link
• the circuitry may include a communication interface configured to communicate and exchange data with a computer or processor (e.g., an application processor) over a network (e.g. the internet) via a wired or a wireless connection such as a mobile telecommunications system which may be based on UMTS, LTE or the like (and implements corresponding communication protocols).
• a computer or processor e.g., an application processor
• a network e.g. the internet
• a wired or a wireless connection such as a mobile telecommunications system which may be based on UMTS, LTE or the like (and implements corresponding communication protocols).
• the circuitry may include data storage capabilities to store data such as memory which may be based on semiconductor storage technology (e.g., RAM, EPROM, etc.) or magnetic storage technology (e.g. a hard disk drive) or the like.
• semiconductor storage technology e.g., RAM, EPROM, etc.
• magnetic storage technology e.g. a hard disk drive
• the information processing device (the circuitry of the information processing device) obtains first time-of-flight data representing a background.
• the first time-of-flight data include at least one in-phase-component value and at least one quadrature-component value.
• the first time-of-flight data include two in-phase-component values (corresponding to a phase-shift of 0° and 180° between the light modulation signal and the demodulation signal) and two quadrature-component values (corresponding to a phase-shift of 90° and 270° between the light modulation signal and the demodulation signal) for each pixel of an image sensor of the time-of-flight device.
• the first time-of-flight data further include the phase and the amplitude for each pixel of the image sensor of the time-of-flight device.
• the first time-of- flight data include only the phase and amplitude for each pixel of the image sensor of the time- of-flight device.
• the first time-of-flight data include the quadraturecomponent (e.g., q4-q2) and/or the in-phase-component (e.g., ql-q3) for each pixel of the image sensor of the time-of-flight device.
• the first time-of-flight data is acquired by the time-of-flight device, loaded from a background database, or simulated based on predetermined distances between the time-of-flight device and the background.
• the background may be an opaque object (e.g., a white wall) or the like which at least partially reflects illumination light from the time-of-flight device.
• an opaque object e.g., a white wall
• the first time-of-flight data representing the background is based on a user input. For example, a distance between the image sensor of the time-of-flight device and the background is manually input representing the background (e.g., in the case of a flat and perpendicular background). Any type of user input may be considered (gradient distance, specific background shape distance information, etc.).
• the first time-of-flight data representing the background is generated by an algorithm that is configured to infer a background (background distances) from time-of-flight data in which a foreground object is represented.
• a background background distances
• the algorithm is configured to generate the first time-of-flight data representing the background by assuming that the wall stays flat behind the foreground object (e.g., a bottle that is in front of a flat wall, it is assumed that the wall stays flat behind the bottle).
• the information processing device obtains, based on time-of-flight data acquired by the time-of-flight device, second time-of-flight data in which an object in front of the background is represented.
• time-of-flight data acquired by the time-of-flight device based on which the second time-of-flight data is obtained, corresponds to a different measurement than the first time-of-flight data.
• the time-of-flight data acquired by the time-of-flight device include at least one in-phase-component value and at least one quadrature-component value.
• the time-of-flight data acquired by the time-of-flight device include two in-phasecomponent values (corresponding to a phase-shift of 0° and 180° between the light modulation signal and the demodulation signal) and two quadrature-component values (corresponding to a phase-shift of 90° and 270° between the light modulation signal and the demodulation signal) for each pixel of an image sensor of the time-of-flight device.
• the time-of- flight data acquired by the time-of-flight device further include the phase and the amplitude for each pixel of the image sensor of the time-of-flight device. In some embodiments, the time-of- flight data acquired by the time-of-flight device include only the phase and amplitude for each pixel of the image sensor of the time-of-flight device. In some embodiments, the time-of-flight data acquired by the time-of-flight device include the quadrature-component (e.g., q4-q2) and/or the in-phase-component (e.g., ql -q3) for each pixel of the image sensor of the time-of-flight device.
• quadrature-component e.g., q4-q2
• the in-phase-component e.g., ql -q3
• the second time-of-flight data correspond to the time-of-flight data acquired by the time-of-flight device.
• the second time-of-flight data include at least an amplitude for each pixel of the image sensor of the time-of-flight device.
• the second time-of-flight data include at least a phase and an amplitude for each pixel of the image sensor of the time-of-flight device.
• the information processing device obtains the second time-of-flight by calculating it from the time- of-flight data acquired by the time-of-flight device.
• the second time-of- flight data include the quadrature-component (e.g., q4-q2) and/or the in-phase-component (e.g., ql-q3) for each pixel of the image sensor of the time-of-flight device.
• quadrature-component e.g., q4-q2
• in-phase-component e.g., ql-q3
• the object is transparent for illumination light from the time-of-flight device or the object is translucent for the illumination light from the time-of-flight device.
• the information processing device (the circuitry of the information processing device) separates data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.
• the circuitry is further configured to determined and set the threshold based on the first time-of-flight data.
• the threshold is determined based on a distribution of data points in the first time-of-flight data in the quadrature-component-in-phase-component space.
• the distribution of the data points may be confined in a circular or elliptic shaped region or the like of the quadrature-component-in-phase-component space such that any data point falling in this region is considered to be a data point corresponding to the background.
• the foreground object time-of-flight data include each data point of the second time-of-flight data which does not fall within a region of the quadrature- component-in-phase-component space which is associated with the background determined by the threshold.
• the data points of first time-of-flight data corresponding to the same pixels as the foreground object time-of-flight data are subtracted from the foreground object time-of-flight data before applying a spatial filter.
• the first time-of-flight data is subtracted for each pixel from the second time-of-flight data before the threshold is determined.
• any data points in the second time-of-flight data which correspond to the background may be close to the origin of the quadrature-component-in-phase-component space after subtracting the data points of the first time-of-flight data such that the threshold is determined by a distance to the origin of the quadrature-component-in-phase-component space.
• the foreground object data include the data points in the second time-of- flight data which have a distance (amplitude) larger than the threshold to the origin of the quadrature-component-in-phase-component space, wherein the data points are either the differences between the corresponding data points of the first time-of-flight data and the second time-of-flight data or the original data points of the second time-of-flight data.
• the threshold is defined by the Euclidean distance in the quadrature- component-in-phase-component space (IQ space or IQ plot), wherein:
• d IQ is the Euclidean distance in the IQ plot
• Al and AQ are the distances between foreground and background objects in the I and Q axes, respectively.
• a threshold is determined in the IQ plot, and the Euclidean distance between the foreground data (second time-of-flight data) and the background data (first time-of-flight data) can be used to retrieve a silhouette of the foreground object.
• the threshold is further based on the second time-of-flight data.
• the circuitry is further configured to apply a spatial filter to the foreground object time-of-flight data for increasing a contrast of the object in front of the background.
• the spatial filter is a contour filter or a silhouette filter.
• the time-of-flight data acquired by the time-of-device include quadraturecomponent values and in-phase-component values for each pixel of an image sensor of the time- of-flight device.
• the threshold corresponds to a region in a quadrature-component-in- phase-component space, as discussed above.
• the difference between the first time-of-flight data and the second time-of-flight data may be dominant along a particular axis of the quadrature-component-in- phase-component space (hence, along either the in-phase-component axis or the quadraturecomponent axis).
• the Euclidean distance is dominant along one particular axis of the IQ plot (for example the Q axis). It has been recognized that, in some embodiments, any changes in amplitude due to, for example, a part of a translucent plastic bottle without label, is due to a reduction of signal strength and, thus, reduces the measured amplitude compared to the amplitude of the background data (first time-of-flight data) in the IQ plot (quadrature-component-in-phase-component space) along a particular axis (e.g., a reduction of the Q-component).
• any changes in phase due to, for example, the label of the translucent plastic bottle results in a change of the angle in the IQ plot, which may further result in a reduction of amplitude along the particular axis (e.g., a reduction of the Q- component).
• a threshold on the particular axis may be determined to retrieve a contour or a silhouette of the foreground object (to separate the data points in the second time-of-flight data to obtain foreground object time-of-flight data). This may allow to reduce the total number of correlation measurements required in some embodiments (only quadrature-component values or in-phase-component values are required).
• the threshold corresponds to a threshold on a quadraturecomponent axis or an in-phase-component axis in a quadrature-component-in-phase-component space.
• the threshold separates the quadrature-component-in-phase- component space in two parts.
• the number of correlation measurements can be reduced in some embodiments from four to two.
• the time-of-flight data acquired by the time-of-flight device include only quadrature-component values or in-phase-component values for each pixel of an image sensor of the time-of-flight device.
• the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature- component-in-phase-component space, as discussed above.
• the threshold separates the quadrature-component-in-phase-component space in two parts, as discussed above.
• the information processing device includes the time-of-flight device.
• the correlation measurements can be reduced from four to two when the first time-of-flight data is aligned along a particular axis of the quadrature-component-in-phase- component space, as also discussed above.
• the time-of-flight device and the background have a predetermined distance to each other such that the first time-of-flight data is aligned along a quadrature-component axis or an in-phase-component axis in a quadrature-component-in-phase- component space.
• the background and the object in front of the background have a distance to each other which corresponds to a quarter of the unambiguous range of the time-of-flight device.
• the first time-of-flight data is aligned along a particular axis of the IQ-space.
• any opaque parts of the foreground object are aligned along the perpendicular axis of the IQ-space.
• the time-of-flight device acquires only quadrature-component values or in-phase-component values for each pixel of an image sensor of the time-of-flight device depending on the alignment of the first time-of-flight data.
• the number of correlation measurements can be reduced from four to one in some embodiments.
• the time-of-flight data acquired by the time-of-flight device include only one quadrature-component value or one in-phase-component value for each pixel of an image sensor of the time-of-flight device, wherein each pixel is a two-tapped pixel.
• the threshold corresponds to a threshold on a quadrature-component axis or an inphase-component axis in a quadrature-component-in-phase-component space. In such embodiments, the threshold separates the quadrature-component-in-phase-component space in two parts.
• the two taps of a two-tapped CAPD pixel are driven with a phase-shift of 180° and, thus, even when only one correlation measurement with, for example, 90° phase-shift between the light modulation signal and the demodulation signal is performed, a single two-tapped pixel is still able to generate two quadrature-component values or two in-phase-component values, for example, the ones corresponding to a phase-shift of 90° and 270° between the light modulation signal and the demodulation signal.
• the integrated voltages of the two taps are not subtracted but individually output.
• a single correlation measurement can be used as the Q-component or as the I-component which may further reduce a motion blur.
• the Q-component or as the I-component which may further reduce a motion blur.
• the number of correlation measurements is reduced to one and, thus, a single capture method may be provided (further reducing motion blur).
• a single capture method may be provided (further reducing motion blur).
• it may not provide depth information (due to the reduced number of captures) but may be used for contour detection which may be useful, for example, in detecting the presence of translucent objects, or other applications where a foreground contour is required.
• Some embodiments pertain to an information processing method for a time-of-flight device, wherein the time-of-flight device is configured to acquire time-of-flight data
• the information processing method includes: obtaining first time-of-flight data representing a background; obtaining, based on time-of-flight data acquired by the time-of-flight device, second time- of-flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.
• the information processing method may be performed by the information processing device as described herein.
• the methods as described herein are also implemented in some embodiments as a computer program causing a computer and/or a processor to perform the method, when being carried out on the computer and/or processor.
• a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.
• At least one of the following may be achieved/provided in some embodiments:
• iToF cameras for transparent or translucent object detection, where the unique properties of an iToF pixel aid in the detection (compared to, for example, an RGB image sensor), wherein a threshold to separate foreground objects from the background is applied.
• a silhouette filter is in some embodiments applied to the pixels having a depth relative to the threshold corresponding to the foreground object.
• iToF devices perform two correlation measurements for both the I and Q axes (four captures total), because this cancels out effects of ambient light, as well as pixel related mismatches. In some embodiments, there is a subtraction of the background data, hence the ambient light and pixel mismatches may be cancelled during this subtraction.
• a silhouette/contour of a foreground object is extracted from its background with high robustness, especially in low contrast conditions.
• the high robustness is achieved by working in the IQ-space and applied to transparent or translucent objects.
• the number of correlation measurements is reduced to two (or one in some embodiments), while maintaining motion robustness for the reduced number of required captures.
• a (low) processing load is provided suitable for applications where realtime high framerate processing is required.
• phase and amplitude may be present in situations where regular iToF may be difficult, for example, where phases may not be accurately reconstructed.
• This may include applications such as translucent object detection, or similarly where the scene may be challenging, for example, when large variations in phase and amplitude are present (high contrast objects, objects which provide inconsistent phase values).
• the information processing device and/or the information processing method and/or the time-of- flight device described herein may be used, for example, in the following applications:
• Another application may include automation in the beverage industry, where full beverage containers (which may be difficult for iToF to accurately measure due to scattering) may need to be detected on a production line.
• full beverage containers which may be difficult for iToF to accurately measure due to scattering
• some foods may suffer from sub-surface scattering, yielding poor depth images from multiple reflections of the light within the food, it could be used to accurate detect the contour of the food and hence track it along a conveyor belt.
• motion robustness may also benefit, such as in surveillance where persons are moving, however, because of the limitation of distance between foreground and background, these use cases may be more limited.
• FIG. 5 there is schematically illustrated an embodiment of a threshold 21 in a quadrature-component-in-phase-component space, which is discussed in the following under reference of Fig. 1 and Fig. 5.
• IQ-plot The quadrature-component-in-phase-component space shown in Fig. 5, which is referred to as IQ-plot in the following, is spanned by a quadrate-component (Q) axis and an in-phasecomponent (I) axis.
• the ToF device 1 acquires first ToF data representing the background 9 (e.g., before the translucent object 10 is placed in front of the background 9), which is then transmitted to the information processing device 8-1, which obtains the first ToF data.
• the first ToF data include two quadrature-component values (90° and 270°) and two in-phase-component values (0° and 180°).
• the information processing device 8-1 calculates a phase and an amplitude for the first ToF data to obtain data points 20, as illustrated in the IQ-plot.
• the data points 20 represent the background 9 in the IQ-plot.
• the ToF device 1 acquires subsequent ToF data in which the translucent object 10 is represented, which are then transmitted to the information processing device 8-1, which obtains the subsequent ToF data.
• the subsequent ToF data include two quadrature-component values (90° and 270°) and two in-phase-component values (0° and 180°).
• the information processing device 8-1 calculates a phase and an amplitude for the subsequent ToF data to obtain second ToF data including data points 26, 22 and 23, as illustrated in the IQ- plot.
• the data points 26 correspond to a background part.
• the data points 22 correspond to other parts than the label 11 of the translucent object 10.
• the data points 23 correspond to the label 11 of the translucent object 10.
• the IQ-plot represents a situation where the data points 20 have a high amplitude value, with low standard deviation in phase.
• the other parts than the label 11 of the translucent object 10 result in a reduction in amplitude as well as a large standard deviation in phase due to the multipath reflections from the background 9 as well as within the translucent object 10.
• the label 11 of the translucent object 10 gives a reflection, resulting in higher amplitude with a change in phase depending on the distance between the ToF device 1 and the translucent object 10. This may make it difficult to reconstruct an accurate depth map.
• the information processing device 8-1 determines and sets the threshold 21 based on the data points 20, wherein the threshold 21 corresponds to a circular region in the IQ-plot which includes all data points 20.
• the information processing device 8-1 separates the data points 26, 22 and 23 in the second ToF data based on the threshold 21 for obtaining foreground object ToF data 25.
• the foreground object ToF data 25 include the data points 22 and 23 which do not fall within the circular region determined by the threshold 21, as illustrated by the solid elliptical lines around the data points 22 and around the data points 23.
• the foreground object ToF data 25 include data points 22 and 23 which do not fall within the circular region determined by the threshold 21 subtracted by data points of the data points 20 which correspond to the same pixel.
• the information processing device 8-1 applies a spatial filter such as contour or silhouette filter to the foreground object data for increasing a contrast of the translucent object 10 in front of the background 9.
• a spatial filter such as contour or silhouette filter
• the information processing device 8-1 detects cluster of data points 22 and 23 not present in the data points 20 as further illustrated by the solid elliptical lines in the IQ- plot. In such embodiments, the information processing device 8-1 determines the threshold 21 based on an Euclidian distance between the a center of the data points 20 and a center of the clusters of data points, for example, the threshold 21 may correspond to the minimum of both Euclidian distances.
• Fig. 6 schematically illustrates an embodiment of a threshold 24 in a quadrature-component-in- phase-component space, which is discussed in the following.
• the present embodiment is based on the embodiment of Fig. 5 such that unnecessary repetition of similar parts is omitted.
• the data points 20 have a high amplitude value along the quadraturecomponent axis (Q) compared to the data points 23 and the data points 22.
• the information processing device 8-1 detects such a situation, for example, from a plurality of subsequent measurements.
• the information processing device 8-1 may determine and set the threshold 24, based on the data points 20, such that the threshold 24 corresponds to a threshold on the quadrature-component axis.
• the threshold 24 separates the IQ-space in two parts along a line parallel to the in-phase-component axis.
• the information processing device 8-1 separates the data points 26, 22 and 23 in the second ToF data based on the threshold 24 for obtaining foreground object ToF data 25.
• the foreground object ToF data 25 include the data points 22 and 23 which do not fall within the region of the IQ-plot above the threshold 24, as illustrated by the solid elliptical lines around the data points 22 and around the data points 23.
• the information processing device 8-1 may instruct the ToF device 1 (e.g., transmits a command to the control 5) to reduce the number of correlation measurements from four to two to only acquire the quadrature-component values (e.g., q4 and q2) based on which the quadrature-component can be obtained.
• the ToF device 1 e.g., transmits a command to the control 5
• the quadrature-component values e.g., q4 and q2
• the ToF device 1 starts acquiring only the quadraturecomponent values (e.g., q4 and q2) and outputs them to the information processing device 8-1.
• the quadraturecomponent values e.g., q4 and q2
• the information processing device 8-1 obtains the second ToF data by calculating the quadrature-component from the ToF data acquired by the ToF device 1.
• the information processing device 8-1 is able to separate the data points in the second ToF data based on the threshold 24 to obtain the foreground object ToF data 25 even with the reduced number of correlation measurements.
• the information processing device can obtain a contour or a silhouette from the foreground object ToF data 25.
• the information processing device 8-1 may further reduce the number of correlation measurements from four to one, since the ToF sensor 3 includes two-tapped CAPD pixels which can output two quadrature-component values at once such that only one correlation measurement must be performed to obtain the quadrature-component.
• Fig. 7 schematically illustrates in a block diagram an embodiment of a ToF device 1 and an information processing device 8-1 (or 8-2), which is discussed in the following.
• the ToF device 1 corresponds to the ToF device 1, however, the ToF device 1 and the background 9 have a predetermined distance (d) to each other, for example, the distance (d) corresponds to the unambiguous range of the ToF device 1 such that the first ToF data is aligned, for example, along the quadrature-component axis (Q) in the IQ-plot (the quadrature-component- in-phase-component space).
• the unambiguous range of the ToF device 1 is related to the modulation frequency of the light modulation signal (LMS).
• the background 9 and the translucent object 10 are thus constrained to a fixed separation distance relative to the modulation frequency, which may allow the detection of objects which are not at the same depth as the background.
• the ToF data acquired by the ToF device 1 corresponding to the label 11 of the translucent object 10 is aligned along the perpendicular axis of the IQ-plot, here along the in- phase-component axis (I).
• the ToF device 1 is instructed to only perform the correlation measurements corresponding to the quadrature-component. This reduces the total number of correlation measurements required for obtaining a contour or silhouette of the translucent object, which may reduce motion blur.
• the background 9 is to be aligned approximately with one axis (e.g., the Q-axis).
• the foreground object e.g., the translucent object
• c the speed of light
• any received light with reduced amplitude may manifest as a reduction on the axis aligned with the background (e.g., a very low Q-value).
• the foreground object is aligned with the perpendicular axis, objects at this distance will have a value very close to zero on the background axis (e.g., a Q-value close to zero).
• both changes and phase and amplitude are detected, while only utilising the background axis values (i.e., only Q values are required to be analysed).
• a simple threshold can be used to detect any changes in the background axis values from a difference between the first ToF data and the second ToF data, yielding the contour of the foreground object.
• other edge detection algorithms could be used on this difference, for example, canny edge detection.
• Other approaches might use machine learning or Al (“artificial intelligence”) to learn the background, which may further improve the extraction of the foreground contour.
• any reduction of signal strength may manifest in a change in amplitude.
• any changes in phase may manifest in a change in angle in the IQ plot.
• FIG. 8 schematically illustrates in a block diagram an embodiment of an object contour detection system 30, which is discussed in the following under reference of Fig. 8 and Fig. 9.
• the object contour detection system 30 is a bottle refund system.
• the object contour detection system 30 includes a background 31, a conveyor belt 32 on which bottles are transported from left to right, a ToF device 34 and an information processing device 35.
• the ToF device 34 acquires first ToF data representing the background 31 in a period between two bottles 33 (e.g., when no bottle is in the field of view of the ToF device 34, since two bottles have a minimum distance to each other which can be used for acquiring the first ToF data).
• the ToF device 34 further acquires subsequent ToF data in which a bottle in front of the background 31 is represented.
• the ToF device 34 and the background 9 have a predetermined distance (d) to each other, for example, the distance (d) corresponds to the unambiguous range of the ToF device 34 such that the first ToF data is aligned, for example, along the quadrature-component axis (Q) in the IQ- plot (the quadrature-component-in-phase-component space).
• the ToF device 34 acquires only quadrature-component values and, thus, is only required to perform two correlation measurements instead of four for obtaining a contour of the bottles 33.
• Fig. 9 schematically illustrates an embodiment of a threshold 41 in a quadrature-component-in- phase-component space (which is referred to as IQ-plot in the following).
• the information processing device 35 obtains data points 40 for the first ToF data by calculating the quadrature-component and, thus, the data points 40 are aligned on the Q-axis.
• the information processing device 35 calculates the quadrature-component for the subsequent ToF data to obtain the second ToF data including data points 42 (corresponding to other parts than a label on the bottles 33) and 45 (corresponding to a part of the background 31).
• the data points 43 correspond to a label on the bottles 33 and are thus aligned on the I-axis due to the distance configuration of the object contour detection system 30.
• the information processing device 35 classifies each calculated data point as one of the data points 43 when the quadrature-component is zero or close to zero (e.g., by applying a predetermined threshold).
• the information processing device 35 determines and sets the threshold 41 based on the data points 40, wherein the threshold 41 corresponds to a threshold on the quadrature-component axis.
• the information processing device 35 obtains foreground object ToF data 44 including the data points 42 which are below the threshold 41 (and the data points 43), as illustrated by the solid elliptical line around the data points 42 and the data points 43. Then, the information processing device 35 can obtain the contour of the bottles 33, moreover, the information processing device 35 may further apply a spatial filter to the foreground object ToF data for increasing a contrast of the bottles 33.
• Fig. 10 schematically illustrates in a flow diagram an embodiment of an information processing method 100, which is discussed in the following.
• the information processing method 100 may be performed by the information processing device as described herein.
• first time-of-flight data representing a background are obtained, as discussed herein.
• second time-of-flight data is obtained in which an object in front of the background is represented, as discussed herein.
• data points in the second time-of-flight data are separated based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time- of-flight data, as discussed herein.
• An information processing device for a time-of-flight device wherein the time-of-flight device is configured to acquire time-of-flight data, including circuitry configured to: obtain first time-of-flight data representing a background; obtain, based on time-of-flight data acquired by the time-of-flight device, second time-of- flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.
• circuitry is further configured to apply a spatial filter to the foreground object time-of-flight data for increasing a contrast of the object in front of the background.
• circuitry is further configured to determine and set the threshold based on the first time-of-flight data.
• time-of-flight data acquired by the time-of-flight device include quadrature-component values and in-phasecomponent values for each pixel of an image sensor of the time-of-flight device.
• the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature- component-in-phase-component space.
• time-of-flight data acquired by the time-of-flight device include only quadrature-component values or inphase-component values for each pixel of an image sensor of the time-of-flight device.
• the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature- component-in-phase-component space.
• time-of-flight data acquired by the time-of-flight device include only one quadrature-component value or one in-phase-component value for each pixel of an image sensor of the time-of-flight device, wherein each pixel is a two-tapped pixel.
• An information processing method for a time-of-flight device wherein the time-of-flight device is configured to acquire time-of-flight data, including: obtaining first time-of-flight data representing a background; obtaining, based on time-of-flight data acquired by the time-of-flight device, second time- of-flight data in which an object in front of the background is represented; and separating data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.
• (20) A computer-readable medium having stored thereon the computer program of (19).
• (21) A computer program comprising program code causing a computer to perform the method according to (18), when being carried out on a computer.
• (22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to (18) to be performed.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Computer Networks & Wireless Communication (AREA)
• Radar, Positioning & Navigation (AREA)
• Remote Sensing (AREA)
• Electromagnetism (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Theoretical Computer Science (AREA)
• Optical Radar Systems And Details Thereof (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=80446746

Family Applications (1)

Country Status (3)

Family Cites Families (3)

• 2023
  • 2023-02-02 EP EP23702359.3A patent/EP4476560A1/de active Pending
  • 2023-02-02 WO PCT/EP2023/052522 patent/WO2023148252A1/en not_active Ceased
  • 2023-02-02 JP JP2024545817A patent/JP2025505558A/ja active Pending

Also Published As

Similar Documents

Legal Events