EP4476560A1

EP4476560A1 - Information processing device, information processing method, computer program and computer-readable medium

Info

Publication number: EP4476560A1
Application number: EP23702359.3A
Authority: EP
Inventors: Sean CHARLESTON
Original assignee: Sony Depthsensing Solutions NV SA; Sony Semiconductor Solutions Corp
Current assignee: Sony Depthsensing Solutions NV SA; Sony Semiconductor Solutions Corp
Priority date: 2022-02-07
Filing date: 2023-02-02
Publication date: 2024-12-18
Also published as: WO2023148252A1; JP2025505558A

Abstract

An information processing device for a time-of-flight device, wherein the time-of-flight device is configured to acquire time-of-flight data, wherein the information processing device includes circuitry configured to: obtain first time-of-flight data representing a background; obtain, based on time-of-flight data acquired by the time-of-flight device, second time-of- flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

Description

INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD, COMPUTER PROGRAM AND COMPUTER-READABLE MEDIUM

TECHNICAL FIELD

The present disclosure generally pertains to an information processing device for a time-of-flight device, an information processing method for a time-of-flight device, a computer program and a computer-readable medium.

TECHNICAL BACKGROUND

Generally, time-of-flight (ToF) devices are known, which are used for determining a distance to objects in a scene or a depth map of (the objects in) the scene that is illuminated with light. Typically, ToF devices include an illumination device (e.g. an array of light emitting diodes (LED)), optical parts (e.g. lenses), a ToF sensor such as an image sensor (e.g. array of current assisted photonic demodulator (CAPD) pixels) with read-out circuitry and a control.

For capturing a depth image, the ToF device typically illuminates the scene with, for instance, a modulated light wave and images the backscattered/reflected light wave on the ToF sensor. A gain of the pixels of the ToF sensor, for example, is modulated according to a demodulation signal which may be phase-shifted with respect to the modulation of the emitted light wave for generating ToF data (time-of-flight data) indicative for the distance to the objects in the scene.

It is generally known that transparent or translucent objects (transparent or translucent for the illumination light of the ToF device) may be difficult to detect and that it may be difficult to obtain a silhouette of such objects, in particular in front of a background (e.g., a white wall). Typically, the phase and amplitude images are utilized to attempt to separate the foreground object from the background.

However, in some cases, transparent or translucent objects may be difficult to detect from phase images due to potential noise and low signal returning from the object, while amplitude images may suffer in object detection when there is limited change in a returned signal level.

Although there exist techniques for foreground object extraction based on time-of-flight data, it is generally desirable to improve the existing techniques. SUMMARY

According to a first aspect the disclosure provides an information processing device for a time- of-flight device, the time-of-flight device being configured to acquire time-of-flight data, comprising circuitry configured to: obtain first time-of-flight data representing a background; obtain, based on time-of-flight data acquired by the time-of-flight device, second time-of- flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

According to a second aspect the disclosure provides an information processing method for a time-of-flight device, the time-of-flight device being configured to acquire time-of-flight data, comprising: obtaining first time-of-flight data representing a background; obtaining, based on time-of-flight data acquired by the time-of-flight device, second time- of-flight data in which an object in front of the background is represented; and separating data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

According to a third aspect the disclosure provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the information processing method of according to the second aspect.

According to a fourth aspect the disclosure provides a computer-readable medium having stored thereon the computer program according to the third aspect.

Further aspects are set forth in the dependent claims, the following description and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are explained by way of example with respect to the accompanying drawings, in which:

Fig. 1 schematically illustrates in a block diagram an embodiment of a time-of-flight device and an information processing device; Fig. 2 schematically illustrates an embodiment of a light modulation signal, a received light signal and four demodulation signals;

Fig. 3 schematically illustrates an embodiment of a light modulation signal, a received light signal and integrated voltages of a two-tapped pixel;

Fig. 4 schematically illustrates an embodiment of data points in a quadrature-component-in- phase-component space;

Fig. 5 schematically illustrates an embodiment of a threshold in a quadrature-component-inphase-component space;

Fig. 6 schematically illustrates an embodiment of a threshold in a quadrature-component-in- phase-component space;

Fig. 7 schematically illustrates in a block diagram an embodiment of a time-of-flight device and an information processing device;

Fig. 8 schematically illustrates in a block diagram an embodiment of an object silhouette detection system;

Fig. 9 schematically illustrates an embodiment of a threshold in a quadrature-component-inphase-component space; and

Fig. 10 schematically illustrates in a flow diagram an embodiment of an information processing method.

DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of Fig. 5 is given, general explanations are made.

As mentioned in the outset, time-of-flight (ToF) devices are known which typically illuminate the scene with, for instance, a modulated light wave and images the backscattered/reflected light wave on a ToF sensor such as an image sensor (e.g. array of current assisted photonic demodulator (CAPD) pixels) with read-out circuitry. A gain of the pixels of the ToF sensor, for example, is modulated according to a demodulation signal which may be phase-shifted with respect to the modulation of the emitted light wave for generating ToF data (time-of-flight data) indicative for the distance to the objects in the scene.

As further mentioned in the outset, it is generally known that transparent or translucent objects (transparent or translucent for the illumination light of the ToF device) may be difficult to detect and that it may be difficult to obtain a silhouette of such objects, in particular in front of a background (e.g., a white wall). Typically, the phase and amplitude images are utilized to attempt to separate the foreground object from the background.

For enhancing the general understanding of the present disclosure an embodiment of a ToF device 1 is discussed in the following under reference of Fig. 1, Fig. 2, Fig. 3, and Fig. 4, which also applies to other embodiments of the present disclosure.

Fig. 1 schematically illustrates in a block diagram an embodiment of the ToF device 1 and an information processing device 8-1 or 8-2, which is discussed in the following.

The ToF device 1 is an indirect ToF system (iToF system).

The ToF device 1 includes an illumination device 2, a ToF sensor 3, a lens portion 4, a control 5, a communication interface 6 and a data bus interface 7.

The information processing device 8-1 is a computer with an application processor. In some embodiments, the information processing device 8-2 includes the ToF device 1, for example, the information processing device 8-2 may be a smartphone or camera system with an application processor that processes ToF data obtained from the ToF device 1.

In the following the information processing device 8-1 is discussed for the sake of clarity.

The information processing device 8-1 can communicate and exchange data with the ToF device 1 via the data bus interface 7.

The illumination device 2 includes a light source (e.g., an LED array or laser diode array).

The ToF sensor 3 includes a plurality of pixels (not shown), which are two-tapped CAPD pixels.

The illumination device 2 illuminates a scene with light modulated in accordance with a light modulation signal (e.g., LMS in Fig. 2 and Fig. 3) received from the control 5.

The scene includes a background 9 (e.g., a white wall) and a translucent object 10 (e.g., a plastic bottle) which has an opaque label 11.

The background 9 and the label 11 reflect at least part of the light towards the ToF device 1 which the lens portion 4 gathers and images onto the ToF sensor 3 (received light signal (e.g., RLS in Fig. 2 and Fig 3)). Moreover, the rest of the translucent object 10 (except of the label 11) reflects and scatters part of the illumination light, however, it lets some illumination light pass through such that it is reflected at the background 9 which then passes again through the translucent object 10 until reaching the ToF sensor 3.

Typically, iToF devices perform four correlation measurements - in contrast to intensity-based image sensors which typically measure variations in signal strength - where the modulation light signal is used to drive both the ToF sensor 3 (as a demodulation signal) and the light source, and the correlation between the received light signal and the ToF sensor response driven by the demodulation signal is output. In the four correlation measurements, the phase offset between the light modulation signal applied to the light source and the demodulation signal applied to the ToF sensor 3 is shifted between each correlation measurement (0°, 90°, 180°, 270°), where the correlation measurements with 0° and 180° represent the “in-phase-component” (I), and the correlation measurements with 90° and 270° represent the “quadrature-component” (Q). These I- and Q-component are then used to compute the phase and amplitude of each pixel in the scene.

In this embodiment, the control 5 basically controls the overall operation of the ToF device 1 and controls the ToF device 1 such that four correlation measurements are repeatedly performed for obtaining depth information about the scene. In particular, the control 5 applies a demodulation signal to the ToF sensor 3, which has a predetermined phase-shift with respect to the light modulation signal applied to the light source for generating the ToF data (time-of-flight data).

Fig. 2 schematically illustrates an embodiment of a light modulation signal (LMS), a received light signal (RLS) and four demodulation signals (DM 1-4), which is discussed in the following.

The light modulation signal LMS applied to the illumination device 2 of Fig. 1 is a rectangular modulation signal with a modulation period T. An intensity of emitted infrared light of the light source is then modulated in time according to the light modulation signal LMS. The emitted infrared light is at least partially reflected at the background 9 and the translucent object 10.

The received light signal RLS is basically an intensity of the reflected illumination light at the ToF sensor 3 of Fig. 1, which is phase-shifted with respect to the light modulation signal LMS and varies according to the intensity-modulation of the emitted infrared light. The phase is proportional to distances to objects in the scene.

The ToF sensor 3 performs four correlation measurements corresponding to the demodulation signals DM1, DM2, DM3 and DM4.

The demodulation signal DM1 is phase-shifted by 0° with respect to the light modulation signal LMS. When the demodulation signal DM1 is high, the ToF sensor 3 (each of the plurality of two-tapped CP AD pixels) accumulates an electrical charge ql in accordance with an amount of light incident on the respective pixel and an overlap of the received light signal RLS and the demodulation signal DM1.

The demodulation signal DM2 is phase-shifted by 90° with respect to the light modulation signal LMS. When the demodulation signal DM2 is high, the ToF sensor 3 (each of the plurality of two-tapped CP AD pixels) accumulates an electrical charge q2 in accordance with an amount of light incident on the respective pixel and an overlap of the received light signal RLS and the demodulation signal DM2.

The demodulation signal DM3 is phase-shifted by 180° with respect to the light modulation signal LMS. When the demodulation signal DM3 is high, the ToF sensor 3 (each of the plurality of two-tapped CP AD pixels) accumulates an electrical charge q3 in accordance with an amount of light incident on the respective pixel and an overlap of the received light signal RLS and the demodulation signal DM3.

The demodulation signal DM4 is phase-shifted by 270° with respect to the light modulation signal LMS. When the demodulation signal DM4 is high, the ToF sensor 3 (each of the plurality of two-tapped CP AD pixels) accumulates an electrical charge q4 in accordance with an amount of light incident on the respective pixel and an overlap of the received light signal RLS and the demodulation signal DM4.

The electrical charges ql, q2, q3 and q4, as generally known, are proportional to or representative for, e.g., a voltage signal (electric signal) of the respective pixel output by the ToF sensor 3 and, thus, representative for ToF data (time-of-flight data).

The voltage signal represented by the electrical charge ql corresponds to an in-phase-component value and the voltage signal represented by the electrical charge q3 corresponds to an in-phasecomponent value.

The voltage signal represented by the electrical charge q2 corresponds to a quadraturecomponent value and the voltage signal represented by the electrical charge q4 corresponds to a quadrature-component value.

Then, the phase between the light modulation signal LMS and the received light signal RLS is given by:

Q = q4 - q2,

I = ql — q3.

Here, Q is the quadrature-component (of a pixel) calculated from the two quadrature-component values q2 and q4, and I is the in-phase-component (of a pixel) calculated from the two in-phasecomponent values ql and q3. Together Q and I are also known as IQ value.

Then, the distance is given by:

Moreover, in some embodiments, the amplitude is given by:

VQ² + I² amplitude = - - - .

The amplitude typically represents a signal strength and a confidence level of the corresponding data point.

Hence, in some embodiments, the ToF data acquired by the ToF device 1 include the two in- phase-component values ql and q3 and the two quadrature-component values q2 and q4. In some embodiments, the ToF data acquired by the ToF device 1 further include the phase and the amplitude for each pixel. In some embodiments, the ToF data acquired by the ToF device 1 include only the phase and amplitude for each pixel. In some embodiments, the ToF data acquired by the ToF device 1 include the quadrature-component (e.g., q4-q2) and/or the inphase-component (e.g., ql-q3) for each pixel of the ToF sensor 3.

Fig. 3 schematically illustrates an embodiment of a light modulation signal (LMS), a received light signal (RLS) and integrated voltages (IV) of a two-tapped pixel, which is discussed in the following.

As discussed under reference of Fig. 1, the ToF sensor 3 includes a plurality of CAPD pixels and read-out circuitry, wherein each CAPD pixel has two taps: Tap A and Tap B.

The upper graph schematically illustrates the light modulation signal (LMS) with which the light source of the illumination device 2 of the ToF device 1 is modulated such that the light source emits light to the scene (illuminates the scene) modulated in time according to the LMS. The upper graph schematically illustrates only two modulation periods T1 and T2 for the sake of illustration and simplicity only.

As discussed under reference of Fig. 1, the background 9 and the translucent object 10 in front of the background 9 reflect (and scatter) at least part of the modulated illumination light back to the ToF device 1 which receives it and the lens portion 4 of the ToF device 1 images the received light signal (RLS) onto the ToF sensor 3. The middle graph schematically illustrates the RLS which includes a modulation light (ML) part and an ambient light (AL) part.

The read-out circuitry applies a demodulation signal to Tap A and to Tap B, wherein the demodulation signal applied to Tap A is phase-shifted by 180° with respect to the demodulation signal applied to Tap B. A phase-shift between the LMS and the demodulation signal applied to Tap A is 0°.

The lower graph schematically illustrates the integrated voltages (IV) at Tap A and Tap B (or at a capacitance connected with Tap A or Tap B respectively) during the modulation periods.

In the modulation period Tl, the voltage of Tap A (dashed line) changes during sub-period Tl-1 due to photoelectric conversion of light corresponding to a part Al of the RLS.

Further, in the modulation period Tl, the voltage of Tap B (dashed dotted line) changes during sub-period Tl-2 due to photoelectric conversion of light corresponding to a part Bl of the RLS.

In the modulation period T2, the voltage of Tap A further changes during sub-period T2-1 due to photoelectric conversion of light corresponding to a part A2 of the RLS.

Further, in the modulation period T2, the voltage of Tap B further changes during sub-period T2- 2 due to photoelectric conversion of light corresponding to a part B2 of the RLS.

A signal contribution (SC) acquired by the pixel is given by the difference between the IV of Tap A and Tap B such that typically the ambient light (AL) part is cancelled out. The SC corresponds to the voltage change caused by the modulation light (ML) part of the RLS which thus includes distance information.

Fig. 4 schematically illustrates an embodiment of data points 12 in a quadrature-component-in- phase-component space, which is discussed in the following under reference of Fig. 1 and Fig. 4.

At first, the ToF device 1 acquires ToF data which include the two in-phase-component values ql and q3 and the two quadrature-component values q2 and q4.

The information processing device 8-1 obtains the ToF data acquired by the ToF device 1. The information processing device 8-1 obtains, based on the ToF data acquired by the ToF device 1, second ToF data by calculating, based on the two in-phase-component values ql and q3 and the two quadrature-component values q2 and q4, a phase and an amplitude for each pixel of the ToF sensor 3 of the ToF device 1.

However, as mentioned above, in some embodiments, the ToF data obtained from the ToF device 1 include the phase and amplitude for each pixel such that the second ToF data may correspond to the ToF data acquired by the ToF device 1.

Hence, the second ToF data include the data points 12.

The data points 12 are thus an IQ value for each pixel of the ToF sensor 3 and are plotted in the quadrature-component-in-phase-component space (which may be referred to as IQ-plot), which is spanned by a quadrature-component (Q) axis and an in-phase-component (I) axis.

The angle of a data point in the quadrature-component-in-phase-component space corresponds to its phase and the distance to the origin of the quadrature-component-in-phase-component space corresponds to its amplitude.

Generally, as the scene of Fig. 1 includes the translucent object 10 having the label 11 in front of the background 9, the data points 12 have a large standard deviation and are scattered over a region of the quadrature-component-in-phase-component space, since the phase and the amplitude are typically different for the background 9, the label 11 and the rest of the translucent object 10.

Returning to the general explanations, as has been recognized from Fig. 4, it may be difficult to obtain an accurate depth map of the foreground object (the translucent object 10) based on the data points 12, since it is not clear which part of the data points 12 corresponds to the background 9 and which to the rest of the translucent object 10 through which illumination light can pass and by which illumination light is scattered.

It has thus been recognized that the data points (e.g., the data points 12) in the second ToF data should be separated into background time-of-flight data and into foreground object time-of-flight data.

It has been recognized that the data points should be separated based on a known background for extracting a silhouette or a contour of the foreground object, in particular in cases in which a single pair of phase/amplitude image may include limited information about the foreground object (e.g., for transparent or translucent objects). Hence, some embodiments pertain to an information processing device for a time-of-flight device, wherein the time-of-flight device is configured to acquire time-of-flight data, wherein the information processing device includes circuitry configured to: obtain first time-of-flight data representing a background; obtain, based on time-of-flight data acquired by the time-of-flight device, second time-of- flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

The information processing device may be a computer, a server, a smartphone, a camera (system or module) or the like.

The circuitry may be based on or may include or may be implemented as integrated circuity logic or may be implemented by one or more CPUs (central processing unit), one or more application processors, one or more graphical processing units (GPU), a microcontroller, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit) or the like. The functionality may be implemented by software executed by a processor such as an application processor or the like.

The circuitry may be based on or may include or may be implemented by typical electronic components configured to achieve the functionality as described herein. The circuitry may be based on or may include or may be implemented in parts by typical electronic components and integrated circuitry logic and in parts by software.

The circuitry may include a data bus interface for transmitting (and receiving) data over a data bus.

The data bus interface may be a Camera Serial Interface (CSI) in accordance with MIPI (Mobile Industry Processor Interface) specifications (e.g., MIPII CSI-2 or the like), an I²C (Inter- Integrated Circuit) interface, a Controller Area Network (CAN) bus interface, an FDP-link (Flat Panel Display link), a GSML (Gigabit Multimedia Serial Link), etc. The data bus is in accordance with the corresponding interface specifications.

The circuitry may include a communication interface configured to communicate and exchange data with a computer or processor (e.g., an application processor) over a network (e.g. the internet) via a wired or a wireless connection such as a mobile telecommunications system which may be based on UMTS, LTE or the like (and implements corresponding communication protocols).

The circuitry may include data storage capabilities to store data such as memory which may be based on semiconductor storage technology (e.g., RAM, EPROM, etc.) or magnetic storage technology (e.g. a hard disk drive) or the like.

The information processing device (the circuitry of the information processing device) obtains first time-of-flight data representing a background.

In some embodiments, the first time-of-flight data include at least one in-phase-component value and at least one quadrature-component value. In some embodiments, the first time-of-flight data include two in-phase-component values (corresponding to a phase-shift of 0° and 180° between the light modulation signal and the demodulation signal) and two quadrature-component values (corresponding to a phase-shift of 90° and 270° between the light modulation signal and the demodulation signal) for each pixel of an image sensor of the time-of-flight device. In some embodiments, the first time-of-flight data further include the phase and the amplitude for each pixel of the image sensor of the time-of-flight device. In some embodiments, the first time-of- flight data include only the phase and amplitude for each pixel of the image sensor of the time- of-flight device. In some embodiments, the first time-of-flight data include the quadraturecomponent (e.g., q4-q2) and/or the in-phase-component (e.g., ql-q3) for each pixel of the image sensor of the time-of-flight device.

In some embodiments, the first time-of-flight data is acquired by the time-of-flight device, loaded from a background database, or simulated based on predetermined distances between the time-of-flight device and the background.

The background may be an opaque object (e.g., a white wall) or the like which at least partially reflects illumination light from the time-of-flight device.

In some embodiments, the first time-of-flight data representing the background is based on a user input. For example, a distance between the image sensor of the time-of-flight device and the background is manually input representing the background (e.g., in the case of a flat and perpendicular background). Any type of user input may be considered (gradient distance, specific background shape distance information, etc.).

In some embodiments, the first time-of-flight data representing the background is generated by an algorithm that is configured to infer a background (background distances) from time-of-flight data in which a foreground object is represented. For example, the foreground object is in front of a flat wall and a part of the flat wall is imaged as well, the algorithm is configured to generate the first time-of-flight data representing the background by assuming that the wall stays flat behind the foreground object (e.g., a bottle that is in front of a flat wall, it is assumed that the wall stays flat behind the bottle).

The information processing device (the circuitry of the information processing device) obtains, based on time-of-flight data acquired by the time-of-flight device, second time-of-flight data in which an object in front of the background is represented.

In the following it is assumed that an object in front of the background is present, however, generally, it is only required that the time-of-flight data acquired by the time-of-flight device, based on which the second time-of-flight data is obtained, corresponds to a different measurement than the first time-of-flight data.

In some embodiments, the time-of-flight data acquired by the time-of-flight device include at least one in-phase-component value and at least one quadrature-component value. In some embodiments, the time-of-flight data acquired by the time-of-flight device include two in-phasecomponent values (corresponding to a phase-shift of 0° and 180° between the light modulation signal and the demodulation signal) and two quadrature-component values (corresponding to a phase-shift of 90° and 270° between the light modulation signal and the demodulation signal) for each pixel of an image sensor of the time-of-flight device. In some embodiments, the time-of- flight data acquired by the time-of-flight device further include the phase and the amplitude for each pixel of the image sensor of the time-of-flight device. In some embodiments, the time-of- flight data acquired by the time-of-flight device include only the phase and amplitude for each pixel of the image sensor of the time-of-flight device. In some embodiments, the time-of-flight data acquired by the time-of-flight device include the quadrature-component (e.g., q4-q2) and/or the in-phase-component (e.g., ql -q3) for each pixel of the image sensor of the time-of-flight device.

In some embodiments, the second time-of-flight data correspond to the time-of-flight data acquired by the time-of-flight device. In some embodiments, the second time-of-flight data include at least an amplitude for each pixel of the image sensor of the time-of-flight device. In some embodiments, the second time-of-flight data include at least a phase and an amplitude for each pixel of the image sensor of the time-of-flight device. In some embodiments, the information processing device obtains the second time-of-flight by calculating it from the time- of-flight data acquired by the time-of-flight device. In some embodiments, the second time-of- flight data include the quadrature-component (e.g., q4-q2) and/or the in-phase-component (e.g., ql-q3) for each pixel of the image sensor of the time-of-flight device.

In some embodiments, the object is transparent for illumination light from the time-of-flight device or the object is translucent for the illumination light from the time-of-flight device.

The information processing device (the circuitry of the information processing device) separates data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

In some embodiments, the circuitry is further configured to determined and set the threshold based on the first time-of-flight data.

In some embodiments, the threshold is determined based on a distribution of data points in the first time-of-flight data in the quadrature-component-in-phase-component space. For example, the distribution of the data points may be confined in a circular or elliptic shaped region or the like of the quadrature-component-in-phase-component space such that any data point falling in this region is considered to be a data point corresponding to the background.

Hence, in some embodiments, the foreground object time-of-flight data include each data point of the second time-of-flight data which does not fall within a region of the quadrature- component-in-phase-component space which is associated with the background determined by the threshold.

In some embodiments, the data points of first time-of-flight data corresponding to the same pixels as the foreground object time-of-flight data are subtracted from the foreground object time-of-flight data before applying a spatial filter.

In some embodiments, the first time-of-flight data is subtracted for each pixel from the second time-of-flight data before the threshold is determined. For example, any data points in the second time-of-flight data which correspond to the background may be close to the origin of the quadrature-component-in-phase-component space after subtracting the data points of the first time-of-flight data such that the threshold is determined by a distance to the origin of the quadrature-component-in-phase-component space.

In such embodiments, the foreground object data include the data points in the second time-of- flight data which have a distance (amplitude) larger than the threshold to the origin of the quadrature-component-in-phase-component space, wherein the data points are either the differences between the corresponding data points of the first time-of-flight data and the second time-of-flight data or the original data points of the second time-of-flight data. In some embodiments, the threshold is defined by the Euclidean distance in the quadrature- component-in-phase-component space (IQ space or IQ plot), wherein:

Where d_IQ is the Euclidean distance in the IQ plot, in some embodiments, and Al and AQ are the distances between foreground and background objects in the I and Q axes, respectively.

Hence, in some embodiments, a threshold is determined in the IQ plot, and the Euclidean distance between the foreground data (second time-of-flight data) and the background data (first time-of-flight data) can be used to retrieve a silhouette of the foreground object.

In some embodiments, the threshold is further based on the second time-of-flight data.

As discussed above, typically, two correlation measurements are performed for both the in- phase-component (I) and the quadrature-component (Q) (four correlation measurements in total), because this cancels out effects of ambient light, as well as pixel related mismatches in some embodiments. In some embodiments, as discussed above, there is a subtraction of the background data (first time-of-flight data) from the foreground data (second time-of-flight data) and, thus, the ambient light and pixel mismatches may be cancelled by this subtraction.

In some embodiments, the circuitry is further configured to apply a spatial filter to the foreground object time-of-flight data for increasing a contrast of the object in front of the background.

In some embodiments, the spatial filter is a contour filter or a silhouette filter.

In some embodiments, the time-of-flight data acquired by the time-of-device include quadraturecomponent values and in-phase-component values for each pixel of an image sensor of the time- of-flight device.

In some embodiments, the threshold corresponds to a region in a quadrature-component-in- phase-component space, as discussed above.

It has been recognized that the difference between the first time-of-flight data and the second time-of-flight data may be dominant along a particular axis of the quadrature-component-in- phase-component space (hence, along either the in-phase-component axis or the quadraturecomponent axis).

In some embodiments, the Euclidean distance is dominant along one particular axis of the IQ plot (for example the Q axis). It has been recognized that, in some embodiments, any changes in amplitude due to, for example, a part of a translucent plastic bottle without label, is due to a reduction of signal strength and, thus, reduces the measured amplitude compared to the amplitude of the background data (first time-of-flight data) in the IQ plot (quadrature-component-in-phase-component space) along a particular axis (e.g., a reduction of the Q-component).

It has been recognized that, in some embodiments, any changes in phase due to, for example, the label of the translucent plastic bottle, results in a change of the angle in the IQ plot, which may further result in a reduction of amplitude along the particular axis (e.g., a reduction of the Q- component).

It has thus been recognized that, in some embodiments, by measuring only a reduction of the Q- component, a threshold on the particular axis may be determined to retrieve a contour or a silhouette of the foreground object (to separate the data points in the second time-of-flight data to obtain foreground object time-of-flight data). This may allow to reduce the total number of correlation measurements required in some embodiments (only quadrature-component values or in-phase-component values are required).

Hence, in some embodiments, the threshold corresponds to a threshold on a quadraturecomponent axis or an in-phase-component axis in a quadrature-component-in-phase-component space. In such embodiments, the threshold separates the quadrature-component-in-phase- component space in two parts.

As discussed above, it has been recognized that the number of correlation measurements can be reduced in some embodiments from four to two.

Hence, in some embodiments, the time-of-flight data acquired by the time-of-flight device include only quadrature-component values or in-phase-component values for each pixel of an image sensor of the time-of-flight device. In such embodiments, the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature- component-in-phase-component space, as discussed above. In such embodiments, the threshold separates the quadrature-component-in-phase-component space in two parts, as discussed above.

In some embodiments, the information processing device includes the time-of-flight device.

In some embodiments, the correlation measurements can be reduced from four to two when the first time-of-flight data is aligned along a particular axis of the quadrature-component-in-phase- component space, as also discussed above. Hence, in some embodiments, the time-of-flight device and the background have a predetermined distance to each other such that the first time-of-flight data is aligned along a quadrature-component axis or an in-phase-component axis in a quadrature-component-in-phase- component space.

In some embodiments, the background and the object in front of the background have a distance to each other which corresponds to a quarter of the unambiguous range of the time-of-flight device.

Thus, in such embodiments, due to the predetermined distances between the time-of-flight device, the background and the foreground object (object in front of the background), the first time-of-flight data is aligned along a particular axis of the IQ-space. In such embodiments, any opaque parts of the foreground object are aligned along the perpendicular axis of the IQ-space.

Hence, in some embodiments, the time-of-flight device acquires only quadrature-component values or in-phase-component values for each pixel of an image sensor of the time-of-flight device depending on the alignment of the first time-of-flight data.

It has further been recognized that the number of correlation measurements can be reduced from four to one in some embodiments.

In some embodiments, the time-of-flight data acquired by the time-of-flight device include only one quadrature-component value or one in-phase-component value for each pixel of an image sensor of the time-of-flight device, wherein each pixel is a two-tapped pixel. In such embodiments, the threshold corresponds to a threshold on a quadrature-component axis or an inphase-component axis in a quadrature-component-in-phase-component space. In such embodiments, the threshold separates the quadrature-component-in-phase-component space in two parts.

In some embodiments, as discussed under reference of Fig. 3, the two taps of a two-tapped CAPD pixel are driven with a phase-shift of 180° and, thus, even when only one correlation measurement with, for example, 90° phase-shift between the light modulation signal and the demodulation signal is performed, a single two-tapped pixel is still able to generate two quadrature-component values or two in-phase-component values, for example, the ones corresponding to a phase-shift of 90° and 270° between the light modulation signal and the demodulation signal. In such embodiments, the integrated voltages of the two taps are not subtracted but individually output. Hence, in some embodiments, a single correlation measurement can be used as the Q-component or as the I-component which may further reduce a motion blur. However, there may be a residual mismatch which is not cancelled by a subtraction with the background data (first time-of-flight data), hence a further calibration step may be required in such embodiments.

Accordingly, in some embodiments, the number of correlation measurements is reduced to one and, thus, a single capture method may be provided (further reducing motion blur). However, it may not provide depth information (due to the reduced number of captures) but may be used for contour detection which may be useful, for example, in detecting the presence of translucent objects, or other applications where a foreground contour is required.

Some embodiments pertain to an information processing method for a time-of-flight device, wherein the time-of-flight device is configured to acquire time-of-flight data, wherein the information processing method includes: obtaining first time-of-flight data representing a background; obtaining, based on time-of-flight data acquired by the time-of-flight device, second time- of-flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

The information processing method may be performed by the information processing device as described herein.

The methods as described herein are also implemented in some embodiments as a computer program causing a computer and/or a processor to perform the method, when being carried out on the computer and/or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

Generally, at least one of the following may be achieved/provided in some embodiments:

It is aimed in some embodiments to utilize iToF cameras for transparent or translucent object detection, where the unique properties of an iToF pixel aid in the detection (compared to, for example, an RGB image sensor), wherein a threshold to separate foreground objects from the background is applied. A silhouette filter is in some embodiments applied to the pixels having a depth relative to the threshold corresponding to the foreground object. Typically, iToF devices perform two correlation measurements for both the I and Q axes (four captures total), because this cancels out effects of ambient light, as well as pixel related mismatches. In some embodiments, there is a subtraction of the background data, hence the ambient light and pixel mismatches may be cancelled during this subtraction.

In some embodiments, a silhouette/contour of a foreground object is extracted from its background with high robustness, especially in low contrast conditions.

In some embodiments, the high robustness is achieved by working in the IQ-space and applied to transparent or translucent objects.

In some embodiments, the number of correlation measurements is reduced to two (or one in some embodiments), while maintaining motion robustness for the reduced number of required captures.

In some embodiments, a (low) processing load is provided suitable for applications where realtime high framerate processing is required.

In some embodiments, it may be useful in situations where regular iToF may be difficult, for example, where phases may not be accurately reconstructed. This may include applications such as translucent object detection, or similarly where the scene may be challenging, for example, when large variations in phase and amplitude are present (high contrast objects, objects which provide inconsistent phase values).

The information processing device and/or the information processing method and/or the time-of- flight device described herein may be used, for example, in the following applications:

Suitable for applications in which the background can be well controlled, and where the iToF signal is particularly challenging to measure using state of the art techniques. Applications where motion robustness is important may also benefit. Based on the examples discussed herein, detection of a transparent or translucent bottle, which lends itself to the application of a reverse vending machine, where bottles move quickly on a conveyor belt and must be rapidly detected and their contour measured, in order to provide a correct refund to the customer.

Another application may include automation in the beverage industry, where full beverage containers (which may be difficult for iToF to accurately measure due to scattering) may need to be detected on a production line. Similarly, for food or fruit sorting systems: some foods may suffer from sub-surface scattering, yielding poor depth images from multiple reflections of the light within the food, it could be used to accurate detect the contour of the food and hence track it along a conveyor belt. Applications where motion robustness is critical may also benefit, such as in surveillance where persons are moving, however, because of the limitation of distance between foreground and background, these use cases may be more limited.

Returning to Fig. 5, there is schematically illustrated an embodiment of a threshold 21 in a quadrature-component-in-phase-component space, which is discussed in the following under reference of Fig. 1 and Fig. 5.

The quadrature-component-in-phase-component space shown in Fig. 5, which is referred to as IQ-plot in the following, is spanned by a quadrate-component (Q) axis and an in-phasecomponent (I) axis.

The ToF device 1 acquires first ToF data representing the background 9 (e.g., before the translucent object 10 is placed in front of the background 9), which is then transmitted to the information processing device 8-1, which obtains the first ToF data.

In this embodiment, the first ToF data include two quadrature-component values (90° and 270°) and two in-phase-component values (0° and 180°).

The information processing device 8-1 calculates a phase and an amplitude for the first ToF data to obtain data points 20, as illustrated in the IQ-plot.

The data points 20 represent the background 9 in the IQ-plot.

Then, the ToF device 1 acquires subsequent ToF data in which the translucent object 10 is represented, which are then transmitted to the information processing device 8-1, which obtains the subsequent ToF data.

In this embodiment, the subsequent ToF data include two quadrature-component values (90° and 270°) and two in-phase-component values (0° and 180°).

The information processing device 8-1 calculates a phase and an amplitude for the subsequent ToF data to obtain second ToF data including data points 26, 22 and 23, as illustrated in the IQ- plot.

The data points 26 correspond to a background part.

The data points 22 correspond to other parts than the label 11 of the translucent object 10.

The data points 23 correspond to the label 11 of the translucent object 10.

Basically, the IQ-plot represents a situation where the data points 20 have a high amplitude value, with low standard deviation in phase. The other parts than the label 11 of the translucent object 10 result in a reduction in amplitude as well as a large standard deviation in phase due to the multipath reflections from the background 9 as well as within the translucent object 10. The label 11 of the translucent object 10 gives a reflection, resulting in higher amplitude with a change in phase depending on the distance between the ToF device 1 and the translucent object 10. This may make it difficult to reconstruct an accurate depth map.

Hence, the information processing device 8-1 determines and sets the threshold 21 based on the data points 20, wherein the threshold 21 corresponds to a circular region in the IQ-plot which includes all data points 20.

As depicted in Fig. 5, the data points 26, which correspond to a part of the background 9, fall within the threshold 21.

The information processing device 8-1 separates the data points 26, 22 and 23 in the second ToF data based on the threshold 21 for obtaining foreground object ToF data 25.

The foreground object ToF data 25 include the data points 22 and 23 which do not fall within the circular region determined by the threshold 21, as illustrated by the solid elliptical lines around the data points 22 and around the data points 23.

In some embodiments, the foreground object ToF data 25 include data points 22 and 23 which do not fall within the circular region determined by the threshold 21 subtracted by data points of the data points 20 which correspond to the same pixel.

Then, the information processing device 8-1 applies a spatial filter such as contour or silhouette filter to the foreground object data for increasing a contrast of the translucent object 10 in front of the background 9.

In some embodiments, the information processing device 8-1 detects cluster of data points 22 and 23 not present in the data points 20 as further illustrated by the solid elliptical lines in the IQ- plot. In such embodiments, the information processing device 8-1 determines the threshold 21 based on an Euclidian distance between the a center of the data points 20 and a center of the clusters of data points, for example, the threshold 21 may correspond to the minimum of both Euclidian distances.

Fig. 6 schematically illustrates an embodiment of a threshold 24 in a quadrature-component-in- phase-component space, which is discussed in the following.

The present embodiment is based on the embodiment of Fig. 5 such that unnecessary repetition of similar parts is omitted. In this embodiment, the data points 20 have a high amplitude value along the quadraturecomponent axis (Q) compared to the data points 23 and the data points 22.

The information processing device 8-1 detects such a situation, for example, from a plurality of subsequent measurements.

In such cases, the information processing device 8-1 may determine and set the threshold 24, based on the data points 20, such that the threshold 24 corresponds to a threshold on the quadrature-component axis. The threshold 24 separates the IQ-space in two parts along a line parallel to the in-phase-component axis.

The information processing device 8-1 separates the data points 26, 22 and 23 in the second ToF data based on the threshold 24 for obtaining foreground object ToF data 25.

The foreground object ToF data 25 include the data points 22 and 23 which do not fall within the region of the IQ-plot above the threshold 24, as illustrated by the solid elliptical lines around the data points 22 and around the data points 23.

As the foreground object ToF data 25 is obtained based solely on a comparison of the quadrature-component of the data points 22 and 23 with the threshold 24, the information processing device 8-1 may instruct the ToF device 1 (e.g., transmits a command to the control 5) to reduce the number of correlation measurements from four to two to only acquire the quadrature-component values (e.g., q4 and q2) based on which the quadrature-component can be obtained.

Then, after receiving the instruction, the ToF device 1 starts acquiring only the quadraturecomponent values (e.g., q4 and q2) and outputs them to the information processing device 8-1.

Then, the information processing device 8-1 obtains the second ToF data by calculating the quadrature-component from the ToF data acquired by the ToF device 1.

Hence, the information processing device 8-1 is able to separate the data points in the second ToF data based on the threshold 24 to obtain the foreground object ToF data 25 even with the reduced number of correlation measurements.

Then, the information processing device can obtain a contour or a silhouette from the foreground object ToF data 25.

Moreover, the information processing device 8-1 may further reduce the number of correlation measurements from four to one, since the ToF sensor 3 includes two-tapped CAPD pixels which can output two quadrature-component values at once such that only one correlation measurement must be performed to obtain the quadrature-component.

Fig. 7 schematically illustrates in a block diagram an embodiment of a ToF device 1 and an information processing device 8-1 (or 8-2), which is discussed in the following.

The ToF device 1 corresponds to the ToF device 1, however, the ToF device 1 and the background 9 have a predetermined distance (d) to each other, for example, the distance (d) corresponds to the unambiguous range of the ToF device 1 such that the first ToF data is aligned, for example, along the quadrature-component axis (Q) in the IQ-plot (the quadrature-component- in-phase-component space). As generally known, the unambiguous range of the ToF device 1 is related to the modulation frequency of the light modulation signal (LMS).

Then, the translucent object 10 is placed relative to the background 9 at a distance (d_fg=d- d_bg), wherein the background 9 and the translucent object 10 have a distance (d bg) to each other which corresponds to a quarter of the unambiguous range. The background 9 and the translucent object 10 are thus constrained to a fixed separation distance relative to the modulation frequency, which may allow the detection of objects which are not at the same depth as the background.

Thereby, the ToF data acquired by the ToF device 1 corresponding to the label 11 of the translucent object 10 is aligned along the perpendicular axis of the IQ-plot, here along the in- phase-component axis (I).

Hence, in this embodiment, only component values from the same axis as the background 9 are required to be stored (here the Q-axis), i.e., only two correlation measurements are required instead of four, as also discussed under reference of Fig. 6.

Thus, the ToF device 1 is instructed to only perform the correlation measurements corresponding to the quadrature-component. This reduces the total number of correlation measurements required for obtaining a contour or silhouette of the translucent object, which may reduce motion blur.

A specific example is discussed in the following without limiting the disclosure in this regard.

The background 9 is to be aligned approximately with one axis (e.g., the Q-axis). The foreground object (e.g., the translucent object 10) is then to be physically offset by a distance equivalent to (a quarter) of the unambiguous range d u (d_u=c/(2f_m ), wherein f rn is the modulation frequency and c is the speed of light). For example, with a modulation frequency of 100 MHz (d_u=1.5 m), the foreground object and the background 9 should be separated by approximately 0.375 m.

As a result, any received light with reduced amplitude (for example, the other parts than the label 11 of the translucent object 10) may manifest as a reduction on the axis aligned with the background (e.g., a very low Q-value). Similarly, because the foreground object is aligned with the perpendicular axis, objects at this distance will have a value very close to zero on the background axis (e.g., a Q-value close to zero). Hence, both changes and phase and amplitude are detected, while only utilising the background axis values (i.e., only Q values are required to be analysed).

Hence, in some embodiments, a simple threshold can be used to detect any changes in the background axis values from a difference between the first ToF data and the second ToF data, yielding the contour of the foreground object. In some embodiments, other edge detection algorithms could be used on this difference, for example, canny edge detection. Other approaches might use machine learning or Al (“artificial intelligence”) to learn the background, which may further improve the extraction of the foreground contour.

As discussed above, only two subsequent correlation measurements are required (e.g., Q=C_90- C 270, where C_<|> is a correlation measurement with a particular phase offset <|) on the demodulation signal), which may result in greater motion robustness compared to typical iToF.

As discussed above, any reduction of signal strength (due to the other parts than the label 11 of the translucent object 10) may manifest in a change in amplitude.

As discussed above, any changes in phase (due to the label 11 of the translucent object 10) may manifest in a change in angle in the IQ plot.

Fig. 8 schematically illustrates in a block diagram an embodiment of an object contour detection system 30, which is discussed in the following under reference of Fig. 8 and Fig. 9.

The object contour detection system 30 is a bottle refund system.

The object contour detection system 30 includes a background 31, a conveyor belt 32 on which bottles are transported from left to right, a ToF device 34 and an information processing device 35.

The ToF device 34 acquires first ToF data representing the background 31 in a period between two bottles 33 (e.g., when no bottle is in the field of view of the ToF device 34, since two bottles have a minimum distance to each other which can be used for acquiring the first ToF data). The ToF device 34 further acquires subsequent ToF data in which a bottle in front of the background 31 is represented.

The ToF device 34 and the background 9 have a predetermined distance (d) to each other, for example, the distance (d) corresponds to the unambiguous range of the ToF device 34 such that the first ToF data is aligned, for example, along the quadrature-component axis (Q) in the IQ- plot (the quadrature-component-in-phase-component space).

The bottles 33 have a distance (d_fg=d-d_bg) to the ToF device 34, wherein the background 31 and the bottles 33 have a distance (d bg) to each other which corresponds to a quarter of the unambiguous range.

Hence, in this embodiment, the ToF device 34 acquires only quadrature-component values and, thus, is only required to perform two correlation measurements instead of four for obtaining a contour of the bottles 33.

Fig. 9 schematically illustrates an embodiment of a threshold 41 in a quadrature-component-in- phase-component space (which is referred to as IQ-plot in the following).

The information processing device 35 obtains data points 40 for the first ToF data by calculating the quadrature-component and, thus, the data points 40 are aligned on the Q-axis.

The information processing device 35 calculates the quadrature-component for the subsequent ToF data to obtain the second ToF data including data points 42 (corresponding to other parts than a label on the bottles 33) and 45 (corresponding to a part of the background 31).

The data points 43 correspond to a label on the bottles 33 and are thus aligned on the I-axis due to the distance configuration of the object contour detection system 30. For example, the information processing device 35 classifies each calculated data point as one of the data points 43 when the quadrature-component is zero or close to zero (e.g., by applying a predetermined threshold).

The information processing device 35 determines and sets the threshold 41 based on the data points 40, wherein the threshold 41 corresponds to a threshold on the quadrature-component axis.

The information processing device 35 obtains foreground object ToF data 44 including the data points 42 which are below the threshold 41 (and the data points 43), as illustrated by the solid elliptical line around the data points 42 and the data points 43. Then, the information processing device 35 can obtain the contour of the bottles 33, moreover, the information processing device 35 may further apply a spatial filter to the foreground object ToF data for increasing a contrast of the bottles 33.

Fig. 10 schematically illustrates in a flow diagram an embodiment of an information processing method 100, which is discussed in the following.

The information processing method 100 may be performed by the information processing device as described herein.

At 101, first time-of-flight data representing a background are obtained, as discussed herein.

At 102, based on time-of-flight data acquired by the time-of-flight device, second time-of-flight data is obtained in which an object in front of the background is represented, as discussed herein.

At 103, data points in the second time-of-flight data are separated based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time- of-flight data, as discussed herein.

It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

Note that the present technology can also be configured as described below.

(1) An information processing device for a time-of-flight device, wherein the time-of-flight device is configured to acquire time-of-flight data, including circuitry configured to: obtain first time-of-flight data representing a background; obtain, based on time-of-flight data acquired by the time-of-flight device, second time-of- flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

(2) The information processing device of (1), wherein the circuitry is further configured to apply a spatial filter to the foreground object time-of-flight data for increasing a contrast of the object in front of the background.

(3) The information processing device of (2), wherein the spatial filter is a contour filter or a silhouette filter.

(4) The information processing device of any one of (1) to (3), wherein the circuitry is further configured to determine and set the threshold based on the first time-of-flight data.

(5) The information processing device of any one of (1) to (4), wherein the time-of-flight data acquired by the time-of-flight device include quadrature-component values and in-phasecomponent values for each pixel of an image sensor of the time-of-flight device.

(6) The information processing device of (5), wherein the threshold corresponds to a region in a quadrature-component-in-phase-component space.

(7) The information processing device of (5) or (6), wherein the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature- component-in-phase-component space.

(8) The information processing device of any one of (1) to (4), wherein the time-of-flight data acquired by the time-of-flight device include only quadrature-component values or inphase-component values for each pixel of an image sensor of the time-of-flight device.

(9) The information processing device of (8), wherein the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature- component-in-phase-component space.

(10) The information processing device of any one of (1) to (4), wherein the time-of-flight data acquired by the time-of-flight device include only one quadrature-component value or one in-phase-component value for each pixel of an image sensor of the time-of-flight device, wherein each pixel is a two-tapped pixel.

(11) The information processing device of (10), wherein the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature- component-in-phase-component space. (12) The information processing device of any one of (1) to (11), including the time-of-flight device.

(13) The information processing device of (12), wherein the time-of-flight device and the background have a predetermined distance to each other such that the first time-of-flight data is aligned along a quadrature-component axis or an in-phase-component axis in a quadrature- component-in-phase-component space.

(14) The information processing device of (13), wherein the background and the object in front of the background have a distance to each other which corresponds to a quarter of the unambiguous range of the time-of-flight device.

(15) The information processing device of (14), wherein the time-of-flight device acquires only quadrature-component values or in-phase-component values for each pixel of an image sensor of the time-of-flight device depending on the alignment of the first time-of-flight data.

(16) The information processing device of any one of (1) to (15), wherein the first time-of- flight data is acquired by the time-of-flight device, loaded from a background database, or simulated based on predetermined distances between the time-of-flight device and the background.

(17) The information processing device of any one of (1) to (16), wherein the object is transparent for illumination light from the time-of-flight device or the object is translucent for the illumination light from the time-of-flight device.

(18) An information processing method for a time-of-flight device, wherein the time-of-flight device is configured to acquire time-of-flight data, including: obtaining first time-of-flight data representing a background; obtaining, based on time-of-flight data acquired by the time-of-flight device, second time- of-flight data in which an object in front of the background is represented; and separating data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

(19) A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the information processing method of (18).

(20) A computer-readable medium having stored thereon the computer program of (19). (21) A computer program comprising program code causing a computer to perform the method according to (18), when being carried out on a computer.

(22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to (18) to be performed.

Claims

1. An information processing device for a time-of-flight device, the time-of-flight device being configured to acquire time-of-flight data, comprising circuitry configured to: obtain first time-of-flight data representing a background; obtain, based on time-of-flight data acquired by the time-of-flight device, second time-of- flight data in which an object in front of the background is represented; and separate data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

2. The information processing device according to claim 1, wherein the circuitry is further configured to apply a spatial filter to the foreground object time-of-flight data for increasing a contrast of the object in front of the background.

3. The information processing device according to claim 2, wherein the spatial filter is a contour filter or a silhouette filter.

4. The information processing device according to claim 1, wherein the circuitry is further configured to determine and set the threshold based on the first time-of-flight data.

5. The information processing device according to claim 1, wherein the time-of-flight data acquired by the time-of-flight device include quadrature-component values and in-phasecomponent values for each pixel of an image sensor of the time-of-flight device.

6. The information processing device according to claim 5, wherein the threshold corresponds to a region in a quadrature-component-in-phase-component space.

7. The information processing device according to claim 5, wherein the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature-component-in-phase-component space.

8. The information processing device according to claim 1, wherein the time-of-flight data acquired by the time-of-flight device include only quadrature-component values or in-phasecomponent values for each pixel of an image sensor of the time-of-flight device.

9. The information processing device according to claim 8, wherein the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature-component-in-phase-component space.

10. The information processing device according to claim 1, wherein the time-of-flight data acquired by the time-of-flight device include only one quadrature-component value or one inphase-component value for each pixel of an image sensor of the time-of-flight device, wherein each pixel is a two-tapped pixel.

11. The information processing device according to claim 10, wherein the threshold corresponds to a threshold on a quadrature-component axis or an in-phase-component axis in a quadrature-component-in-phase-component space.

12. The information processing device according to claim 1, comprising the time-of-flight device.

13. The information processing device according to claim 12, wherein the time-of-flight device and the background have a predetermined distance to each other such that the first time- of-flight data is aligned along a quadrature-component axis or an in-phase-component axis in a quadrature-component-in-phase-component space.

14. The information processing device according to claim 13, wherein the background and the object in front of the background have a distance to each other which corresponds to a quarter of the unambiguous range of the time-of-flight device.

15. The information processing device according to claim 14, wherein the time-of-flight device acquires only quadrature-component values or in-phase-component values for each pixel of an image sensor of the time-of-flight device depending on the alignment of the first time-of- flight data.

16. The information processing device according to claim 1, wherein the first time-of-flight data is acquired by the time-of-flight device, loaded from a background database, or simulated based on predetermined distances between the time-of-flight device and the background.

17. The information processing device according to claim 1, wherein the object is transparent for illumination light from the time-of-flight device or the object is translucent for the illumination light from the time-of-flight device.

18. An information processing method for a time-of-flight device, the time-of-flight device being configured to acquire time-of-flight data, comprising: obtaining first time-of-flight data representing a background; obtaining, based on time-of-flight data acquired by the time-of-flight device, second time- of-flight data in which an object in front of the background is represented; and separating data points in the second time-of-flight data based on a threshold for obtaining foreground object time-of-flight data, wherein the threshold is based on the first time-of-flight data.

19. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the information processing method of claim 18.

20. A computer-readable medium having stored thereon the computer program of claim 19.