EP4413397A1

EP4413397A1 - Apparatuses and methods for event guided depth estimation

Info

Publication number: EP4413397A1
Application number: EP22789239.5A
Authority: EP
Inventors: Manasi MUGLIKAR; Diederik Paul MOEYS; Davide SCARAMUZZA
Original assignee: Universitaet Zuerich; Zurich Universitaet Institut fuer Medizinische Virologie; Sony Semiconductor Solutions Corp
Current assignee: Universitaet Zuerich; Zurich Universitaet Institut fuer Medizinische Virologie; Sony Semiconductor Solutions Corp
Priority date: 2021-10-08
Filing date: 2022-09-30
Publication date: 2024-08-14
Also published as: WO2023057343A1

Abstract

The present disclosure relates to an apparatus for event-based depth measurement, the apparatus comprising an event-based vision sensor configured to detect one or more areas of motion in a scene, and a depth detector configured to determine depth information for the detected one or more areas of motion with a higher spatial resolution than for a remainder of the scene.

Description

APPARATUSES AND METHODS FOR EVENT GUIDED DEPTH ESTIMATION

Field

The present disclosure generally relates to optical imaging devices, particularly optical assemblies for depth estimation. Examples relate to apparatuses and methods for event guided depth estimation.

Background

High-resolution and high-speed depth estimation is desirable in many applications such as robotics, augmented reality (AR) and virtual reality (VR), and 3D modeling. However, these come with an associated cost of increasing the camera resolution and framerate, leading to higher bandwidth. Event-based Structured Light (SL) systems employing event cameras circumvent this problem by suppressing redundant information and generating events asynchronously only where the logarithmic brightness change exceeds a threshold. Therefore, no redundant data may be created or processed, and in turn, that processing may be faster than frame-based algorithms. This type of data may encode location, polarity, and timing of logarithmic changes in brightness perceived by pixels of an event camera.

An event camera, also known as event-based vision sensor (EVS) or dynamic vision sensor (DVS), refers to an imaging sensor that responds to local changes in brightness. Pixels of an EVS operate independently and asynchronously. This property allows every pixel to generate an event exactly at the point in time when its illumination changes, leading to a very fast response time; the latency typically lies in the order of tens of microseconds. An increase in brightness may trigger a so called ON-event, a decrease may trigger an OFF-event. Modem EVS have microsecond temporal resolution, 120 dB dynamic range, and less under/overex- posure and motion blur than frame cameras.

Previously presented event-based SL systems demonstrated the capabilities of such systems with low-resolution EVS. EVS manufacturers are now producing up to IMP resolution EVS. However, high-resolution EVS lead to a higher event rate with significant background noise in the events, leading to higher depth errors. This error may be due to buffering and readout delays that precede the timestamping mechanism. The more events have to wait in a queue before this operation, the less precise their timestamps will be. Different compromises and assumptions can be made to design readout strategies to increase throughput at the cost of precision (scanning vs arbitrated readouts).

This problem can be solved (without modifying the pixel architecture) by sparsely sampling the scene. However, sparse sampling strategies such as grid-based sampling or random sampling do not consider the scene geometry.

Thus, there is a demand for improved depth estimation concepts.

Summary

This demand is addressed by apparatuses and methods in accordance with the independent claims. Possibly advantageous embodiments are addressed by the dependent claims.

According to a first aspect, the present disclosure proposes an apparatus for event-based depth measurement. The apparatus comprises an event-based vision sensor (event camera) which is configured to detect one or more areas of motion (events) in a scene. The apparatus further comprises a depth detector configured to determine depth (distance) information for the detected one or more areas of motion with a higher spatial resolution than for a remainder of the scene. Thus, the present disclosure proposes a bio-inspired event-guided depth estimation concept. Inspired from human ability to focus with higher resolution on areas of interest and decrease the resolution for background, it is proposed an event-based vision sensor (EVS) based depth sampling as events naturally encode relative motion of the scene.

In some embodiments, the apparatus further comprises analog and/or digital processing circuitry configured to, based on information on the detected one or more areas of motion in the scene, control an illumination source of the depth detector to illuminate or scan the detected one or more areas of motion with a higher spatial resolution. Thus, the processing circuitry may adaptively control a scanning frequency based on the information on the detected one or more areas of motion. In some embodiments, the apparatus further comprises an optical splitter configured to provide the same field of view of the scene for the event-based vision sensor and the depth detector. In this way, the same field of view may be shared among depth detector (or its illumination source) and the EVS.

In some embodiments, the apparatus further comprises analog and/or digital processing circuitry configured to compute depth information based on sensor output signals of the depth detector. For example, the processing circuitry may comprise one or more machine learning networks for so-called depth completion.

In some embodiments, the depth detector comprises a laser point projector (as illumination source) configured to scan the detected one or more areas of motion with the higher spatial resolution than the remainder of the scene.

In some embodiments, the depth detector comprises a second event-based vision sensor sensitive to a wavelength and pulse frequency of an illumination source of the depth detector. The apparatus may further comprise analog and/or digital processing circuitry configured to compute depth information based on triangulating illumination events of the illumination source and detected events of the second event-based vision sensor. Here, one EVS may be used for motion detection and a second EVS may be used for depth estimation.

In some embodiments, the event-based vision sensor comprises a first set of pixels configured to detect the one or more areas of motion and a second set of pixels sensitive to a wavelength and pulse frequency of an illumination source of the depth detector. The apparatus further comprises analog and/or digital processing circuitry configured to compute depth information based on triangulating illumination events of the illumination source and detected events of the second set of pixels. Here, the second EVS may be omitted. Instead, a single EVS may be used for motion detection and depth estimation.

In some embodiments, the apparatus further comprises analog and/or digital processing circuitry configured to determine a respective bounding box around the detected one or more areas of motion. The depth detector may be configured to scan areas within a respective bounding box with a higher spatial resolution than outside the bounding box. In some embodiments, the depth detector is configured to perform an initial scan of the scene with full spatial resolution and to adapt the spatial resolution for subsequent scans based on the detected one or more areas of motion in the scene.

According to a further aspect, the present disclosure proposes a method for event-based depth measurement. The method includes detecting one or more areas of motion in a scene with an event-based vision sensor and determining depth information for the detected one or more areas of motion with a higher spatial resolution than for a remainder of the scene.

Embodiments of the present disclosure take inspiration from human perception, which involves foveating: areas of interest are scanned with the highest resolution while the other regions are scanned with lower resolution. The area of interest for robotics applications is often large contrast features or dynamic objects in the scene. Therefore, identifying these areas with low latency in challenging environments is essential. Therefore, it is proposed to use an event camera to identify these areas of interest as events naturally correspond to moving objects (assuming brightness constancy) and therefore do not require further processing to segment the area of interest. The present disclosure provides a depth sensor that can scan dynamic objects with low latency. The skilled person having benefit from the present disclosure will appreciate that ideas presented here are applicable for a wide range of depth sensing modalities like LIDAR, Time of Flight (ToF) and standard stereo.

Brief description of the Figures

Some examples of apparatuses and/or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which

Fig. 1 shows an apparatus for event-based depth measurement according to a first embodiment;

Fig. 2A shows an apparatus for event-based depth measurement according to a second embodiment;

Fig. 2B shows an apparatus for event-based depth measurement according to a third embodiment; Fig. 3 A shows a top view of an EVS pixel matrix with a first set of pixels configured to detect events (motion) and a second set of pixels sensitive to a wavelength and pulse frequency of an illumination source of the depth detector for depth estimation;

Fig. 3B shows a side view of two adjacent EVS pixels, one for event (motion) detection, the other one for depth estimation;

Fig. 4 show an example event time surface with sparse sampling and adaptive fove- ating depth;

Fig. 5 shows a method for event-based depth measurement according to a first embodiment; and

Fig. 6 shows a method for event-based depth measurement according to a second embodiment.

Detailed Description

Some examples are now described in more detail with reference to the enclosed figures. However, other possible examples are not limited to the features of these embodiments described in detail. Other examples may include modifications of the features as well as equivalents and alternatives to the features. Furthermore, the terminology used herein to describe certain examples should not be restrictive of further possible examples.

Throughout the description of the figures same or similar reference numerals refer to same or similar elements and/or features, which may be identical or implemented in a modified form while providing the same or a similar function. The thickness of lines, layers and/or areas in the figures may also be exaggerated for clarification.

When two elements A and B are combined using an 'or', this is to be understood as disclosing all possible combinations, i.e. only A, only B as well as A and B, unless expressly defined otherwise in the individual case. As an alternative wording for the same combinations, "at least one of A and B" or "A and/or B" may be used. This applies equivalently to combinations of more than two elements.

If a singular form, such as “a”, “an” and “the” is used and the use of only a single element is not defined as mandatory either explicitly or implicitly, further examples may also use several elements to implement the same function. If a function is described below as implemented using multiple elements, further examples may implement the same function using a single element or a single processing entity. It is further understood that the terms "include", "including", "comprise" and/or "comprising", when used, describe the presence of the specified features, integers, steps, operations, processes, elements, components and/or a group thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, processes, elements, components and/or a group thereof.

Besides, radar, lidar, time-of-flight (ToF), stereo cameras, the problem of depth estimation may also be considered using an illumination source, such as, for example, a laser point projector (laser scanner), and an EVS (event camera). The latter concept is also commonly referred to as “structured light” and is considered a reliable technique to estimate depth. A scene may be illuminated by a laser beam inside the laser scanner, moving in a raster scanning fashion. For example, one or more moveable mirrors may be used to steer the laser beam. Light reflected from the scene may generate events in the EVS. An EVS generates an event ek = (xk, tk, pk) at time tk when a logarithmic brightness at the pixel xk = xk, yk)^T increases or decreases by predefined threshold C: L( k, tk) - (xk, tk - tk = pkC, where pk G {-1, +1 } denotes the sign (polarity) of the brightness change, and Afe is the time since the last event at the same pixel location. Assuming the travel time for light from the laser scanner to the camera being small, an event pixel may be mapped to the corresponding projector pixel using the event timestamp. Event-based Structured Light (SL) uses the event timestamp to map the event pixel to the projector pixel. Disparity (d) may be converted to depth (Z), given a system calibration parameters baseline (Z>) and focal length (f) as:

Depth, thus, depends on accurate event timestamps. However, non-idealities of events may lead to noise in the timestamps. Currently, available EVS have a minimum possible temporal resolution of 1 ps. However, in practice, latency, transistor noise and variable readout delays become jitter and noise sources in the data, which may decrease the precision of the event timestamping. Such non-idealities are design- and manufacturer-specific. Latency may be defined as the time it takes for an event to be registered since the moment logarithmic change in intensity exceeds the threshold, it can currently range on average from a few microseconds to hundreds of milliseconds, depending on bias settings, manufacturing process and illumination level. Transistor noise may be defined as the random transistor noise of the circuits, it may also depend on settings and illumination. This noise randomly changes the measured signal, leading to threshold-comparisonjitter. Other non-idealities may encompass parasitic photocurrents and junction leakages, these effects bias event generation to one specific polarity. Read-out architectures may be arbitrated architectures, which preserve or partially preserve the order of the pixels’ firing. They may lead to significant queuing delays before the timestamping operation. This may be particularly noticeable when the number of active pixels (and resolution) scale up. Scanning readouts, on the other hand, may limit the possible delays by sacrificing event timing resolution. Jitter may be defined as the random variation that appears in timestamps. It may depend on all of the aforementioned factors, all of which increase the unpredictability and imprecision of the event timing.

As mentioned above, accuracy of an event-based SL system depends on the accuracy of event timestamps. Therefore, for consecutive events triggered by laser to be uniquely distinguished the timestamps should differ by at least 1 ps, which is the event camera timestamp resolution. The time it takes for the laser beam of the projector to move to the next pixel, At, is inversely proportional to the scanning frequency f and spatial resolution of W ^x H pixels:

This approach may be defined as dense sampling since it illuminates every pixel. At is lower than 1 ps for almost every sensor resolution for scanning frequency higher than 60 Hz. Since At depends on the scanning frequency f and sensor resolution W x H. there is no simple way to ensure At is higher than EVS temporal resolution. Similarly, another disadvantage of dense sampling arises from high-event rates. Event-rate, measured in MEv/s, is directly proportional to the scanning frequency f and spatial resolution W x H. Event-rate significantly affects the event timestamps as a high event rate would mean a high latency in the event timestamps in the order of ms. High latency can lead to event being dropped due to an overcrowded eventbuffer queue or wrong timestamps allocated to events based on the read-out architecture. The present disclosure therefore focuses on sparse sampling to increase efficiency.

Sparse sampling increases the At between consecutive events by scanning only a fraction of the pixels. The At is now proportional to the number of pixels skipped between two illuminated pixels (defined as TV). This allows to control the timestamp errors up to a certain extend.

Sparse sampling, however, does not consider a scene’s geometry. This can lead to undersampling of areas of fast-moving objects and oversampling planar areas, leading to redundant information. Conversely, sparse sampling is not always desirable in areas of interest (usually of object motion). In an autonomous driving scenario, for example, if a fast-moving vehicle or pedestrian moves in front of a car, it is useful to have more depth information about this. For robotics applications, dynamic objects in the scene may be important for obstacle avoidance or local planning of trajectories. These scenarios make it desirable to have a higher resolution in these areas of interest at a low latency. EVS are well suited for this application as they naturally encode for motion in their data (assuming constant brightness), eliminating static redundant information. A strong parallelism is present in the human retina, together with low processing latency and high dynamic range.

Sparse sampling may have the disadvantage of under-sampling areas of interest and over- sampling planar areas. The present disclosure therefore proposes to use an event-based sampling for depth. As events naturally encode the relative motion of the scene with high temporal resolution, they are ideal for this task.

Fig. 1 schematically illustrates an apparatus 100 for event-based depth measurement according to an embodiment of the present disclosure.

Apparatus 100 comprises an EVS 110 configured to detect one or more areas of motion 112 in a scene 114 under observation. For example, the scene may be a (partial) surrounding of a vehicle. In contrast to conventional cameras that produce images at a fixed rate, pixels in EVS 110 may operate independently and asynchronously, responding to intensity changes by producing events. Events may be represented by the x, y pixel location and timestamp t (in microseconds) of an intensity change as well as its polarity (i.e., whether the pixel became darker or brighter). An area of motion 112 may trigger one or more events in EVS 110 and may comprise one or more objects moving relative to the EVS 110. For example, a moving object may be a moving vehicle, a pedestrian, a ball, or the like. The skilled person having benefit from the present disclosure will appreciate that there are numerous examples for moving objects.

Apparatus 100 further comprises a depth detector 120 configured to determine depth information 122 for the detected one or more areas of motion 112 with a higher spatial resolution than for a remainder 116 of the scene 114. For this purpose, EVS 110 may be communicatively coupled to depth detector 120 as indicated by reference numeral 115. Communication interface 115 may convey spatial information on the detected one or more areas of motion 112 from EVS 110 to depth detector 120. Depth detector 120 may then be controlled based on the spatial information on the detected one or more areas of motion 112. In particular, depth detector 120 may be controlled to increase its spatial scanning or sampling resolution for the detected one or more areas of motion 112 compared to the remainder 116 of the scene 114. Put differently, depth detector 120 may be controlled to decrease its spatial scanning or sampling resolution for remainder 116 of the scene 114 compared to the detected one or more areas of motion 112.

Depth detector 120 may illuminate the same scene 114 as observed by the EVS 110 and may comprise any ranging device capable of determining depth (distance) information at different scanning or sampling rates, such as radar devices, lidar devices, stereo cameras, a time-of- flight (ToF) sensors, or an event-based Structured Light (SL) system including a laser point projector / laser scanner (as optical transmitter) in conjunction with an EVS (as optical receiver), for example. Depth information 122 may comprise spatial information of the illuminated scene 114 together with associated depth or distance information. For example, depth information 122 may come in the form of triplets comprising spatial coordinates (x, y) and associated distance (d), for example, (x, y, d). Spatial scanning may take place with higher spatial resolution, for example higher scanning frequency, for the detected one or more areas of motion 112 compared to areas 116 of less or no motion (for example, a background of the scene).

In some embodiments, depth information 122 may comprise raw sensor output signals of the depth detector 120. Alternatively, depth information 122 may comprise depth information obtained after further processing the raw sensor output signals. This processing may take place in analog and/or digital depth processing circuitry. In some embodiments, the depth processing circuitry may implement machine learning algorithms, such as (fully) convolutional neural networks, for example, for depth completion.

Fig. 2A illustrates a more specific embodiment of an event-based SL apparatus 200 where the depth detector 120 comprises a laser point projector / laser scanner 220-Tx as illumination source and a second EVS 220-Rx tuned to observe events generated by the laser point projector 220-Tx. Laser point projector 220-Tx and second EVS 220-Rx together constitute a depth detector 220. In some embodiments, laser point projector 220-Tx may emit infrared (IR) light beams. Laser point projector 220-Tx may use one or more moveable mirrors to steer the laser beams. EVS 110 is configured to detect areas of motion 112 in the scene 114 and to provide corresponding control data for controlling the spatial (scanning) resolution of depth detector 220. As can be seen from Fig. 2A, some or all of the devices 110, 220-Tx, and 220-Rx may include lens assemblies to improve optical imaging properties of the respective devices.

Event-based depth measurement apparatus 200 comprises an optical splitter 240 coupled to both, laser point projector 220-Tx and EVS 110. Optical splitter 240 is configured to provide the same field of view of the scene 114 for the EVS 110 and depth detector 200 or its illumination source 220-Tx. Optical splitter 240 may be transparent to laser beams 224 emitted from laser point projector 220-Tx towards the scene 114 and may at the same time limit the field of view of the laser point projector 220-Tx. Optical splitter 240 may comprise a mirror assembly 242 which is configured to direct light reflected from the scene 114 to EVS 110. Thus, by using optical splitter 240, the EVS 110 and the illumination source (projector) 220-Tx may share the same field of view avoiding any occluded regions.

Events detected by EVS 110 may be used to guide the laser point projector 220-Tx to illuminate the detected areas 112 of relative motion at higher scanning frequency. For this purpose, EVS 110 may forward motion data to control circuitry 230 (e.g., one or more DSPs, ASICs, FPGAs, etc.). Control circuitry 230 may determine the areas 112 of relative motion based on the motion data received from EVS 110 and based on certain predefined threshold values, for example. Control signals based on the areas 112 of relative motion may then be fed back to the laser point projector 220-Tx to adapt its scanning frequency accordingly. That is, laser point projector 220-Tx may apply a higher scanning frequency for the areas 112 of relative motion whereas a lower scanning frequency may be applied for the remainder of scene 114.

In the event-based structured light (SL) example illustrated in Fig. 2A, depth detector 220 comprises a second EVS 220-Rx sensitive to a wavelength (e.g., IR) and pulse/scanning frequency of laser point projector 220-Tx. Second EVS 220-Rx may be used to compute depth information based on triangulating illumination events of laser point projector 220-Tx and detected events of the second EVS 220-Rx. Laser point projector 220-Tx illuminates the scene 114 whose surface is to be measured at an angle. EVS 220-Rx registers the scattered light corresponding to events (motion). Knowing the beam direction and the distance between EVS 220-Rx and light source 220-Tx, the distance from the (moving) object to the EVS 220-Rx can be determined. The connection between EVS 220-Rx and light source 220-Tx as well as the two beams from and to the (moving) object 112 form a triangle, hence the name triangulation. If the procedure is carried out in a raster-like or continuously moving manner, the surface relief can be determined with high accuracy. This computation of depth information 122 may be performed via one or more digital signal processing circuits which may be implemented together with control circuitry 230. After calibrating this system EVS and projector, depth may be estimated by triangulating corresponding points on the time surface of the second EVS 220-Rx and the projector 220-Tx.

Fig. 2B illustrates an embodiment of an event-based depth measurement apparatus 250 where the positions of the second EVS 220-Rx and the projector 220-Tx have been switched without deviating from the working principle. Here, optical splitter 240 is configured to provide the same field of view of the scene 114 for the first EVS 110 for motion detection and the second EVS 220-Rx for depth estimation.

While Figs. 2A, B illustrate embodiments using two distinct EVS, one for event detection, the other for depth estimation, Fig. 3A, B illustrate embodiments using merely one EVS 310 for both event (motion) detection and depth estimation. EVS 310 comprises a first set of pixels 312 configured to detect the one or more areas of motion and a second set of pixels 314 sensitive to a wavelength (e.g., in the IR region) and pulse frequency of laser point projector 220-Tx. The skilled person having benefit from the present disclosure will appreciate that the second set of pixels 314 may be temporally synchronized with laser point projector 220-Tx. Again, processing circuitry 230 may be used to compute depth information based on triangulating illumination events of the laser point projector 220-Tx and detected events 324 of the second set of pixels 314. Thus, instead of the two EVS 110 and 220-Rx in Fig. 2, a single EVS sensor 310 with hybrid IR-coated pixels 314 and uncoated pixels 312 or differently biased pixels (this can have separate readouts for the two event data streams) may be used. While Fig. 3 A shows a schematic top view of EVS 310, Fig. 3B illustrates a sectional view of EVS 310 comprising an upper layer implementing photodiodes of the respective pixels 312, 314 and a lower layer implementing respective read-out-circuitry on a common semiconductor substrate.

Active pixels may be defined as pixels of EVS 110 that correspond to one or more moving objects 112. Assuming brightness constancy, events observed by EVS 110 correspond to edges moving either due to ego-motion or object motion. The distinction between these two types of events (ego-motion events and object motion events) may be simple in the case of a stationary EVS since all the events are caused due to object motion. On the other hand, differentiating these two motion types may not be as simple in the case of moving setup and may typically be solved using optimization or by assuming only rotational motion using Inertial Measurement Unit (IMU) priors. Since the proposed application aims for low latency, all events (ego-motion and object motion) may be considered as equal contributors and it may not be necessary to distinguish between them. Optionally, fined tuned faster algorithms for ego motion segmentation may be embedded onto ASICs or FPGAs to reduce latency overhead. Active pixels define the area 112 of dense scanning for depth sensor 120. By sampling a small area of pixels, the proposed system can lead to a low-power and high framerate depth sensor.

Depth information 122 may be estimated by triangulating pixels from the event-plane and the projector-plane. For triangulation, the geometry of the measurement setup 200 has to be known. Data association may be achieved by using corresponding event timestamps from illumination source 220-Tx and EVS 220-Rx or 310. Assuming EVS 220-Rx or 310 is synchronized with projector 220-Tx, events due to the projector 220-Tx may be represented as a time surface. An example time surface is shown in Fig. 4 illustrating exemplary event time surfaces with sparse sampling (a) and the proposed adaptive foveating depth (b) within a rectangular bounding box 410 surrounding an area of motion in a scene.

The point laser projector 220-Tx may be modeled as an inverse EVS (event camera), allowing to represent the projector illumination as a time surface and simplifying the structured light (SL) system as a special case of stereo event camera. Assuming a calibrated system of EVS and projector, the two time surfaces can be rectified, and depth information 122 can be computed by triangulating illumination events from the projector 220-Tx and regular events from EVS 220-Rx or 310. The projector 220-Tx traverses its pixels precise time in the scanning period, allowing to define a time surface over the projector’s pixel grid. Due to this similarity with event time surface and inverse camera behavior (where camera perceives light, a projector emits light), one may think of the projector as an “inverse” event camera.

When conservatively sampling a scene, a natural question to ask is: ’’Can the complete depth profile be recovered from sparse depth measurements?” Several works address the problem of full scene depth estimation from sparse samples. Recently, the depth completion problem was formulated with compressive sensing literature and provided exactness and stability analysis for 3D depth completion. Proposition 6 of Fangchang Ma, Luca Carlone, Ulas Ayaz, and Sertac Karaman, “Sparse depth sensing for resource-constrained robots”, showed that exact recovery of depth from sparse set of samples is possible if the sparse sample set covers the edges and their adjacent pixels. Only sampling the edges satisfies algebraic condition for depth recovery but does not guarantee recovery of the true depth profile. To ensure full recovery, also adjacent pixels may be sampled. Since events naturally correspond to edges, an event-based sampling method satisfies the algebraic condition for depth recovery. In practice however, event cameras can be noisy and therefore representation of events over a duration of time is commonly used in robotics and vision. An event-image representation followed by median filter is proposed to reduce the influence of noise. A bounding box may be applied around events to account for alignment errors. The pixels within the bounding box may be sampled densely whereas other areas follow sparse sampling model. Fig. 4 shows the sampling strategy described here. Thus, processing circuitry 230 may be configured to determine a respective rectangular bounding box 410 around the detected one or more areas of motion. The depth detector 120, 220 may be configured to scan areas within a respective bounding box 410 with a higher spatial resolution than outside the bounding box 410. Fig- 5 summarizes the proposed concept by illustrating a flowchart of a method 500 for event- guided depth estimation.

Method 500 includes an act 502 of detecting one or more areas of motion 112 in a scene 114 with an EVS 110 and an act 504 of determining depth information for the detected one or more areas of motion 112 with a higher spatial resolution than for a remainder of the scene.

Events are produced by EVS 110 tuned to detect temporal changes in the scene brightness. These events effectively locate moving objects in a scene, which are associated with regions of interest. A pulsed IR laser projector 220-Tx synchronized with a second EVS sensor 220- Rx or pixel looking at the same scene is then activated to compute the depth of the moving objects with high-density (non-redundant information), in order to save power and increase speed, while only very sparsely sampling the depth of the rest of the scene.

In more detail, the algorithm may work as illustrated in Fig. 6.

1. An initial full scan of the scene depth may be performed with the calibrated pair of lasersensitive EVS sensor 220-Rx + pulsed IR laser 220-Tx. A threshold of such a EVS sensor 220-Rx can be made high enough that it is only sensitive to the pulsed-laser 220-Tx and not the scene motion.

2. The data from the tuned EVS sensor 110 relating only to scene motion may be fed to the DSP 230 which can estimate one or more regions of event data spatial and temporal density, according to a selectable threshold.

3. The correlation of the data can be recognized thanks to an embedded background activity filter and basic clustering algorithm of the literature.

4. The projector spatial sampling may be increased at the location of detected motion thanks to the control signals from the DSP 230, while the rest 116 of the scene 114 is only very sparsely sampled, if at all.

5. Depth may only computed at the areas of enough events by the same DSP 230.

The proposed concept of event-based depth sampling may be used for down-stream task such as depth completion aiming to recover dense depth maps from sparse depth measurements. Depth completion task is a sub-problem of depth estimation. In the sparse-to-dense depth completion problem, one wants to infer the dense depth map of a 3D scene given an RGB image and its corresponding sparse reconstruction in the form of a sparse depth map obtained either from a depth detector. With the advances of deep learning methods, many depth completion approaches based on convolutional neural networks (CNNs) have been proposed. The mainstream of these methods is to directly input the sparse depth maps (with/without color images) into an encoder-decoder network and predict dense depth maps.

In the following, a neural network-based approach for depth completion is described. The general network architecture is inspired from U-Net which was used previously used for monocular depth estimation with events and combining events with images. Similar to Daniel Gehrig, Michelle Ruegg, Mathias Gehrig, Javier Hidalgo-Carrio, and Davide Scaramuzza, “Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction,” IEEE Robotic and Automation Letters, the proposed architecture uses skip connections and residual block.

Depth representation: The metric depth D_m may be first converted into a normalized log depth map £> G [0, 1] facilitating large depth variation learning. Logarithmic depth may be computed according to: where, D_max = 1000 m and a = 5.7.

Event representation: Events may be converted to fixed-size tensor to utilize existing convolutional neural network architectures. Events in a time window AT may be drawn into a voxel grid with spatial dimensions H x W and B temporal bins. The number of bins B may be set to 5 for all experiments.

The proposed event-guided depth estimation is agnostic to depth estimation method (SL, ToF or LiDAR) and may be superior to dense sampling. It can be shown that in different scenarios like autonomous driving and indoor application, the active pixels correspond to 10% of the pixel area on average. Dense methods may have a very high event rates of 6.8MEv/s, whereas the sparse sampling may have a lowest event rate of 265kEv/s. The event rate for the proposed event-guided depth is data dependent and for the wall scene this varies between 600kEv/s and 5MEv/s depending on the motion. It may be observed that with sparse sampling, a lower reconstruction error may be achieved because the event timestamps are accurate to produce accurate depth maps. One the other hand, dense sampling has the highest reconstruction error due to the noise in the event timestamps present at such high event rates. The proposed concept strikes a balance between these two approaches and achieves a low reconstruction error and a low event-rate rate.

The present disclosure presents a bio-inspired event-guided depth estimation method. Inspired from the human ability to focus with higher resolution on areas of interest and decrease the resolution for background, event-camera based depth sampling is proposed as event naturally encode relative motion of the scene. It may be shown that in natural scenes like autonomous driving and indoor environments moving edges correspond to less than 10% of the scene. Thus, the proposed setup requires the sensor to scan only 10% of the scene. This would equate to almost 90% less power consumption by the illumination source and a significant gain in the framerate of the depth sensor.

Note that the present technology can also be configured as described below.

Example 1 is an apparatus for event-based depth measurement. The apparatus comprising an event-based vision sensor configured to detect one or more areas of motion in a scene, and a depth detector configured to determine depth information for the detected one or more areas of motion with a higher spatial resolution than for a remainder of the scene.

In Example 2, the apparatus of Example 1 optionally further comprises processing circuitry configured to, based on information on the detected one or more areas of motion in the scene, control an illumination source of the depth detector to illuminate the detected one or more areas of motion with a higher spatial resolution.

In Example 3, the apparatus of Example 1 or 2 optionally further comprises an optical splitter configured to provide the same field of view of the scene for the event-based vision sensor and the depth detector. In Example 4, the apparatus of any one of Examples 1 to 3 further optionally comprises processing circuitry configured to compute depth information based on sensor output signals of the depth detector.

In Example 5, the depth detector of any one of Examples 1 to 4 further optionally comprises a laser point projector configured to scan the detected one or more areas of motion with the higher spatial resolution than the remainder of the scene.

In Example 6, the depth detector of any one of Examples 1 to 5 further optionally comprises a second event-based vision sensor sensitive to a wavelength and pulse frequency of an illumination source of the depth detector. The apparatus further comprises processing circuitry configured to compute depth information based on triangulating illumination events of the illumination source and detected events of the second event-based vision sensor.

In Example 7, the event-based vision sensor of any one of Examples 1 to 5 further optionally comprises a first set of pixels configured to detect the one or more areas of motion and a second set of pixels sensitive to a wavelength and pulse frequency of an illumination source of the depth detector. The apparatus further comprises processing circuitry configured to compute depth information based on triangulating illumination events of the illumination source and detected events of the second set of pixels.

In Example 8, the apparatus of any one of Examples 1 to 7 further optionally comprises processing circuitry configured to determine a respective rectangular bounding box around the detected one or more areas of motion, and wherein the depth detector is configured to scan areas within a respective bounding box with a higher spatial resolution than outside the bounding box.

In Example 9, the depth detector of any one of Examples 1 to 8 is further optionally configured to perform an initial scan of the scene with full spatial resolution and to adapt the spatial resolution for subsequent scans based on the detected one or more areas of motion in the scene.

Example 10 is a method for event-based depth measurement, the method comprising detecting one or more areas of motion in a scene with an event-based vision sensor, and determining depth information for the detected one or more areas of motion with a higher spatial resolution than for a remainder of the scene.

The aspects and features described in relation to a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the further example.

Examples may further be or relate to a (computer) program including a program code to execute one or more of the above methods when the program is executed on a computer, processor or other programmable hardware component. Thus, steps, operations or processes of different ones of the methods described above may also be executed by programmed computers, processors or other programmable hardware components. Examples may also cover program storage devices, such as digital data storage media, which are machine-, processor- or computer-readable and encode and/or contain machine-executable, processor-executable or computer-executable programs and instructions. Program storage devices may include or be digital storage devices, magnetic storage media such as magnetic disks and magnetic tapes, hard disk drives, or optically readable digital data storage media, for example. Other examples may also include local computer devices (e.g. personal computer, laptop, tablet computer or mobile phone) with one or more processors and one or more storage devices or may be a distributed computer system (e.g. a cloud computing system with one or more processors and one or more storage devices distributed at various locations, for example, at a local client and/or one or more remote server farms and/or data centers). The computer system may comprise any circuit or combination of circuits. In one embodiment, the computer system may include one or more processors which can be of any type. As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), for example, of a microscope or a microscope component (e.g. camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system may be a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as, for example, one or more circuits (such as a communication circuit) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system may include one or more storage devices, which may include one or more memory elements suitable to the particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and/or one or more drives that handle removable media such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like. The computer system may also include a display device, one or more speakers, and a keyboard and/or controller, which can include a mouse, trackball, touch screen, voice-recognition device, or any other device that permits a system user to input information into and receive information from the computer system.

Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a non- transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier. In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the present invention is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary. A further embodiment of the present invention is an apparatus as described herein comprising a processor and the storage medium.

A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

A further embodiment comprises a processing means, for example, a computer or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus. It is further understood that the disclosure of several steps, processes, operations or functions disclosed in the description or claims shall not be construed to imply that these operations are necessarily dependent on the order described, unless explicitly stated in the individual case or necessary for technical reasons. Therefore, the previous description does not limit the execution of several steps or functions to a certain order. Furthermore, in further examples, a single step, function, process or operation may include and/or be broken up into several sub-steps, - functions, -processes or -operations.

If some aspects have been described in relation to a device or system, these aspects should also be understood as a description of the corresponding method. For example, a block, device or functional aspect of the device or system may correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described in relation to a method shall also be understood as a description of a corresponding block, a corresponding element, a property or a functional feature of a corresponding device or a corresponding system.

The following claims are hereby incorporated in the detailed description, wherein each claim may stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent or independent claim. Such combinations are hereby explicitly proposed, unless it is stated in the individual case that a particular combination is not intended. Furthermore, features of a claim should also be included for any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

Claims

22 Claims

1. Apparatus for event-based depth measurement, the apparatus comprising an event-based vision sensor configured to detect one or more areas of motion in a scene; and a depth detector configured to determine depth information for the detected one or more areas of motion with a higher spatial resolution than for a remainder of the scene.

2. The apparatus of claim 1, further comprising processing circuitry configured to, based on information on the detected one or more areas of motion in the scene, control an illumination source of the depth detector to illuminate the detected one or more areas of motion with a higher spatial resolution.

3. The apparatus of claim 1, further comprising an optical splitter configured to provide the same field of view of the scene for the event-based vision sensor and the depth detector.

4. The apparatus of claim 1 , further comprising processing circuitry configured to compute depth information based on sensor output signals of the depth detector.

5. The apparatus of claim 1, wherein the depth detector comprises a laser point projector configured to scan the detected one or more areas of motion with the higher spatial resolution than the remainder of the scene.

6. The apparatus of claim 1, wherein the depth detector comprises a second event-based vision sensor sensitive to a wavelength and pulse frequency of an illumination source of the depth detector, the apparatus further comprising processing circuitry configured to compute depth information based on triangulating illumination events of the illumination source and detected events of the second event-based vision sensor.

7. The apparatus of claim 1, wherein the event-based vision sensor comprises a first set of pixels configured to detect the one or more areas of motion and a second set of pixels sensitive to a wavelength and pulse frequency of an illumination source of the depth detector, the apparatus further comprising processing circuitry configured to compute depth information based on triangulating illumination events of the illumination source and detected events of the second set of pixels.

8. The apparatus of claim 1, further comprising processing circuitry configured to determine a respective rectangular bounding box around the detected one or more areas of motion, and wherein the depth detector is configured to scan areas within a respective bounding box with a higher spatial resolution than outside the bounding box.

9. The apparatus of claim 1, wherein the depth detector is configured to perform an initial scan of the scene with full spatial resolution and to adapt the spatial resolution for subsequent scans based on the detected one or more areas of motion in the scene.

10. Method for event-based depth measurement, the method comprising detecting one or more areas of motion in a scene with an event-based vision sensor; determining depth information for the detected one or more areas of motion with a higher spatial resolution than for a remainder of the scene.