EP4314704A1

EP4314704A1 - Depth sensor device and method for operating a depth sensor device

Info

Publication number: EP4314704A1
Application number: EP22708131.2A
Authority: EP
Inventors: Michael Gassner
Original assignee: Sony Advanced Visual Sensing AG; Sony Semiconductor Solutions Corp
Current assignee: Sony Advanced Visual Sensing AG; Sony Semiconductor Solutions Corp
Priority date: 2021-03-29
Filing date: 2022-02-25
Publication date: 2024-02-07
Also published as: WO2022207201A1

Abstract

A depth sensor device (100) for measuring a depth map of an object (200) comprises a projector unit (110) configured to illuminate the object (200) with a series of illumination patterns, a receiver unit (120) comprising a plurality of pixels, the receiver unit (120) being configured to detect on a pixel basis intensities of light reflected from the object (200) while it is illuminated with the illumination patterns, and a control unit (130) configured to determine by triangulation a distance of a point (210) on the object (200) to a single pixel, which point (210) lies on the line of sight of the single pixel, by determining from intensity information obtained from the single pixel for the series of illumination patterns a direction of a pointing line running from the projector unit (110) to said point (210).

Description

DEPTH SENSOR DEVICE AND METHOD FOR OPERATING A DEPTH SENSOR DEVICE

FIELD OF THE INVENTION

The present disclosure relates to a depth sensor device and a method for operating a depth sensor device. In particular, the present disclosure is related to the estimation of distances by using structured light.

BACKGROUD

In recent years techniques for automatic measuring of distances by sending and receiving light have drawn much attention to them. One such technique is the usage of structured light, i.e. the illumination of an object with static or time varying light patterns such as line or bar patterns or checkerboard patterns. For a known orientation of light source and camera it is possible to determine the shape and the distance of an object from triangulation based on the known positions of the light source, the camera, the orientation of the emitted light in space, and the position of the according light signal on the camera.

It is desirable to improve the resolution and the accuracy of the depth maps obtained from triangulation.

SUMMARY OF THE INVENTION

In conventional systems for depth estimation a set of illumination patterns providing high intensities at predetermined solid angles is sent out to an object and the distribution of light reflected from the object is measured by a receiver such as a camera. The task is then to find for the known solid angles of light emission, the solid angles of maximum light reception on the receiver. Due to the limited density of intensity changes in the illumination pattern and the limited pixel resolution, for the determination of the solid angle of maximum light reception a fit of the expected intensity distribution to the measured intensity values has to be made. While for objects with a uniformly reflecting surface such a fit provides an accurate estimate of the direction of maximal reflectance, and hence of the position of the object, this is most often not true for objects with non- uniform reflecting surfaces. Thus, for everyday objects accuracy of the conventional depth estimation techniques is limited.

The present disclosure mitigates these shortcomings of conventional depth estimation techniques.

To this end, a depth sensor device for measuring a depth map of an object is provided, which comprises a projector unit configured to illuminate the object with a series of illumination patterns and a receiver unit comprising a plurality of pixels, which is configured to detect on a pixel basis intensities of light reflected from the object while it is illuminated with the illumination patterns. The depth sensor device comprises further a control unit that is configured to determine by triangulation a distance of a point on the object to a single pixel, which point lies on the line of sight of the single pixel, by determining from intensity information obtained from the single pixel for the series of illumination patterns a direction of a pointing line running from the projector unit to said point. Further, a method for measuring a depth map of an object with a depth sensor device is provided, which depth sensor device comprises a projector unit and a receiver unit comprising a plurality of pixels, where the method comprises: illuminating, by the projector, the object with a series of illumination patterns; detecting, by the receiver unit, on a pixel basis intensities of light reflected from the object while it is illuminated with the illumination patterns; and determining by triangulation a distance of a point on the object to a single pixel, which point lies on the line of sight of the single pixel, by determining from intensity information obtained from the single pixel for the series of illumination patterns a direction of a pointing line running from the projector unit to said point.

Instead of using the known solid angles of the emitted light signals to determine the unknown solid angle of incidence on the receiver, the above device and method go the other way round. They start from the known line of sight (or central bearing vector or central ray) of a single pixel. For this single pixel information about the received intensity is captured for a series of illumination patterns. This allows understanding changes in the received intensities on the single pixel that are based on the varying solid angles of light emission. From this information it is possible to determine the direction of a virtual light ray starting from the projector unit and hitting the object such that it is reflected along the line of sight onto the single pixel.

Thus, the solid angle or direction of a pointing line from the projector unit to the object for optimal reflection onto the receiver unit is determined. Since the baseline between projector unit and receiver unit can be easily determined, and since the line of sight of the pixel is known, knowledge of the direction of the pointing line allows triangulation of the (virtual) point of reflection on the object, and thus determination of the distance between object and projector unit and/or reception unit. Further, since the point of optimal reflection is determined, the accuracy of the method is the same for uniformly reflecting surfaces and non-uniformly reflecting surfaces.

This leads to an improvement of accuracy against the conventional methods that estimate the optimal angle of reception based on signal values of a plurality of pixels instead of estimating the optimal angle of emission based on a plurality of signal values of a single pixel.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a simplified block diagram of a depth sensor device;

Fig. 2 shows in a simplified manner an operation principle of the depth sensor device;

Figs. 3Ato 3C are further simplified illustrations regarding the operation principle of the depth sensor device;

Figs. 4A to 4C are simplified illustrations regarding an operation of a conventional depth sensor device;

Fig. 5 schematically shows a process flow of a depth sensing method; Fig. 6 schematically shows a process flow for generating a lookup table;

Fig. 7 schematically shows a process flow for obtaining intensity information;

Fig. 8 schematically shows another process flow for obtaining intensity information;

Fig. 9 schematically shows another process flow for obtaining intensity information;

Fig. 10 schematically shows another process flow for obtaining intensity information;

Figs. 11 A to llC show schematically different exemplary applications of a depth sensor device.

Fig. 12 is a block diagram depicting an example of a schematic configuration of a vehicle control system.

Fig. 13 is a diagram of assistance in explaining an example of installation positions of an outside- vehicle information detecting section and an imaging section of the vehicle control system of Fig. 12.

Fig. 1 is a schematic block diagram of a depth sensor device 100 that can be used to obtain a depth map on an object 200. The depth sensor device 100 comprises a projector unit 110, a receiver unit 120, and a control unit 130. Optionally, the depth sensor device 100 also comprises an output device 140 such as a display, a computer vision interface or a data output port for providing data for further image processing or is connected to such an output device i40.

The projector unit 110 comprises a light source and optics that allow the projector unit 110 to emit a series of different illumination patterns to an object 200. For example, the projector unit 110 might include a laser or a laser diode such as a vertical-cavity surface-emitting laser (VCSEL), the light of which is deflected by the optics, including e.g. a lens, a diffractive optical element, a micro-electro-mechanical mirror, or the like. In principle, the structure and layout of the projector 110 can be arbitrary as long it is capable to repeatedly emit illumination patterns having a stable and known light distribution, i.e. a stable and known intensity distribution as function of the solid angle. The used wavelength range may be arbitrary as long as the emitted light can be shaped by the optics and reflected from the object. For example, infrared light or visible light might be used.

The receiver unit 120 comprises a plurality of pixels 122 that are preferably arranged as a two-dimensional matrix, but that might also be arranged in any other pattern. Each of the pixels 122 is configured to detect intensity of incoming light that is used to generate intensity information. The intensity information may for example be a signal that indicates the detected intensity. In this case the receiver unit 120 may be a standard digital camera as known to a skilled person.

The intensity information may also depend on an event count carried out by the pixel 122. Here, an “event” refers to a change of intensity that is larger than a predetermined threshold, where an increase of intensity is referred to as positive polarity event, while a decrease is denoted as negative polarity event. The intensity information may then be the number of events of a given polarity during a predetermined time period or the duration after starting illumination with an illumination pattern that it takes for the first event of a specific polarity to occur. In this case the receiver unit 120 may be a dynamic vision sensor, DVS, or a dynamic and active pixel vision sensor, DAVIS. The intensity information can also be of combined type, i.e. refer to the detected intensity as well as to an event count. In this case the pixel 122 can be configured to produce both kinds of information. Alternatively, a combination of pixels of different type might be necessary to generate both types of information. In the latter, in referring to “a pixel”, if applicable, a combination of such an intensity sensitive pixel and an event sensitive pixel should also be included.

In order to combine the advantages of dynamic vision sensors, such as fast readout times, with the possibility to have the full intensity signal, the depth sensor device 100 may be capable to derive the intensity signal from a temporal sequence of events of a given polarity obtained from a single pixel by referring to a predetermined mapping of the temporal sequence of events to the intensity of light received at the single pixel. For example, by employing a pre-trained pixel model, obtained e.g. by machine learning techniques, or by employing an explicit model on pixel dynamics (e.g. optimization, filtering), a mapping between the temporal sequence of polarity events and the received light intensity can be obtained.

As the constructional details of such receiver units 120 are known, such as e.g. the specific circuitry used and the like, a detailed description can be omitted here. The receiver unit 120 can be of arbitrary construction as long as it allows producing intensity information by its pixels in response of observing illumination of the object 200 with the illumination patterns emitted by the projector unit 110.

Here, each of the pixels 122 of the receiver unit 120 receives light from a specific solid angle, e.g. via optics such as a lens, a diffractive optical element of the like. Thus, a line of sight can be assigned to each of the pixels 120 that corresponds e.g. to the central line of the solid angle observable by the pixel 120 or to the direction of maximal sensitivity. These lines of sights or central bearing vectors of the fields of view of the pixels are factory dependent and can be determined by calibration. How to define for a given receiver unit 120 the central bearing vector of each pixel 122 within the field of view of the respective pixel 122 is arbitrary, as long as there is only one such central bearing vector per pixel 122.

In the depth sensor 100 the projector unit 110 and the receiver unit 120 may have a fixed spatial orientation to each other that is known to the control unit 130, i.e. at least a baseline connecting the projector unit 110 and the receiver unit 120 is known. Moreover, the orientation of the solid angles of light emission and light reception of projector unit 110 and receiver unit 120, respectively, may also be known. For example projector unit 110 and receiver unit 120 may have fixed locations and orientations, e.g. within a single device. Additionally or alternatively, the projector unit 110 and the receiver unit 120 may be capable to determine their relative distance (and orientation) before and/or during each depth measurement, e.g. by a time of flight measurement or the like. In this case projector unit 110 and receiver unit 120 must not necessarily have fixed locations. In principle, it is also conceivable that there is movement between these units during a depth measurement, as long as the mutual distance (and orientation) can be detected and recorded during the measurement. As in conventional depth sensors also the depth sensor device 100 detects the distance of the projector unit 110 and/or the receiver unit 120 from the object 200 by triangulation. That is, a triangle is searched that connects the projector unit 110, the receiver unit 120 and the object 200 along the light propagation path from the projector unit 110 via the object 200 to the receiver unit 120. Knowing the baseline it is necessary for the triangulation to know the angles of the triangle at the projector unit 110 and the receiver unit 120. From this knowledge the distance to the object 200 follows.

To carry out the according calculations the control unit 130 is present that receives from the projector unit 110 and the receiver unit 120 information about emission and reception of light in synchronized form. The control unit 130 may comprise a processor 132 and a memory 134 to calculate and store all necessary information for the triangulation and the depth detection that is based thereon. The control unit 130 may be for example a (also external) computer, a chip, a processor or the like. The control unit 130 may also be a distributed computation system such as a network or a cloud. The control unit 130 may be implemented as hardware, as software or as a mixture thereof. Implementation of the control unit 130 is arbitrary as long as it is capable to carry out the functions described below.

Conventional depth sensors operate according to the principle that the known location of high intensity areas of the emitted illumination patterns is used as one input parameter of triangulation. Knowing the baseline it is then tried to deduce the angle of incidence onto the receiver that corresponds to a high intensity area. Having the angle of light emission and the baseline as input parameters, the triangle can be closed by determining the angle of incidence on the receiver.

In contrast to this conventional method, with the depth sensor device 100 of Fig. 1 it is possible to use as input information the angle of incidence of light onto the receiver unit 120 that comes from a specific point 210 on the object 200. Then, the emission angle that leads to a light ray from the projector unit 110 to this point 210 is deduced. Accordingly, the depth sensor device 100 differs fundamentally from the conventional depth sensor, in that it uses receiver unit side information as input to determine a projector unit side angle.

In particular, for a single pixel 122a of the receiver unit 120 intensity information is obtained from the intensities of light reflected from the object 200 for a series of illumination patterns. By using the knowledge about the illumination patterns and the knowledge about the line of sight of the single pixel 122a, it can be deduced which intensity information has to be expected for a specific distribution of light on the point 210 on the object 200 that lies on the line of sight of the single pixel 122a. By comparing the intensity information actually obtained by the single pixel 122a for different illumination patterns, one can then determine which part of the illumination patterns must have hit the point 210, which means that one can determine a direction of a pointing line that runs from the projector unit 110 to said point 210.

Thus, it is possible to deduce the solid angle under which the pointing line leaves the projector unit 110 to hit the point 210 on the object 200 from which light is reflected to ran along the line of sight onto the single pixel 122a. Based on this information it is then possible for the control unit 130 to calculate the distance of this point 210 from the receiver unit 120 (and/or the projector unit 110) by using triangulation. This principle of operation will be further explained with respect to Figs. 2 and for a non-limiting example with respect to Figs. 3A to 3C.

Fig. 2 shows exemplary the arrangement of the projector unit 110, the object 200 and the pixels 122 of the receiver unit 120.The projector unit 110 emits a plurality of illumination patterns that can in principle be arbitrary as indicated by the differing shapes of the patterns of Fig. 2. Each of the illumination patterns has a predetermined projection magnitude function that indicates its distribution of light intensity in space. This means, for each illumination pattern it is known how much intensity is emitted under which solid angle.

On the receiver unit side the intensity information obtained from a single pixel 122a is schematically illustrated by different grayscale values. For the first illumination pattern there is e.g. a low intensity obtained by the single pixel 122a, indicated by a black pixel, while the second pattern produced high intensity (white pixel) and the third an intermediate value (grey pixel). On the other hand, it is possible to estimate how much light is reflected from the object 200 along the line of sight of the single pixel 122a for a specific illumination pattern and a specific location of the point 210 on the object 200 surface along this line of sight. By using a series of illumination patterns this estimate can be improved until an accurate determination of the location of the point 210 on the line of sight can be made that fits all intensity information observed by the single pixel 122a. In this manner the distance of this point 210 can be determined. By reiterating the process for all pixels 122 of the receiver unit 120, a full depth map can be established, i.e. a distance value can be ascribed to each of the pixels 122.

This process can be implemented for example by establishing a mathematical model that maps the predetermined projection magnitude functions of the illumination patterns into specific intensity information, like e.g. intensity values, event counts or the like. By using these models a lookup table can be established by mapping a series of detected intensity information to values of the predetermined projection magnitude functions obtained for a specific solid angle of the pointing line, which allows then a mapping to one of these specific solid angles for a specific observation. The lookup table in form of the mapping from predetermined projection magnitude functions to a specific solid angle can be generated before the actual measurement, since it does not rely on the measured parameters. This means that during the measurement the depth sensing does not require too much processing power, since it is only necessary to retrieve the entry of the lookup table for a measurement series. The processing power needed is therefore not larger than the one necessary in conventional triangulation schemes.

To exemplify the above further, the following description will refer to the specific case of light sheets (observed as light lines on the object) as illumination patterns, i.e. of a light beam extending in one direction perpendicular to its propagation direction and generating therefore a light plane or sheet in space. It has to be noted that this serves only for the demonstration of the working principle of the depth sensor device 100 and does not imply any limitation to such line shaped patterns. The methods described below will also work analogously for different illumination patterns.

Fig. 3 A shows a setup in which projector unit 110 emits a series of light sheets or light lines l_k (k = 1, ... , N) to the object 200. Each of the light sheets l_k is emitted under a projection angle p^max _k which is defined on a plane including the projector unit 110 and the receiver unit 120 (here the drawing plane) and which angle is measured on the plane starting from a reference direction r. Illustrated with long dashed lines is a light sheet l_mj, its two adjacent light sheets l_mj_i, l_mj+i, and their reflections on surface elements Sj_i, Sj, Sj+i of the object 200. These light sheets have predetermined projection magnitude functions Pk(p) indicated in Fig. 3B. As shown, each of the light sheets l_k has a peak at the corresponding projection angle p"^la and has e.g. a Gaussian profde over the angle p.

On the receiver unit side a line of pixels 122 is shown that lies in the drawing plane. Here, the pixels are denoted by Ui (i = 1, ..., M). In the following as the single pixel 122a the pixel U_j will be used.

In Fig. 3C the distribution of intensities over the pixels for the different light sheets mj-1, mj, and mj+1 is shown by the curves R_k. Here, although the full curves are shown, it has to be noted that only the values at the pixels represented by dots on the curves can be observed. It is a general observation schematically indicated in Figs. 3B and 3C that due to non-uniform reflectance on the object 200 the well behaved shape of the light sheet intensities represented by the predetermined projection magnitude functions P_k(p) will be distorted to shapes as shown in Fig. 3C. This makes it genuinely difficult to obtain a good estimate of an incidence angle fitting the observed intensities R_k.

This will be briefly explained with respect to Figs. 4A to 4C. What is tried in such a conventional setup is to find out a matching incidence angle 6, for the emission of a single light sheet (. This single light sheet produces an intensity response in a continuous group of pixels 122. If the object has a sufficiently uniform reflectance, the received intensities will be well behaved as shown in Fig. 4B, which allows finding the correct intensity peak by a Gaussian fit G to the observed intensities. For non-uniform reflectance, the situation corresponds to Fig. 4C. Here, the true intensity curve is shown as the light dashed curve. The Gaussian fit G produces a curve having a maximum that is laterally shifted away from the true maximum. Nevertheless the maximum of the Gaussian fit will be set as angle of incidence 6"^laj and will be used for triangulation, where it leads to wrong estimation of the distance to the object. In this manner non-uniform reflectance deteriorates the measurement accuracy of conventional depth sensors.

Therefore, the depth sensor device 100 does not try to determine an angle of incidence onto the receiver unit 120. In contrast, for the pixel U_j the central bearing vector V_j (i.e. the line of sight) is set as one side of the triangulation, and it is asked, which virtual light sheet would have produced a reflection along this central bearing vector V_j. In this manner a pointing line p is determined that runs from the projector unit 110 to the point 210 on the object that lies on the central bearing vector V_j. In terms of light sheets or light lines this can be formulated such that the virtual projection angle p_j is found that indicates the angle from the reference direction r under which a virtual light sheet would intersect the drawing plane of Fig. 3B along the pointing line p.

This can be achieved by referring to the intensity information R_k obtained at a single pixel U_j for different light sheets l_k (which will be observed as light lines on the object 200). In particular, it has been found that it is sufficient for the determination of the virtual projection angle p_j to refer to intensity information R_k of the light sheet l_mj that passes the closest by the point 210 on the object 200, meaning that generates a reflection that is closest to the central bearing vector Vj, and to at least one of its neighboring light sheets l_mj_i, l_mj+i. Here, the closest or main light sheet is indicated with the index “k=mj” in Figs. 3 A to 3C. Which of the light sheets l_k is the main light sheet l_mj can be deduced by comparing the different intensity information obtained by the single pixel U_j for all light sheets l_k. Based on a mathematical model connecting the predetermined projection magnitude functions P_k(p) and the intensity information R_k|j | it is then possible to deduce the virtual projection angle p_j.

This will be discussed in the following for three examples. First, as intensity information the measured intensity will be used. Second, as intensity information an event count will be used. And third, as intensity information a delay time until event detection will be used. These examples serve only for better understanding. In principle, also other manners of detecting the virtual projection angle p_j can be imagined.

First, the intensity information R_k|j | obtained for the k-th light sheet or line at pixel U_j will be equated to the intensity I[j] at the pixel U_j. This intensity can be modeled based on the predetermined projection magnitude function P_k(p) in the following manner:

RkO] = a(j) · Pk(Pj) + bG), with a ) representing signal attenuation for the reflecting surface element (taking into account the distance and the surface reflectance properties), b ) background light at the surface element and p_j the virtual projection angle, where integrals over the pixel surface have been omitted for readability.

The main light sheet l_mj is obtained by searching for the largest value of R_k[j] = I[j], i.e. for the largest intensity that is obtained for all the different light sheets: m_; = argma x(R_k\j]) k

Based on the mathematical model it is possible to establish the following equation:

Here, on the left hand side of the equation only measurement value are input, while on the right hand side only the predetermined projection magnitude functions P_k(p_j) are used. In particular, the response ratio of intensities obtained at pixel U_j for the main light sheet l_mj and the two neighboring light sheets l_mj_i, l_mj+i is equal to the respective ratio of the predetermined projection magnitude functions of these light sheets at the virtual projection angle p_j. This value is denoted as _mj(p_j) above, i.e. it is a scalar value obtainable for a specific light sheet (or light line) and a specific virtual projection angle p_j. The quantity /._k(p) can be calculated in advance for different values of k, i.e. for different light sheets l_k, by inserting different values for the projection angle p into the respective predetermined projection magnitude functions. In this manner a lookup table can be created including a mapping of _k(p) to a value of the projection angle p for each k. The virtual projection angle p_j for pixel U_j can then be obtained by finding the main light sheet l_mj from all obtained intensity values, by calculating the response ratio _mj(pj), and by searching for the corresponding angle in the lookup table for the main light sheet l_mj.

Based on the virtual projection angle p_j, the central bearing vector V_j, and the baseline between projector unit 110 and receiver unit 120 the distance from the depth sensor 100 to the point 210 on the object 200 can be determined by triangulation, i.e. the distance from pixel U_j to the point 210 on the object 200 along the central bearing vector V_j can be calculated and set as a depth value of pixel U_j. By reiterating the process for all pixels (either serially or at least partially in parallel) a complete depth map can be established that shows the distances of different surface elements of the object 200 to the depth sensor 100.

It should be noted that one can supplement or replace the above method by a calculation of a different response ratio, as e.g.

In this manner one can e.g. build a control value or ensure that at least one well behaved response ratio is present, if the differences of the predetermined projection magnitude functions in the denominators in the response ratios tend to zero.

A simplification of the above can be achieved, if the receiver unit 120 is capable to detect only differences in illumination, i.e. if the receiver unit 120 is capable to eliminate background light from the measured intensities. This is e.g. the case for an event sensor with a linear frontend. In this case the term b(j) in the model for the intensity R_k[j] is not present. Therefore, a meaningful response ratio can be defined with reference to only one of the two neighboring light sheets of the main light sheet l_mj. For example, one can use the response ratio in this case. Again, it is then possible to deduce the virtual projection angle p_j from a previously computed lookup table.

In a second example the intensity information can be equated to the number of events of a given polarity during illumination with a light sheet pulse. For example, for a logarithmic response to the incoming light signal, i.e. for a log frontend, one can set the intensity information R_j[k] of pixel U_j obtained for light sheet l_k to the number of polarity events. This quantity can be modelled in the following manner:

So, also in this case it is possible to obtain a map from the observed intensity information R_k[j] at pixel U_j for all light sheets l_k to the virtual projection angle p_j, e.g. by using a previously calculated lookup table stored by the control unit 130.

Another manner for obtaining the virtual projection angle p_j is to use the delay of event generation after emission of the light sheet as the intensity information R_k| j | . This delay can for example be modelled as where a(j), b(j), and c(j) are as defined above. The constant K and the photocurrent in function of the background light I_photo(b(j)) are predetermined for each receiver unit pixel 122 and can be determined in calibration. Using the three delay times for the main light sheet and its two adjacent sheets as left hand side inputs R_ni||j |. R_mj_i[j], R_nl|-i|j| one obtains three equations with the unknowns a(j), b(j), and, via the known expressions of P_k(p), P_j· By solving these equations, e.g. by applying a numerical solver in the control unit 130, for p_j the virtual projection angle can be obtained. Again, it is also possible to create a lookup table in advance, which will in this case have three input entries for R_mj[j], R_mj-i [j], and R_mj+i|jL by solving the equations in advance for different values of RmjljL Rmj-_l[jL and Rmj_+l[j]·

Then, just as explained before it is possible to retrieve the distance of the pixel U_j to the object 200 and to generate a depth map of the object 200.

The above examples of intensity information can of course be generalized to any useful set of information that allows retrieving the virtual projection angle pj from the information captured by a single pixel Uj. The method can then be summarized as shown in Fig. 5. At SI 10 a set L of light sheets or lines being countable in consecutive order by index k = 1, N are emitted from the projector unit 110 to the objection 200. Further, the intensity information R_k[j] is obtained for all pixels U_j with j = 1, M. Here, the lines may be serially projected in any order. The index k does only indicate the spahal ordering, i.e. k increases with the projection angle of the line, but not the temporal order that can be arbitrary. In particular, each of the lines may be projected only once during establishing the depth map of the object in order to save power necessary for line emission.

Moreover, to speed up the process multiple lines may be projected simultaneously instead of one line after the other. Also here each line might be projected only once. When projecting mulhple lines simultaneously lines can be identified on the receiver side from their spatio-temporally distinct neighborhood, i.e. by taking into account the distinct temporal response sequence left and right of a line to uniquely identifying the line. The projector pattern that allows doing this can either be handcrafted or computer generated.

Alternatively or additionally, each of the pixels 122 may obtain a temporal sequence of events of a given polarity during an exposure period of the receiver unit 120. It is then possible for the control unit 130 to determine from the temporal sequence of events obtained at one pixel 122 which amount of intensity has been received at the respechve pixels 122 from which light sheet.

As S120 reiterahon of the process of determining the virtual projechon angle p_j for each pixel U_j is performed. It should be noted that this could be performed frame-wise, i.e. after intensity information of all pixels 122 is obtained or intermittently on a single pixel or pixel group basis. Differently stated, the temporal order of virtual projection angle determination and light line emission is arbitrary as long as it allows the determination to be based on a sufficient amount of light lines for the pixel 122 at hand.

As long as not all pixels 122 with coordinate U_j are processed at S130 the main line or light sheet with index k=mj is determined, i.e. the light line passing closest to the point 210 on the object 200 through which the line of sight (or central bearing vector) of the pixel U_j runs. If the intensity or the event count is used as intensity information, this will be the line that causes the largest response. If the delay time is used, it will be the line with the shortest response time.

Here, as an option it can be checked at S 132 whether the intensity information R_mj[k] obtained for the main line is above a predetermined noise threshold. Only if this is the case, the pixel U_j is used for depth map determination. If not, the pixel is invalidated at S 134.

At SI 40 the response ratio applicable for the used intensity information is calculated to look up the virtual projection angle p_j for the calculated response ratio at S150. Of course, if the calculated response ratio lies between values explicittly noted in the used lookup table, the correct virtual projection angle p_j can be obtained by standard interpolation techniques such as e.g. linear interpolation.

At S 160 the virtual projection angle p_j and the central bearing vector V_j that is known from factory calibration are used together with the baselined between projector unit 110 and receiver unit 120 to triangulate the depth value for the pixel U_j at hand. The value is stored by the control unit 130 at pixel coordinate U_j to generate the complete depth map. Afterwards, the process turns to the next pixel U_j+i at S120.

A process for generating a lookup table is exemplary and schematically illustrated by the flow chart of Fig. 6.

At S210 the process is initialized by loading the necessary input parameter. These comprise the step size d between angles that will be included in the lookup table. Further, the projector calibration is loaded, which includes the projection angles p^ma under which the different light sheets/lines are emitted and their respective predetermined projection magnitude functions P_k(p), quantities known from factory calibration. Finally, the response ratio to be calculated must also be set, for example one of response ratios discussed above.

At S220 one line index is selected, where only those lines can be selected that have the neighboring lines necessary for calculating the response ratio. Thus, if for example the main line and its two adjacent lines are needed for the calculation, the first line k=l and the last line k=M cannot be selected.

At S230 the minimum angle of the lookup table for line k is set. For example, the minimum angle p_min can be set to the projection angle p^max _k_i of the line previous to the line k at stake. This is in particular useful if the line k-1 is needed for calculation of the response ratio. Also the maximum angle of the lookup table for line k is set. Here, the projection angle pⁿⁱⁱ'Vi of the line after the line k at stake can be used. The lookup table includes then the angular range between the peak intensities of the two lines (or light sheets) adjacent to the line at stake. In this angular range it is expected that a one-to-one mapping of projection angle to response ratio is possible, in particular for predetermined projection magnitude functions of Gaussian shape.

When at S240 the current angle p under consideration is smaller than the maximum angle p_nla . the response ratio for this angle p is calculated at S250 and is stored together with the angle p in the lookup table. The angle p is incremented by the step size d in S260 and the process returns to S240. When the maximum angle p_maxhas been reached the table may be optionally sorted at S270 according to the value of the response ratio. Afterwards a new line index is selected at S220 and the process reiterates until all lines have been processed. This allows an efficient calculation of a lookup table.

Here, if the predetermined projection magnitude functions P_k(p) are the same for all pixels 122, a single lookup table for all pixels 122 can be established. However, it might be that projection magnitude functions vary along the light sheet. In particular, for a light sheet extending towards the object and e.g. in the vertical direction, it has to be expected that the thickness and/or position of the light sheet varies along the vertical direction, e.g. due to distortions at the projector unit 110. Then different lookup tables will be needed for pixels seeing different parts of the light sheet, i.e. for pixels at a different height in the vertical direction. This problem might be mitigated by using predetermined projection magnitude functions that depend on the full solid angle. However, while then the same functions can be used for all the pixels 122, the necessary computational power might increase.

Figs. 7 to 10 illustrate schematically and exemplary different methods for obtaining the intensity information R_k[j]. They may therefore be understood as subroutines of S 110 of Fig. 5. Fig. 7 refers to setting the measured intensity I[j] at pixel position U_j as the intensity information R_k[j]. At S310 a line k is selected for emission. The index k indicates the consecutive spatial order of the lines (or light sheets) with increasing projection angle. Emission of the lines must of course not follow this order. The lines can be emitted in any temporal order. Lines may also be emitted at the same time as long as it is possible for the control unit to distinguish which intensity information has been received at which pixel from which line/light sheet, e.g. from the temporal and spatial order of the lines/light sheets.

At S320 frame exposure is started and at S330 the selected line is emitted for a given time duration D. At S340 the frame exposure is ended and the frame I is read out, i.e. all intensity values obtained by all pixels are read out.

At S350 one pixel U_j is chosen and the intensity information of this pixel R_k[j] is set to the corresponding intensity value of the frame at position U_j, I[j] When the intensity information has been assigned to all pixels 122 in this manner, the process returns to S310 to select a different line. Once all lines have been projected the process ends. In this manner a complete set of intensity information can be obtained for all pixels 122 and all light sheets/lines.

Fig. 8 refers to setting the detected number of events at pixel position U_j as the intensity information R_j[k] At S410 a line is selected as explained above with respect to S310 of Fig. 7. At S420 the line is projected and events of a given polarity (positive or negative) are counted during the projection duration D for all pixels 122. At S430 a pixel U_j is selected and the number of positive (or negative) polarity events that did occur during the projection duration is extracted at S440. This number is then stored as the intensity information R_k[j] of that pixel U_j for line l_k. After having treated all pixels 122 of the receiver unit 120 in this manner, the process returns to S410 for selection of a new line. Once all lines have been projected the process ends. In this manner a complete set of intensity information can be obtained for all pixels 122 and all light sheets/lines.

Fig. 9 refers to the case that the time delay between light sheet emission start and first event detection is used as intensity information R_k[j]· Here S510 to S530 correspond to S410 to S430 of Fig. 8 and need not to be described again. At S540 the first polarity event timestamp t_j at which the first polarity event has been measured after starting the projection of line l_k is extracted and stored as the intensity information R_k[j]· After all pixels 122 have been treated the process returns to S510. After all lines have been projected, the process ends.

Fig. 10 illustrates schematically a hybrid method that obtains as intensity information R_k[j] the intensity measured at each pixel U_j whose event count has crossed a predetermined threshold. Here, it has to be ensured that the overlap of the used lines is minimal at the receiver unit 120. Thus, all lines are projected by dividing them into sparser sets of lines. The combination of event count and detecting the intensity may be effected within a single pixel or by combining information of pixels of different sensor arrays.

At S610 it is checked whether all line have already been projected. If this is not the case at S620 a subset of lines that has not been projected is chosen. For example, the set of lines may be divided into three sets of lines containing each only every third line, and being shifted by one line with respect to each other. As S630 frame exposure and event count is started, and at S640 one of the lines within the selected subset of lines is projected for projection duration D_k starting at time instance t_k. At S650 frame exposure is stopped and the intensities of the corresponding frame I are readout.

Although the process of line subset selection is described here in the context of Fig. 10, such an approach can of course also be carried out in any of the methods described above.

Once the frame has been readout, at S660 one of the lines k that has been projected is selected. At S665 the pixel U_j for with the intensity information is to be set is selected. At S670 the events P_j of a given polarity that occurred during the time interval D_k at pixel U_j are extracted. At S675 this event number p_j is compared to a predetermined threshold (that may be different or the same for all pixels 122). If the threshold is crossed, the intensity of the pixel U_j at stake is set at S680 as the intensity information R_k[j]· If not, at S690 the intensity information is set to zero. Then all pixels are reiterated starting again at S665. Once all pixels are finished a new line of the subset of projected lines is chosen at S660 and the process is repeated. After that is finished the process returns to S610, where a new subset of lines is selected or where the process ends, if all lines have been projected.

Thus, in this manner also an event filtered intensity value of each pixel can be used as intensity information, which may allow speeding up the calculation by setting a part of the intensity information to zero.

As a further alternative or supplement of the above machine learning techniques can be used for magnitude estimation by learning (offline) a mapping from a number of polarity events and their respective temporal spacing to the pulse magnitude. Also machine learning can be used for angle estimation by learning (offline) a mapping from the intensity information at one pixel (including the number of polarity events and their respective temporal spacing caused by the light pulses of the main light sheet and the light sheets adjacent to it) to the virtual projection angle. A mapping can be learned for each light sheet taking into account changing line properties along horizontal and vertical directions. To generate the data used to fit the machine learning model one may employ a flat plane calibration target that sweeps through known distance values, e.g. on a rail system, which allows to generate receiver magnitude values for a multitude of projection angles. Furthermore the background light and projector unit power can be modulated to allow the machine learning model to learn a mapping robust to variation in background light and object reflectance. Machine learning models could be for instance a neural network or a regression on handcrafted features (such as event count, event delay, ratios and differences of the latter and the like).

As should have become clear from the above specific examples there are a lot of possibilities to deduce intensity information from the intensities detected by the pixels 122 of the receiver unit 120. Since there is a causal link between the used illumination pattern, the reflected intensities and the intensity information obtained therefrom, it is possible to generate for a wide plurality of cases a mathematical model that links the obtained intensity information with the emitted illumination patterns. This causal link allows it to use the intensity information of a single pixel 122a to deduce which parts of the illumination patterns within the series of illumination patterns have fallen on the point 210 on the object 200 that lies on the line of sight, i.e. on the central bearing vector, of this single pixel 122a. Since the illumination patterns and their propagation in space are known from calibration of the projector unit 110, this knowledge allows to deduce at what distance from the single pixel 122a said point 210 on the object 200 must be located. Thereby, the present depth sensing method allows a determination of the depths based on a series of illumination patterns and intensity information. The fact that for explanation of the working principle several examples have been chosen that referred to single light sheet emission must therefore not be considered as a limitation of the general working principle.

Moreover, since the present depth sensing method relies on the intensity information of a single pixel and the central bearing vector of that pixel, there is no risk that due to non-uniform reflection the estimation of the position of the corresponding surface element on the object 200 contains a lateral error as shown e.g. in Fig. 4C. In contrast, the effects of non-uniform reflection are fully taken into account by the modeling of the causal connection between the intensity distribution of the illumination pattern and the obtained intensity information. For example, in the above example effects of non-uniform reflection are part of the attenuation coefficient a(j). Therefore, the present depth sensing method allows achieving a higher accuracy in comparison with conventional methods.

Of course, the depth sensor device 100 and the method carried out by it can be used in any technical area that is in need of fast and accurate depth sensing. Some examples for uses are indicated in Figs. 11 A to 11C.

For example, as shown in Fig. 11 A the projector unit 110 and the receiver unit 120 may be integrated into smart glasses to allow hand and/or object tracking and mapping. This can in turn be used for interaction and dynamic occlusion in augmented reality, virtual reality, and mixed reality applications.

Another possible application is the integration of projector unit 110 and receiver unit 120 into a mobile terminal shown in Fig. 1 IB to allow mobile 3D mapping, 3D facial capture or 3D photography. Further, as shown in Fig. llC surveillance and security systems may use the depth sensor device 100 for face recognition.

In general, the present depth sensing methods may be applied to any field of computer vision as implemented e.g. in self driving car systems, surveillance systems or automatic product order systems. The depth sensing methods may also be used in image processing to generate e.g. a virtual blur for image parts at predetermined distances or to remove or adapt the background of an image, e.g. during a video conference.

Accordingly, the depth sensor device and the depth sensing method described above can be used in a broad variety of technical areas to provide more accurate depth sensing.

For example, the technology according to the present disclosure may be realized, e.g., as a device mounted in a mobile body of any type such as automobile, electric vehicle, hybrid electric vehicle, motorcycle, bicycle, personal mobility, airplane, drone, ship, or robot.

Fig. 12 is a block diagram depicting an example of schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology of the present disclosure can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. In the example depicted in Fig. 12, the vehicle control system 12000 includes a driving system control unit 12010, a body system control unit 12020, an outside-vehicle information detecting unit 12030, an in-vehicle information detecting unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound/image output section 12052, and a vehicle-mounted network interface (I/F) 12053 are illustrated as a functional configuration of the integrated control unit 12050.

The driving system control unit 12010 controls the operation of devices related to the driving system of the vehicle in accordance with various kinds of programs. For example, the driving system control unit 12010 functions as a control device for a driving force generating device for generating the driving force of the vehicle, such as an internal combustion engine, a driving motor, or the like, a driving force transmitting mechanism for transmitting the driving force to wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, and the like.

The body system control unit 12020 controls the operation of various kinds of devices provided to a vehicle body in accordance with various kinds of programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various kinds of lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or the like. In this case, radio waves transmitted from a mobile device as an alternative to a key or signals of various kinds of switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls a door lock device, the power window device, the lamps, or the like of the vehicle.

The outside-vehicle information detecting unit 12030 detects information about the outside of the vehicle including the vehicle control system 12000. For example, the outside-vehicle information detecting unit 12030 is connected with an imaging section 12031. The outside-vehicle information detecting unit 12030 makes the imaging section 12031 imaging an image of the outside of the vehicle, and receives the imaged image. On the basis of the received image, the outside-vehicle information detecting unit 12030 may perform processing of detecting an object such as a human, a vehicle, an obstacle, a sign, a character on a road surface, or the like, or processing of detecting a distance thereto.

The imaging section 12031 may be or may include a depth sensor device 100 as described above. The imaging section 12031 may output the electric signal as position information identifying pixels having detected an event. The light received by the imaging section 12031 may be visible light, or may be invisible light such as infrared rays or the like.

The in-vehicle information detecting unit 12040 detects information about the inside of the vehicle and may be or may include a solid-state imaging sensor with event detection and photoreceptor modules according to the present disclosure. The in-vehicle information detecting unit 12040 is, for example, connected with a driver state detecting section 12041 that detects the state of a driver. The driver state detecting section 12041, for example, includes a camera focused on the driver. On the basis of detection information input from the driver state detecting section 12041, the in-vehicle information detecting unit 12040 may calculate a degree of fatigue of the driver or a degree of concentration of the driver, or may determine whether the driver is dozing. The microcomputer 12051 can calculate a control target value for the driving force generating device, the steering mechanism, or the braking device on the basis of the information about the inside or outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040, and output a control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform cooperative control intended to implement functions of an advanced driver assistance system (ADAS) which functions include collision avoidance or shock mitigation for the vehicle, following driving based on a following distance, vehicle speed maintaining driving, a warning of collision of the vehicle, a warning of deviation of the vehicle from a lane, or the like.

In addition, the microcomputer 12051 can perform cooperative control intended for automatic driving, which makes the vehicle to travel autonomously without depending on the operation of the driver, or the like, by controlling the driving force generating device, the steering mechanism, the braking device, or the like on the basis of the information about the outside or inside of the vehicle which information is obtained by the outside- vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040.

In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 on the basis of the information about the outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030. For example, the microcomputer 12051 can perform cooperative control intended to prevent a glare by controlling the headlamp so as to change from a high beam to a low beam, for example, in accordance with the position of a preceding vehicle or an oncoming vehicle detected by the outside- vehicle information detecting unit 12030.

The sound/image output section 12052 transmits an output signal of at least one of a sound or an image to an output device capable of visually or audible notifying information to an occupant of the vehicle or the outside of the vehicle. In the example of Fig. 12, an audio speaker 12061, a display section 12062, and an instrument panel 12063 are illustrated as the output device. The display section 12062 may, for example, include at least one of an on-board display or a head-up display.

Fig. 13 is a diagram depicting an example of the installation position of the imaging section 12031, wherein the imaging section 12031 may include imaging sections 12101, 12102, 12103, 12104, and 12105.

The imaging sections 12101, 12102, 12103, 12104, and 12105 are, for example, disposed at positions on a front nose, side-view mirrors, a rear bumper, and aback door of the vehicle 12100 as well as a position on an upper portion of a windshield within the interior of the vehicle. The imaging section 12101 provided to the front nose and the imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle obtain mainly an image of the front of the vehicle 12100. The imaging sections 12102 and 12103 provided to the side view mirrors obtain mainly an image of the sides of the vehicle 12100. The imaging section 12104 provided to the rear bumper or the back door obtains mainly an image of the rear of the vehicle 12100. The imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle is used mainly to detect a preceding vehicle, a pedestrian, an obstacle, a signal, a traffic sign, a lane, or the like. Incidentally, Fig. 13 depicts an example of photographing ranges of the imaging sections 12101 to 12104. An imaging range 12111 represents the imaging range of the imaging section 12101 provided to the front nose. Imaging ranges 12112 and 12113 respectively represent the imaging ranges of the imaging sections 12102 and 12103 provided to the side view mirrors. An imaging range 12114 represents the imaging range of the imaging section 12104 provided to the rear bumper or the back door. A bird's-eye image of the vehicle 12100 as viewed from above is obtained by superimposing image data imaged by the imaging sections 12101 to 12104, for example.

At least one of the imaging sections 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging sections 12101 to 12104 may be a stereo camera constituted of a plurality of imaging elements, an imaging element having pixels for phase difference detection, or a depth sensor device 100 as described above.

For example, the microcomputer 12051 can determine a distance to each three-dimensional object within the imaging ranges 12111 to 12114 and a temporal change in the distance (relative speed with respect to the vehicle 12100) on the basis of the distance information obtained from the imaging sections 12101 to 12104, and thereby extract, as a preceding vehicle, a nearest three-dimensional object in particular that is present on a traveling path of the vehicle 12100 and which travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, equal to or more than 0 km/hour). Further, the microcomputer 12051 can set a following distance to be maintained in front of a preceding vehicle in advance, and perform automatic brake control (including following stop control), automatic acceleration control (including following start control), or the like. It is thus possible to perform cooperative control intended for automatic driving that makes the vehicle travel autonomously without depending on the operation of the driver or the like.

For example, the microcomputer 12051 can classify three-dimensional object data on three-dimensional objects into three-dimensional object data of a two-wheeled vehicle, a standard-sized vehicle, a large-sized vehicle, a pedestrian, a utility pole, and other three-dimensional objects on the basis of the distance information obtained from the imaging sections 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic avoidance of an obstacle. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that the driver of the vehicle 12100 can recognize visually and obstacles that are difficult for the driver of the vehicle 12100 to recognize visually. Then, the microcomputer 12051 determines a collision risk indicating a risk of collision with each obstacle. In a situation in which the collision risk is equal to or higher than a set value and there is thus a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display section 12062, and performs forced deceleration or avoidance steering via the driving system control unit 12010. The microcomputer 12051 can thereby assist in driving to avoid collision.

At least one of the imaging sections 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 can, for example, recognize a pedestrian by determining whether or not there is a pedestrian in imaged images of the imaging sections 12101 to 12104. Such recognition of a pedestrian is, for example, performed by a procedure of extracting characteristic points in the imaged images of the imaging sections 12101 to 12104 as infrared cameras and a procedure of determining whether or not it is the pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputer 12051 determines that there is a pedestrian in the imaged images of the imaging sections 12101 to 12104, and thus recognizes the pedestrian, the sound/image output section 12052 controls the display section 12062 so that a square contour line for emphasis is displayed so as to be superimposed on the recognized pedestrian. The sound/image output section 12052 may also control the display section 12062 so that an icon or the like representing the pedestrian is displayed at a desired position.

The example of the vehicle control system to which the technology according to the present disclosure is applicable has been described above. By applying event-triggered image information, the image data transmitted through the communication network may be reduced and it may be possible to reduce power consumption without adversely affecting driving support.

Additionally, the present technology is not limited to the above, but various changes can be made within the scope of the present technology without departing from the gist of the present technology.

The solid-state imaging device according to the present disclosure may be any device used for analyzing and/or processing radiation such as visible light, infrared light, ultraviolet light, and X-rays. For example, the solid-state imaging device may be any electronic device in the field of traffic, the field of home appliances, the field of medical and healthcare, the field of security, the field of beauty, the field of sports, the field of agriculture, the field of image reproduction or the like.

Specifically, in the field of image reproduction, the depth sensor device may be included in a device for capturing an image to be provided for appreciation, such as a digital camera, a smart phone, or a mobile phone device having a camera function. In the field of traffic, for example, the depth sensor device may be integrated in an in-vehicle sensor that captures the front, rear, peripheries, an interior of the vehicle, etc. for safe driving such as automatic stop, recognition of a state of a driver, or the like, in a monitoring camera that monitors traveling vehicles and roads, or in a distance measuring sensor that measures a distance between vehicles or the like.

In the field of home appliances, the depth sensor device may be integrated in any type of sensor that can be used in devices provided for home appliances such as TV receivers, refrigerators, and air conditioners to capture gestures of users and perform device operations according to the gestures. Accordingly the depth sensor device may be integrated in home appliances such as TV receivers, refrigerators, and air conditioners and/or in devices controlling the home appliances. Furthermore, in the field of medical and healthcare, the depth sensor device may be integrated in any type of sensor, e.g. a solid-state image device, provided for use in medical and healthcare, such as an endoscope or a device that performs angiography by receiving infrared light.

In the field of security, the depth sensor device can be integrated in a device provided for use in security, such as a monitoring camera for crime prevention or a camera for person authentication use. Furthermore, in the field of beauty, the depth sensor device can be used in a device provided for use in beauty, such as a skin measuring instrument that captures skin or a microscope that captures a probe. In the field of sports, the depth sensor device can be integrated in a device provided for use in sports, such as an action camera or a wearable camera for sport use or the like. Furthermore, in the field of agriculture, the depth sensor device can be used in a device provided for use in agriculture, such as a camera for monitoring the condition of fields and crops.

The present technology can also be configured as described below:

(1) A depth sensor device for measuring a depth map of an object, the depth sensor device comprising: a projector unit configured to illuminate the object with a series of illumination patterns; a receiver unit comprising a plurality of pixels, the receiver unit being configured to detect on a pixel basis intensities of light reflected from the object while it is illuminated with the illumination patterns; and a control unit configured to determine by triangulation a distance of a point on the object to a single pixel, which point lies on the line of sight of the single pixel, by determining from intensity information obtained from the single pixel for the series of illumination patterns a direction of a pointing line miming from the projector unit to said point.

(2) The depth sensor device according to (1), wherein each of the illumination patterns has a predetermined projection magnitude function that indicates the distribution of light intensity of the illumination pattern in space; and the control unit is configured to determine the direction of the pointing line based on the intensity information obtained by the single pixel for the series of illumination patterns and based on the predetermined projection magnitude functions of the respective illumination patterns.

(3) The depth sensor device according to (1) or (2), wherein the intensity information obtained by the single pixel for a given illumination pattern can be modeled based on a predetermined projection magnitude function of that illumination pattern; and the predetermined projection magnitude functions are used to generate a lookup table that indicates the direction of the pointing line for intensity information obtained by the single pixel.

(4) The depth sensor device according to any of (1) to (3), wherein the projector unit is configured to project a series of light sheets with different projection angles onto the object as the series of illumination patterns; the projection angles are defined on a plane containing the projector unit and the receiver unit and are formed between a reference direction on the plane and respective intersections of the light sheets with the plane; and the control unit is configured to determine the direction of the pointing line by determining a virtual projection angle of a virtual light sheet passing through said point.

(5) The depth sensor device (100) according to (4), wherein each light sheet is projected only once during measuring the depth map of the object.

(6) The depth sensor device according to (4), wherein more than one light sheet is projected at the same time onto the object; and the control unit is configured to distinguish which intensity information has been received at the single pixel from which light sheet from the temporal and spatial order of the light sheets.

(7) The depth sensor device according to any of (4) to (6), wherein more than one light sheet is projected onto the object during an exposure period of the receiver unit; the single pixel is configured to obtain a temporal sequence of events of a given polarity during the exposure period; and the control unit is configured to determine which intensity information has been received at the single pixel from which light sheet from the temporal sequence of events obtained by the single pixel.

(8) The depth sensor device according to any of (4) to (7), wherein the control unit is configured to determine from the intensity information a main light sheet of the series of light sheets, which passes closest by said point, and to determine the virtual projection angle by using a lookup table that assigns the virtual projection angle to a given main light sheet and given intensity information of the single pixel for each of the series of light sheets.

(9) The depth sensor device according to any of (4) to (8), wherein each of the light sheets has a predetermined projection magnitude function that indicates the light intensity of the light sheet as a function of the angle to the reference direction on the plane; and the control unit is configured to determine the virtual projection angle based on the intensity information obtained by the single pixel for the series of light sheets and based on the predetermined projection magnitude functions of the respective light sheets.

(10) The depth sensor device according to (9), wherein the control unit is configured to determine from the intensity information a main light sheet of the series of light sheets, which passes closest by said point, and to determine the virtual projection angle by referring to the predetermined projection magnitude functions of the main light sheet and of at least one of the two light sheets out of the series of light sheets adjacent to the main light sheet.

(11) The depth sensor device according to (10), wherein the intensity information indicates the intensity of light received at the single pixel during illumination of the object with the series of light sheets; and the control unit is configured to calculate for the single pixel the ratio of and to determine the virtual projection angle by referring with the calculated ratio to a lookup table, where ¾ϋ] indicates the intensity measured for light sheet k at pixel j, with k counting the light sheets in a consecutive order, and m_j denotes the main light sheet, which is the light sheet producing the largest value of R_k[j]. (12) The depth sensor device according to (11), wherein the single pixel obtains a temporal sequence of events of a given polarity; and the control unit is configured to determine the intensity of light received at the single pixel by referring to a predetermined mapping of the temporal sequence of events to the intensity of light received at the single pixel.

(13) The depth sensor device according to (10), wherein the intensity information indicates a number of events of a given polarity detected by the single pixel after switching illumination between light sheets; and the control unit is configured to calculate for the single pixel the ratio of and to determine the virtual projection angle from the calculated ratio by referring with the calculated ratio to a lookup table, where R_k[j] indicates the number of events of the given polarity measured for light sheet k by pixel j, with k counting the light sheets in a consecutive order, m_j denotes the main light sheet, which is the light sheet producing the largest R_k|j | . and c(j) denotes a threshold for an intensity change at pixel j necessary to trigger an event of the given polarity.

(14) The depth sensor device according to any of (8) to (10), wherein the intensity information indicates the temporal delay between a first polarity event detected by the single pixel and the time of switching illumination between light sheets; and the control unit is configured to determine the virtual projection angle from the temporal delays obtained for the main light sheet, which is the light sheet causing the shortest temporal delay, and its two adjacent light sheets.

(15) The depth sensor device according to any of (1) to (14), wherein the control unit is configured to determine distances of points on the object to each of the pixels of the receiver unit.

(16) A method for measuring a depth map of an object with a depth sensor device that comprises a projector unit and a receiver unit comprising a plurality of pixels, the method comprising illuminating, by the projector, the object with a series of illumination patterns; detecting, by the receiver unit, on a pixel basis intensities of light reflected from the object while it is illuminated with the illumination patterns; and determining by triangulation a distance of a point on the object to a single pixel, which point lies on the line of sight of the single pixel, by determining from intensity information obtained from the single pixel for the series of illumination patterns a direction of a pointing line running from the projector unit to said point.

Claims

1. A depth sensor device (100) for measuring a depth map of an object (200), the depth sensor device (100) comprising: a projector unit (110) configured to illuminate the object (200) with a series of illumination patterns; a receiver unit (120) comprising a plurality of pixels (122), the receiver unit (120) being configured to detect on a pixel basis intensities of light reflected from the object (200) while it is illuminated with the illumination patterns; and a control unit (130) configured to determine by triangulation a distance of a point (210) on the object (200) to a single pixel (122a), which point (210) lies on the line of sight of the single pixel (122a), by determining from intensity information obtained from the single pixel (122a) for the series of illumination patterns a direction of a pointing line (p) running from the projector unit (110) to said point (210).

2. The depth sensor device (100) according to claim 1, wherein each of the illumination patterns has a predetermined projection magnitude function that indicates the distribution of light intensity of the illumination pattern in space; and the control unit (130) is configured to determine the direction of the pointing line based on the intensity information obtained by the single pixel (122a) for the series of illumination patterns and based on the predetermined projection magnitude functions of the respective illumination patterns.

3. The depth sensor device (100) according to claim 2, wherein the intensity information obtained by the single pixel (122a) for a given illumination pattern can be modeled based on the predetermined projection magnitude function of that illumination pattern; and the predetermined projection magnitude functions are used to generate a lookup table that indicates the direction of the pointing line for intensity information obtained by the single pixel (122a).

4. The depth sensor device (100) according to claim 1, wherein the projector unit (110) is configured to project a series of light sheets (l_k) with different projection angles (p^ma ) onto the object as the series of illumination patterns; the projection angles (p"^la ) are defined on a plane containing the projector unit (110) and the receiver unit (120) and are formed between a reference direction (r) on the plane and respective intersections of the light sheets (l_k) with the plane; and the control unit (130) is configured to determine the direction of the pointing line by determining a virtual projection angle (p_j) of a virtual light sheet passing through said point (210).

5. The depth sensor device (100) according to claim 4, wherein each light sheet (l_k) is projected only once during measuring the depth map of the object (200).

6. The depth sensor device (100) according to claim 4, wherein more than one light sheet (l_k) is projected at the same time onto the object (200); and the control unit (130) is configured to distinguish which intensity information has been received at the single pixel (122a) from which light sheet (l_k) from the temporal and spatial order of the light sheets (l_k). 7. The depth sensor device (100) according to claim 4, wherein more than one light sheet (l_k) is projected onto the object during an exposure period of the receiver unit

(120); the single pixel (122a) is configured to obtain a temporal sequence of events of a given polarity during the exposure period; and the control unit (130) is configured to determine which intensity information has been received at the single pixel (122a) from which light sheet (l_k) from the temporal sequence of events obtained by the single pixel (122a)

8. The depth sensor device (100) according to claim 4, wherein the control unit (130) is configured to determine from the intensity information a main light sheet (l_mJ) of the series of light sheets (l_k), which passes closest by said point (210), and to determine the virtual projection angle (p_j) by using a lookup table that assigns the virtual projection angle (p_j) to a given main light sheet (l_mJ) and given intensity information of the single pixel (122a) for each of the series of light sheets (l_k).

9. The depth sensor device (100) according to claim 4, wherein each of the light sheets (l_k) has a predetermined projection magnitude function (P_k(p)) that indicates the light intensity of the light sheet (l_k) as a function of the angle (p) to the reference direction (r) on the plane; and the control unit (130) is configured to determine the virtual projection angle (p_j) based on the intensity information obtained by the single pixel (122a) for the series of light sheets (l_k) and based on the predetermined projection magnitude functions (P_k(p)) of the respective light sheets (l_k).

10. The depth sensor device (100) according to claim 9, wherein the control unit (130) is configured to determine from the intensity information a main light sheet (l_mJ) of the series of light sheets (l_k), which passes closest by said point (210), and to determine the virtual projection angle (p_j) by referring to the predetermined projection magnitude functions (P_mj(p)) of the main light sheet (l_mJ) and of at least one of the two light sheets (l_mj_i, l_mj+i) out of the series of light sheets (l_k) adjacent to the main light sheet (l_mj).

11. The depth sensor device (100) according to claim 10, wherein the intensity information indicates the intensity of light received at the single pixel (122a) during illumination of the object (200) with the series of light sheets(l_k); and the control unit (130) is configured to calculate for the single pixel (122a) the ratio of and to determine the virtual projection angle (p_j) by referring with the calculated ratio to a lookup table, where R_k[j] indicates the intensity measured for light sheet k at pixel j, with k counting the light sheets (l_k) in a consecutive order, and nij denotes the main light sheet (l_mj), which is the light sheet (l_k) producing the largest value of R_k[j].

12. The depth sensor device (100) according to claim 11, wherein the single pixel (122a) obtains a temporal sequence of events of a given polarity; and the control unit (130) is configured to determine the intensity of light received at the single pixel (122a) by referring to a predetermined mapping of the temporal sequence of events to the intensity of light received at the single pixel (122a).

13. The depth sensor device (100) according to claim 10, wherein the intensity information indicates a number of events of a given polarity detected by the single pixel (122a) after switching illumination between light sheets (l_k); and the control unit (130) is configured to calculate for the single pixel (122a) the ratio of and to determine the virtual projection angle (p_j) from the calculated ratio by referring with the calculated ratio to a lookup table, where R_k[j] indicates the number of events of the given polarity measured for light sheet k by pixel j, with k counting the light sheets (l_k) in a consecutive order, m_j denotes the main light sheet (l_mj), which is the light sheet (l_k) producing the largest R_k[j], and c(j) denotes a threshold for an intensity change at pixel j necessary to trigger an event of the given polarity.

14. The depth sensor device (100) according to claim 8, wherein the intensity information indicates the temporal delay between a first polarity event detected by the single pixel (122a) and the time of switching illumination between light sheets (l_k); and the control unit (130) is configured to determine the virtual projection angle (p_j) from the temporal delays obtained for the main light sheet (l_mJ), which is the light sheet causing the shortest temporal delay and its two adjacent light sheets (l_mj-i, l_mj+i)·

15. A method for measuring a depth map of an object (200) with a depth sensor device (100) that comprises a projector unit (110) and a receiver unit (120) comprising a plurality of pixels (122), the method comprising illuminating, by the projector (110), the object (200) with a series of illumination patterns; detecting, by the receiver unit (120), on a pixel basis intensities of light reflected from the object (200) while it is illuminated with the illumination patterns; and determining by triangulation a distance of a point (210) on the object (200) to a single pixel (122a), which point (210) lies on the line of sight of the single pixel (122a), by determining from intensity information obtained from the single pixel (122a) for the series of illumination patterns a direction of a pointing line running from the projector unit (110) to said point (210).