EP4689547A1

EP4689547A1 - Apparatuses and methods for polarization based surface normal imaging

Info

Publication number: EP4689547A1
Application number: EP24712843.2A
Authority: EP
Inventors: Tomoo Mitsunaga; Diederik Paul MOEYS; Vincent PARRET; Taishi ONO; Yalcin Incesu
Original assignee: Sony Europe BV; Sony Semiconductor Solutions Corp
Current assignee: Sony Europe BV; Sony Semiconductor Solutions Corp
Priority date: 2023-03-31
Filing date: 2024-03-22
Publication date: 2026-02-11
Also published as: WO2024200292A1

Abstract

The present disclosure relates to an apparatus and a method for shape measurement of a scene. The apparatus comprises a polarizer having a plurality of selectable polarization angles and a polarization selector configured to select a first polarization angle of the polarizer to cause the polarizer to pass light from the scene at the first polarization angle and to subsequently select a second polarization angle to cause the polarizer to pass light from the scene at the second polarization angle. The apparatus further comprises an event-based vision sensor, EVS, configured to detect a first set of events associated with the passed light of the first polarization angle of the polarizer and to subsequently detect a second set of events associated with the passed light of the second polarization angle of the polarizer and a shape estimation processor configured to compute surface normal information of the scene based on the first and second set of events and the corresponding first and second polarization angles of the polarizer.

Description

APPARATUSES AND METHODS FOR POLARIZATION BASED SURFACE NORMAL IMAGING

Field

The present disclosure relates to apparatuses and methods for surface normal estimation based on polarization information.

Background

Surface normal estimation is a useful tool to perform depth estimation and 3D modeling of objects and scenes in many applications, such as augmented reality (AR) and virtual reality (VR), holograms, 3D television, robotics, high-speed defect inspection, and scene analysis for automotive scenarios. Since reflected light has a polarization state corresponding to a surface from which it is reflected, polarization information can be used to obtain surface normal estimation for a 3D model. Such a method is known as Shape from Polarization (SfP).

Traditional frame-based SfP requires a camera to capture an image for fixed, predefined angles of a polarizer in front of the camera. Polarized images can be captured at full resolution of the camera but at different timestamps. Such “division of time” approaches are temporally limited in a speed of image capture by a frame rate of the camera. While this method would perform well for stationary scenes, their performance would not translate well to dynamic scenes. To overcome this, other approaches such as mosaicking have been developed, wherein micro-optical polarizers are directly integrated onto an array of pixels within the sensor plane of the camera imaging system. Each pixel within a group of pixels, usually four, is assigned a different predefined angle for its corresponding polarizer. Polarization information corresponding to all four polarization angles can then be captured simultaneously, but at one quarter of the original resolution. Most commercial polarization sensors use this approach, also known as a “division of focal plane”. As with many imaging applications and depth estimation techniques, there is an inevitable speed vs. accuracy tradeoff. Highly accurate methods are not fast, especially with moving objects. SfP methods in general also have high error rates, particularly with the mean angular error of surface normal estimation, since the underlying physics based on the Fresnel equations is among the most optically complex of all computer vision problems. SfP methods are also susceptible to noise since the captured light intensity is reduced by 50 percent.

Thus, there is a demand for improved concepts for surface normal estimation with polarization information.

Summary

This demand is addressed by apparatuses and methods for surface normal estimation based on polarization information in accordance with the independent claims. Possibly advantageous embodiments are addressed by the dependent claims.

According to a first aspect, the present disclosure proposes an apparatus for shape measurement of a scene. The apparatus comprises at least one polarizer having a plurality of selectable polarization angles. The apparatus further comprises a polarization selector configured to select a first polarization angle of the polarizer to cause the polarizer to pass light from the scene at the first polarization angle and to subsequently select a second polarization angle to cause the polarizer to pass light from the scene at the second polarization angle. Furthermore, the apparatus comprises an event-based vision sensor, EVS, configured to detect a first set of events associated with the passed light of the first polarization angle of the polarizer and to subsequently detect a second set of events associated with the passed light of the second polarization angle of the polarizer. In addition, the apparatus comprises a shape estimation processor configured to compute surface normal information of the scene based on the first and second set of events and the corresponding first and second polarization angles of the polarizer.

In some embodiments, the shape estimation processor may be configured to compute an orientation of one or more surface normals of a portion of the scene based on a change in intensity of passed light between the first and second polarization angles of the polarizer that triggered the EVS to output an event associated with the portion of the scene. In some embodiments, the polarizer comprises a liquid crystal layer including a liquid crystal material and the polarization selector comprises a first pair of electrodes and a second pair of electrodes. The polarization selector may be configured to select the first polarization angle by activating the first pair of electrodes and to select the second polarization angle by activating the second pair of electrodes. The liquid crystal layer may be positioned in between a first electrode and a second electrode of the respective pair of electrodes. The polarizer may comprise a front plate and a back plate. The polarizer may be configured to pass a light signal of the scene through the front plate, subsequently through the liquid crystal material of the liquid crystal layer, and subsequently through the back plate. Additionally, the back plate may comprise parallel grating grooves.

In some embodiments, the polarizer comprises a plurality of polarizer sections. The polarizer may include a first polarizer section that may be fixed to pass light from the scene at the first polarization angle and a second polarizer section that may be fixed to pass light from the scene at the second polarization angle. The polarization selector may be configured to select the first polarization angle by directing light from the scene to the first polarizer section and to select the second polarization angle by directing light from the scene to the second polarizer section. Furthermore, the polarization selector may comprise a reflective surface and an electric drive configured to control an orientation of the reflective surface. The reflective surface may be a MEMS mirror. The polarization selector may also comprise an electromagnetic actuation mechanism or an electrostatic actuation mechanism to control the orientation of the reflective surface.

In some embodiments, the polarizer is a rotatable polarizer and the polarization selector comprises a powered gear for rotating the rotatable polarizer. The polarization selector may be configured to select the first polarization angle by rotating the rotatable polarizer to a first rotation angle and to select the second polarization angle by rotating the rotatable polarizer to a second rotation angle.

In some embodiments, the apparatus comprises a second EVS configured to detect one or more areas of motion in the scene. The polarization selector may comprise a controlling circuit that may be configured to control a switching speed of the polarizer from the first polarization angle to the second polarization angle and from the second polarization angle to a third polarization angle based on information from the one or more areas of motion detected by the second EVS. Furthermore, the controlling circuit may be configured to control the switching speed based on a number of events generated by the second EVS in a predefined time interval. The controlling circuit may also be configured to increase the switching speed when an amount of motion detected by the second EVS increases and to decrease the switching speed of the rotatable polarizer when the amount of motion detected by the second EVS decreases.

In some embodiments, the apparatus comprises a plurality of polarizers and a plurality of polarization selectors. Each polarization selector may correspond to a polarizer. Furthermore, each polarization selector may be configured to sequentially select multiple polarization angles for the corresponding polarizer. In addition, each polarizer may be positioned and oriented to pass light to a uniquely corresponding group of pixels of the EVS.

In some embodiments, the apparatus further comprises a linearly polarizing structure positioned between the plurality of polarizers and the EVS. The linearly polarizing structure may comprise a plurality of sections, each section being fixed to receive light from a corresponding polarizer of the plurality of polarizers and to pass the received light to the group of pixels uniquely corresponding to the polarizer. The linearly polarizing structure may be in the form of a linearly polarizing film. Furthermore, each section of the plurality of sections may comprise a plurality of subsections, wherein each subsection is fixed to pass light at a pre-specified polarization angle that is unique within the respective section. Each subsection may uniquely correspond to a pixel of the EVS.

In some embodiments, the apparatus comprises a plurality of at least two liquid crystal polarizers, wherein the respective back plate of each liquid crystal polarizer comprises parallel grating grooves oriented parallel to the pre-specified polarization angle of a corresponding subsection within each section of the plurality of sections.

Some embodiments of the apparatus may relate to machine learning. The shape estimation processor may comprise a trained machine-learning model configured to predict one or more surface normals of the scene based on a plurality of pre-specified polarization angles of the polarizer and respective events associated with the plurality of pre-specified polarization angles of the polarizer. The machine-learning model may also be configured to implement a supervised learning algorithm. According to a further aspect, the present disclosure proposes a method for shape measurement of a scene. The method includes providing a polarizer with a plurality of selectable polarization angles. The method further includes selecting a first polarization angle of the polarizer to cause the polarizer to pass light from the scene at the first polarization angle and detecting, with an EVS, a first set of events associated with the passed light of the first polarization angle of the polarizer. Additionally, the method includes selecting a second polarization angle to cause the polarizer to pass light from the scene at the second polarization angle and detecting, with the EVS, a second set of events associated with the passed light of the second polarization angle of the polarizer. Furthermore, the method includes computing surface normal information of the scene based on the first and second set of events and the corresponding first and second polarization angles of the polarizer.

Brief description of the Figures

Some examples of apparatuses and/or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which

Fig. 1 schematically illustrates an apparatus for polarization-based surface normal measurement according to a first embodiment;

Fig. 2 depicts a spherical coordinate system, wherein a surface normal unit vector, n, is determined by a zenith angle, 0, and an azimuth angle, (p,

Fig. 3 illustrates a flow chart of an exemplary method for polarization-based surface normal measurement.

Fig. 4 schematically illustrates another apparatus for polarization-based surface normal measurement according to an embodiment comprising a liquid crystal polarizer;

Fig. 5 schematically illustrates another apparatus for polarization-based surface normal measurement according to an embodiment comprising multiple polarizer sections; Fig. 6 schematically illustrates another apparatus for polarization-based surface normal measurement according to an embodiment comprising a rotatable polarizer;

Fig. 7 schematically illustrates another apparatus for polarization-based surface normal measurement according to an embodiment further at least one polarizer upstream to a linearly polarizing structure with multiple sections;

Fig. 8 schematically illustrates another apparatus for polarization-based surface normal measurement according to an embodiment comprising a 2^nd EVS;

Fig. 9 illustrates a flow chart of an exemplary method for polarization-based surface normal measurement applying the 2^nd EVS; and

Fig. 10 schematically illustrates another apparatus for polarization-based surface normal measurement according to an embodiment comprising a machine-learning network.

Detailed Description

Some examples are now described in more detail with reference to the enclosed figures. However, other possible examples are not limited to the features of these embodiments described in detail. Other examples may include modifications of the features as well as equivalents and alternatives to the features. Furthermore, the terminology used herein to describe certain examples should not be restrictive of further possible examples.

Throughout the description of the figures same or similar reference numerals refer to same or similar elements and/or features, which may be identical or implemented in a modified form while providing the same or a similar function. The thickness of lines, layers and/or areas in the figures may also be exaggerated for clarification.

When two elements A and B are combined using an “or”, this is to be understood as disclosing all possible combinations, i.e. only A, only B as well as A and B, unless expressly defined otherwise in the individual case. As an alternative wording for the same combinations, "at least one of A and B" or "A and/or B" may be used. This applies equivalently to combinations of more than two elements.

If a singular form, such as “a”, “an” and “the” is used and the use of only a single element is not defined as mandatory either explicitly or implicitly, further examples may also use several elements to implement the same function. If a function is described below as implemented using multiple elements, further examples may implement the same function using a single element or a single processing entity. It is further understood that the terms "include", "including", "comprise" and/or "comprising", when used, describe the presence of the specified features, integers, steps, operations, processes, elements, components and/or a group thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, processes, elements, components and/or a group thereof.

Fig. 1 schematically illustrates an apparatus 100 for shape measurement according to a first embodiment. More specifically, the apparatus 100 is configured to perform polarization-based surface normal measurement of one or more objects 104 in a scene 102. The apparatus 100 comprises a polarizer 110, a polarization selector 120 connected to the polarizer 110, an eventbased sensor, EVS 130, positioned behind the polarizer 110, and a shape estimation processor 140 positioned downstream to the EVS 130.

The scene 102 may be a static scene or a dynamic scene with motion in a field of view of the EVS 130. The scene 102 may be shaped or structured 3-dimensionally. In other words, the scene 102 may comprise a background 106 and one or more objects 104 in a foreground. For example, the scene 102 may comprise a Lambertian or non-Lambertian surface (e.g. reflective surfaces, transparent glass, etc.). By way of the 3D structure of the scene 102, unpolarized incident light may be incident on a polarizing object 104, which may lead to the light becoming partially polarized. The orientation and degree of polarization of light reflecting off the polarizing object 104 may encode information related to a surface or shape of the object 104. This may occur due to a reflecting surface causing light waves to oscillate more in one direction than another. The degree of partially polarized light caused by reflection may be influenced by multiple factors, which may include an angle of incidence, properties of the surface material, and the wavelength of light. The light may then be reflected in a partially polarized form from the direction of the polarizing object 104 in the scene 102 to the polarizer 110 of the apparatus 100.

The polarizer 110 is an optical filter that lets light waves of a specific polarization to pass or to be transmitted while blocking light waves of other polarizations. Polarization may be described in terms of polarization states specifying an orientation of an electric field for an electromagnetic wave. For example, a polarization may be in a horizontally or vertically polarized state (which may correspond to an orientation of 0° and 90° with respect to a horizontal plane, respectively) or in a diagonally or anti-diagonally polarized state (which may correspond to 45° and 135°, respectively).

The polarizer 110 is able to effectively manipulate the polarization of an incoming light signal, or the orientation of an electric field for an incoming electromagnetic wave. The polarizer 110 may comprise a polarizing material. The polarizing material may be an absorptive material comprising absorptive polymer complexes, such as iodine-doped polyvinyl alcohol chains or other polymer complexes, optionally with one or more dopants, to enhance an absorption of radiation polarized according to the direction to the complexes. Alternatively, the polarizing material may be a reflective material comprising a metallic wire grid or other structures configured to reflect one or more polarization states of light. Further, the material of the linear polarizer 110 may be a beam-splitting polarizer material based on the principles of Fresnel reflection, a birefringent polarizer material, or a dichroic polarizer material. The polarizing material may also be a thin-film polarizer material, which may comprise a special optical coating applied on a glass substrate.

The polarizer 110 is connected to a polarization selector 120. The polarization selector 120 is configured to select two or more polarization angles 112 for the polarizer 110. After selecting a first polarization angle 112A and causing the polarizer 110 to pass light from the scene 102 at the first polarization angle 112A, the polarization selector 120 is further configured to select at least a second polarization angle 112B and cause the polarizer 110 to subsequently pass light from the scene 102 at the second polarization angle 112B. For each subsequent selection of a polarization angle 112, the polarization selector 120 is configured to subsequently cause the polarizer 110 to pass light from the scene 102 at the selected polarization angle 112. The polarization selector 120 may be physically, electrically, and/or communicatively connected to the polarizer 110 to cause the polarizer 110 to change its polarization angle 112 from a first polarization angle 112A to a second polarization angle 112B. The polarization selector may be configured to cause the polarizer 110 to change its polarization angle 112 from the second polarization angle 112B to a third polarization angle 112C (and optionally to any number of further polarization angles 112) or directly back to the first polarization angle 112A. The polarization selector 120 may be configured to select any number of predefined polarization angles 112 in any combination and in any order. The polarization selector 120 may be configured to select two or more predefined polarization angles 112 of the polarizer 110 in a predefined order or to randomize an order of selection of the two or more predefined polarization angles 112.

Behind the polarizer 110 of the apparatus 100 is an event-based vision sensor, EVS 130, which may also be known as a dynamic vision sensor or an event-based vision camera. In general, the EVS 130 is configured to detect events 132 based on changes in intensity within a field of view of the EVS 130. The changes in intensity behind the polarizer 110 may be caused by changes in the polarization angle 112 of the polarizer 110. Thus, any set of events recorded by the EVS 130 may be referred to as polarization events 132. More specifically, the EVS 130 is configured to detect a first set of events 132A associated with a first selected polarization angle 112A. For detecting the first set of events 132A, the polarization angle 112 of the polarizer 110 may have been changed from a previous polarization angle 112 to the first polarization angle 112A, which may have caused a first change in intensity. The first change in intensity may be recorded as a first set of events 132A by the EVS 130 and labeled as being associated with the first polarization angle 112A. The polarization angle 112 may then be changed again to a second polarization angle 112B, which may have been changed from the first polarization angle 112A or another previous polarization angle 112. This may cause a second change in intensity, which may be recorded as a second set of events 132B by the EVS 130 and labeled as being associated with the second polarization angle 112B.

The EVS 130 may be configured to continuously record sets of events 132, each corresponding to a change in intensity caused by a change in polarization angle 112 of the polarizer 110. In other words, as light is continuously being reflected from a polarizing object 104 in the scene 102 toward the EVS 130 and within its the field of view, the polarization selector 120 may be configured to continually change the polarization angle 112 of the polarizer 110. In general, the EVS 130 refers to an imaging sensor that responds to local changes in light intensity. Pixels of an EVS 130 operate independently and asynchronously. This property allows every pixel to generate an event exactly at the point in time when its illumination changes, leading to a very fast response time; the latency typically lies in the order of tens of microseconds. An increase in brightness may trigger a so called ON-event, while a decrease may trigger an OFF-event.

The apparatus 100 further comprises downstream to the EVS 130 a shape estimation processor 140. Information related to the polarizer 110, the polarization selector 120 and the EVS 130, including each set of detected events 132 and the associated polarization angles 112 chosen to create a corresponding change in intensity, may be provided as input to the shape estimation processor 140 for shape measurement of the scene 102. The shape estimation processor 140 may be a hardware apparatus, such as a processor, a microprocessor, a programmable computer, or an electronic circuit, and may take multiple forms. For example, the processing circuitry 102 may be a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which or all of which may be shared.

As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), graphics processing units (GPUs), a neuromorphic processor, or any other type of processor or processing circuit. Other types of circuits that may be included in the shape estimation processor may be a custom circuit, an application-specific integrated circuit (ASIC), or the like. The shape estimation processor 140 may optionally be coupled to, e.g., read only memory (ROM) for storing software, random access memory (RAM) and/or non-volatile memory. Optionally, the shape estimation processor 140 may comprise further processing circuitry.

The light intensity of a light signal that has passed through the polarizer 110 will vary according to how much the polarization components of the light signal from the scene 102 align with the selected polarization angles 112 of the polarizer 110. This will determine the intensity of light incident on each pixel of the EVS 130 and how the intensity changes by changing the polarization angle 112 of the polarizer 110. Each pixel may trigger an event to be stored as event information once a predefined threshold for a change in intensity has been met. More specifically, an EVS generates an event ek = ( k, tk,pk) at time tk when a logarithmic brightness at the pixel xk = xk, yk)^T increases or decreases by predefined threshold C: L( k, tk) ~ (xk, tk - tk) = pkC, where pk G {-1, +1 } denotes the sign (polarity) of the brightness change, and Atk is the time since the last event at the same pixel location.

The EVS 130 exhibits a significant advantage in efficiency compared to a conventional camera detector, which may continuously record an intensity of a light signal for each pixel within a pixel array. EVS pixels only record information based on a change in intensity beyond a predefined threshold, which may enable information of the scene 102 to be recorded and transferred to the shape estimation processor 140 with a significantly reduced amount of input data. This may enable the shape estimation processor 140 to perform computations related to shape estimation with significantly greater efficiency.

In general, an EVS is normally used for dynamic scenes since a static scene would not provide any changes in light intensity. However, this is not the case with the polarizer 110 positioned in front of the EVS 130 and connected to the polarization selector 120 to switch the polarization angle 112. Partially polarized light reflected off a static object 104 of the scene 102 may lead to a varying transmission through the polarizer 110 as the polarization angle of the polarizer 110 changes, depending on the selected polarization angles 112. As such, polarization information regarding the static object 104 can be detected in the form of events 132. While event-based sensors by themselves are only capable of detecting dynamic objects or scenes, such a setup with the polarizer 110 connected to the polarization selector 120 and positioned in front of the EVS 130 enables polarization-based information to be detected in the form of events 132 for both dynamic and static scenes. Such a setup may be capable of extracting shape information of many objects 104 within the scene 102 based on the input provided to the shape estimation processor 140 and its processing features.

The shape estimation processor 140 is configured to compute surface normal information 142 of the scene 102, which may comprise a plurality of surface normal vectors or “surface normals”. The surface normal information 142 may be based on the first and second set of polarization events 132A; 132B and the corresponding first and second polarization angles 112A; to estimate a shape of one or more objects 104 in the scene 102 based on polarization information of the scene 102. Such a method of computational imaging is known as Shape from Polarization (SfP).

The shape estimation processor 140 is configured to perform SfP by computing information related to surface normal vectors 142 of an object 104. The skilled person will appreciate that a “normal” to a non-flat surface at a point P on the surface is a vector perpendicular to the tangent plane to that surface at P. For example, a surface normal may be a unit vector that is perpendicular to a surface at a specific spot. Therefore, different surface normals for different points P on the surface of the scene 102 will yield information about the 3D shape of the scene 102. It makes use of the polarization information of light that is created when unpolarized light is reflected off an object to be imaged. Since natural scenes mostly have common light sources emitting unpolarized light, information regarding objects can be determined by an SfP-analysis of the polarization upon reflection.

When initially unpolarized light is reflected off objects, it becomes partially linear polarized due to the orientation of the molecular electron charge density interacting with the electromagnetic field of the incident light. This applies to both dielectrics and metals. The polarization information of a light signal can include both the polarization direction of light and the degree to which it has become polarized. Various SfP methods use this information differently, often depending on whether the reflection is diffuse, wherein the light rays are scattered into many different angles, or specular, wherein the light rays are reflected to a single outgoing direction.

The degree to which an unpolarized light signal has become polarized is described by its degree of polarization, DOP, which quantifies how much of the total power of a light signal is polarized. Unpolarized light, such as a light signal emitted from common lighting sources, has a DOP of zero, while partially polarized light, such as a light signal reflected off an object, has a DOP between zero and one. Unpolarized light transmitted through a polarizer at multiple polarization angles consistently transmits the same amount of light intensity, regardless of its polarization angle. For partially polarized light, there exists parallel and perpendicular directions of the polarizer that correspond to a maximal transmission I_max and a minimal transmission I_min of intensity. A light signal with a larger difference between I_max and I_m/n has a greater DOP, so a given threshold of a change in light intensity programmed into the EVS 130 can communicate information related to the DOP. The DOP can be calculated based in the following equation.

The shape estimation processor 140 is configured to compute an orientation of one or more surface normals of a portion of the scene based on a change in intensity of passed light between the first and second polarization angles of the polarizer 110 that triggered the EVS 130 to output an event associated with the portion. The orientation of the one or more surface normals may be found in the context of a spherical coordinate system. The DOP of a previously unpolarized light signal upon reflection from an object carries information related to the reflection angle, 0.

As depicted in Fig. 2, the reflection angle can be labeled as the zenith angle 220, 0, which may be between a viewing direction and a surface normal 210. Besides the zenith angle 220 and surface normal 210, an azimuth angle 230 can also be determined, which may be defined as the angle of projection of the surface normal 210 onto a horizontal plane. While the DOP may carry information related to the zenith angle 220, the orientation of polarization of the light signal may carry information related to the azimuth angle 230. Given both angles 220; 230 at a given point, a surface normal 210 for the given point may be determined. The surface normal information 142 may include the surface normal 210 of multiple points of an object 104 in the scene 102, which may be used in a computation of the shape estimation processor 140 to perform a reconstruction of the object. The multiple points may be more highly concentrated along a boundary, a convexity, or curvature of an object.

The shape estimation processor 140 is configured to compute this shape information based on the event information comprising multiple sets of events 132, with each set corresponding to two polarization angles 112 of the polarizer 110. One polarization angle 112 may correspond to a first detected intensity of light, while another polarization angle 112 may correspond to a second detected intensity of light. As the EVS 130 records events 132 according to the change in intensity between the two detected intensities, the shape estimation processor 140 may receive from the EVS and/or the polarization selector 120 the two polarization angles 112 used to induce the change in intensity recorded by the EVS 130. For this purpose, the shape estimation processor 140 may be configured to receive an input for the sets of events 132 coming from the EVS 130 and to receive an input for the corresponding polarization angles 112 of the polarizer 110. The polarization selector 120 may be directly connected to the shape estimation processor 140 and/or to the EVS 130 in order to automatically record the polarization angles 112 of the polarizer 110 as each polarization angle is selected. Based on these inputs, the shape estimation processor 140 may be configured to compute the surface normal information 142 of the scene 102. This may be done in various ways.

To obtain a formal relationship between the DOP and the reflection angle, 0, one can make use of the Fresnel equations, which can describe the reflection and transmission of linear polarized light when incident on the surface of an object, particularly polarized parallel or perpendicular to the angle of incidence. Using equations related to the degree of polarization, the maximum and minimum intensity, and indices of refraction, together with the Fresnel equations, a relationship between the degree of polarization of light and the reflection angle, 0, can be derived. The relationship between the DOP and the reflection angle, 0, is different depending on whether reflection is diffuse dominant or specular dominant.

The DOP for diffuse dominant reflection is given as follows.

This equation can be rearranged to obtain a closed-form estimation of 0, so there is no ambiguity for the reflection angle in the case of diffuse reflection. Also, the dependence of the DOP on the refractive index n is weak compared to its dependence on the zenith angle 0. As such, it is computationally less challenging to compute surface normal vectors of a Lambertian surface. Given the DOP, this formula can be used to find the direction of the surface normal within an expected range of error.

The DOP for specular dominant reflection is given as follows.

This equation offers two solutions for 0, leading to an ambiguity for the reflection angle in the case of specular reflection. Even with a weak dependence of the DOP on the refractive index n, the zenith angle 0 for specular reflection can only be determined up to a large ambiguity. Surfaces that cause light rays to reflect in many different directions / in a diffuse manner are known as Lambertian surfaces, which are usually matte or have rough edges. An important property of Lambertian surfaces is that the brightness appears uniform from any viewing direction, which makes it easier to estimate the shape of 3D Lambertian objects from multiple views. The reflection angle of the surface normal can also be determined based on the DOP in closed form.

Objects that cause light to reflect in the same direction, or in a specular manner, are non- Lambertian objects, which are shiny or transparent objects, including metals, mirrors, or glass. An important property of non-Lambertian surfaces is that its brightness varies greatly, depending on the viewing direction, which makes it more difficult to estimate the 3D shape. This may pose a significant challenge, whereby traditional depth sensors relating to structured light or time of flight fail. As previously shown, the DOP cannot be uniquely determined for non-Lambertian objects and must be accompanied by additional information. This poses a crucial challenge for numerous applications. Particularly in autonomous navigation, an agent would want to avoid running into a glass building.

Shape from Polarization (SfP), also known as 3D reconstruction from polarization information, can be used for estimating the shape of objects with non-Lambertian surfaces. The shape estimation processor 140 as part of the apparatus 100 may enable a faster approach to shape reconstruction using an events-plus-polarization approach, due to the high time resolution and low latency of event sensors. Such an approach may enable a high-speed scanning of non-Lambertian surfaces, leading to a high-speed capture of surface normal information 142 from the scene 102 and surface normal reconstruction without compromising the spatial resolution. This may be performed by the shape estimation processor 140. The shape estimation processor 140, combined with the features of the polarizer 110, the polarization selector 120, and the EVS 130, may perform a fast and dense shape estimation of non-Lambertian surfaces using the principles of SfP. In particular, the polarization selector 120 may enable quickly switching the polarization angle 112 and the EVS 130 may provide a high dynamic range, providing advantages related to the motion of objects 104 in the scene 102.

In certain embodiments, the apparatus 100 may also comprise a circular polarizer (a polarizer inducing circular or elliptical polarization states) or a rotatable wave plate positioned between the polarizer 110 and the EVS 130. This may enable manipulating the incoming lights signals carrying polarization information of the scene 102 in further ways, which may be accordingly recorded in the form of events by the EVS 130 and processed by the shape estimation processor 140. Other embodiments of the apparatus 100 may comprise multiple linear and/or circular polarizers. The skilled person having benefit from the present disclosure will appreciate that there are numerous methods to manipulate the polarization of a light signal to extract polarization-based information from the scene 102.

The shape estimation processor 140 may be configured to compute equations relating to Jones parameters or Stokes parameters and perform computation involving Jones or Mueller calculus, which may include calculating the DOP of a light signal reflected off an object 104 in the scene 102. Information related to zenith angles 220 and azimuthal angles 230 of a surface normal 210 within a spherical coordinate system may be calculated. A zenith angle 220 of a surface normal 210 may correspond to an angle of reflection off the object 104 and may be partially determined by polarization information related to the DOP. An azimuthal angle 230 of a surface normal 210 may correspond to the orientation of polarization of the light signal and may be partially determined by polarization information related to the polarization angle of the polarizer 110. The zenith angle 220 and azimuthal angle 230 of a surface normal 210 may each be calculated by information provided by the polarizer 110 and/or polarization selector 120 and the EVS 130. The orientation of a surface normal vector may be determined by its corresponding zenith angle 220, 0, and azimuth angle 230, cp.

The surface normal information 142 may comprise one or more 2-dimensional arrays of pixels representing a map of polarization-based events 132. Each array of pixels may correspond to a polarization angle of the polarizer 110. Information related to the DOP may be determined on a pixel-by-pixel basis, wherein a predetermined pixel generates an ON-event in one array of pixels corresponding to one polarization angle of the polarizer 110 and the predetermined pixel generates an OFF-event in another array of pixels corresponding to another polarization angle. A greater difference in polarization angles of the polarizer 110 between the ON and OFF-event for the pixel may provide data corresponding to the DOP of the light signal corresponding to the pixel, which may provide information related to the zenith angle 220, Q of the light signal corresponding to the pixel. One or more assumptions may be made in the computation of surface normals and in shape reconstruction of an object 104 in the scene 102, which may include but is not limited to assuming that the light incident on the object before reflection is unpolarized, assuming specular dominant or diffuse dominant reflection, assuming the refractive index of the object is wavelength independent and the refractive index of the air space as one, assuming the object is a dielectric or a metal, assuming specific conditions regarding a concavity or convexity of an object, assuming that the object exhibits a smooth surface structure, and assuming that the surface is composed of planar microfacets of random or specific orientations.

Shape from Polarization (SfP) techniques may follow rule-based methods or algorithms applying physics principles and polarization related equations and measurements. Such methods may produce uncertainties or errors related to polarization measurement and the detection of events. Inevitable imperfections in the material of polarizers may lead to a range in quality. One measure of quality in a polarizer is how well the perpendicular axes of maximal and minimal intensity transmission for various light signals are aligned with a 90° angle. Another measure of quality in a polarizer is its polarization extinction ratio (PER), which is a measure of the degree to which light is confined in a principal polarization mode. It is defined as the ratio of the power of the principal polarization mode to the power of the orthogonal polarization mode after propagation through a device or system, usually expressed in decibels (dB).

The extinction ratio of a more common linear polarizer can range from -30 to -40 dB (a ratio of 1000: 1 to 10,000: 1 between Pprindpai and Porthogonai), while specially made polarizers can have a PER even greater than -60 dB (a ratio of 1,000,000: 1). Regardless of how high the PER is, uncertainty and/or errors may be produced in polarization measurements.

While the EVS 130 may offer a significant improvement with a higher time-resolution and a lower latency compared to conventional cameras, it may also exhibit imperfections that lead to uncertainties or errors related to the detection of events. Limitations of performance for the EVS 130 may be caused by latency, jitter and noise sources. Currently, available EVS have a minimum possible temporal resolution of 1 ps. However, in practice, latency, transistor noise and variable readout delays become jitter and noise sources in the data, which may decrease the precision of the event timestamping. Such non-idealities are design- and manufacturerspecific. Latency may be defined as the time it takes for an event to be registered since the moment the logarithmic change in intensity exceeds the threshold. It can currently range on average from a few microseconds to hundreds of milliseconds, depending on bias settings, manufacturing process and illumination level. Transistor noise may be defined as the random transistor noise of the circuits, which may also depend on settings and illumination. This noise randomly changes the measured signal, leading to threshold-comparison jitter.

Other non-idealities may encompass parasitic photocurrents and junction leakages, as these effects bias event generation to one specific polarity. Read-out architectures may be arbitrated architectures, which preserve or partially preserve an order of the pixels’ firing. They may lead to significant queuing delays before a timestamping operation. This may be particularly noticeable when a number of active pixels (and resolution) scale up. Scanning readouts, on the other hand, may limit possible delays by sacrificing event timing resolution. Jitter may be defined as the random variation that appears in timestamps. It may depend on all of the aforementioned factors, all of which increase the unpredictability and imprecision of the event timing.

Fig- 3 illustrates a flow chart of an exemplary method for polarization-based surface normal measurement, as described according to Figs. 1 and 2. The method 300 includes providing 310 the polarizer 110 with a plurality of selectable polarization angles 112. The method 300 further includes selecting 320 a first polarization angle 112A of the polarizer 110 to cause the polarizer 110 to pass light from the scene 102 at the first polarization angle 112A and detecting 330, with the EVS 130, a first set of events 132A associated with the passed light of the first polarization angle 112A of the polarizer 110. The method also includes selecting 340 a second polarization angle 112B to cause the polarizer 110 to pass light from the scene at the second polarization angle 112B and detecting 350, with the EVS 130, a second set of events 132B associated with the passed light of the second polarization angle 112B of the polarizer 110. Additionally, the method 300 includes computing 360 surface normal information 142 of the scene based on the first and second set of events 132 and the corresponding first and second polarization angles 112A; 112Bof the polarizer. The method 300 may optionally include one or more further features according to further embodiments of the apparatus to be described in Figs. 4 to 9.

The method 300 may enable extracting polarization information related to one or more polarizing objects 104 of the scene 102 within the field of view of the EVS 130. In particular, the method 300 incorporating the polarizer 110, the polarization selector 120, the EVS 130 and the shape estimation processor 140 may enable 3D modeling or depth modeling of objects that are widely considered as challenging to model, such as non-Lambertian surfaces or objects with non-Lambertian surfaces.

Given the multipole ambiguities and variables, particularly for non-Lambertian surfaces, the process to solve for a surface normal may be computationally cumbersome and dependent on further information. In addition to algorithms relying on a rule-based method, solving polarization related equations, the shape estimation processor 140 may also use end-to-end machine learning. Embodiments related thereto will be described in Fig. 10. Furthermore in Figs. 4 to 9, other embodiments comprising various features related to the polarizer 110, the polarization selector 120, and the EVS 130 will be discussed, which may enable the apparatus 100 to capture polarization information of the scene 102 more quickly and more efficiently.

Fig. 4 schematically illustrates an apparatus 400 for polarization-based surface normal measurement according to a further embodiment. The apparatus 400 may comprise a polarizer 110 in the form of a liquid crystal polarizer 410 that may be configured to receive an incoming light signal. The liquid crystal polarizer 410 may comprise a liquid crystal layer with a liquid crystal material 416. Additionally, the polarizer 410 may comprise a front plate 414A above the liquid crystal material 416 and a back plate 414B below the liquid crystal material 416. The liquid crystal polarizer 410 may be configured to receive a light signal, such that the light signal first travels through the front plate 414A, then subsequently travels through the liquid crystal material 416, and then subsequently travels through the back plate 414B. The liquid crystal material 416 may be contained by a solid material of a cylindrical form or another form, which is not depicted.

Liquid crystals in general are materials that may exhibit properties of both solids and liquids. Liquid crystal molecules may flow like a liquid, but its molecules may be oriented in a crystallike way. This is partly because liquid crystal molecules have a particular elongated or rodlike shape. The orientation of the liquid crystal molecules may be controlled or influenced by various means, each of which may be used to manipulate polarization information of an incoming light signal in a specific way. In particular, the molecules of the liquid crystal material 416 of the liquid crystal polarizer 410 may be controlled by an externally applied electric field. The apparatus 400 may comprise one or more pairs of electrodes 412. While Fig. 4 depicts four pairs of electrodes 412 within a common plane, any number of two or more electrodes may surround the liquid crystal polarizer 410 and in any configuration. In one possible configuration, one or more pairs of electrodes may surround the liquid crystal material 416, some or all of which may be in a common plane. In other words, the liquid crystal material 416 may be positioned in between a first electrode 412A and a second electrode 412B of one or more respective pairs of electrodes.

In such an embodiment, the polarization selector 120 may be in the form of an electrode actuator 420. The electrode actuator 420 may be configured to send a voltage signal to a respective pair of electrodes 412 to activate them, which may thereby induce an electric field between them, including in the liquid crystal material 416. This may cause the molecules of the liquid crystal material 416, specifically positioned between the electrodes 412A; 412B, to be aligned with the electric field. The position of the first pair of electrodes 412A; 412B relative to each other and the liquid crystal material 416 may control the angle of alignment of the molecules of the liquid crystal material 416. In particular, the alignment of the molecules of the liquid crystal material 416 may allow the liquid crystal material 416 to act as a polarizer for the incoming light signal, particularly with a direction of polarization aligned with the electric field that induced the alignment of the molecules. The electrode actuator 420 is depicted in Fig. 4 as currently selecting a first pair of electrodes 412A; 412B by sending a voltage signal to activate them.

The electrode actuator 420 may be configured to select the first polarization angle 112A of the liquid crystal polarizer 410 by sending a voltage signal to the first pair of electrodes 412A; 412B and activating them. The liquid crystal polarizer 410 may further comprise a second pair of electrodes 412C; 412D . The liquid crystal material 416 may also be positioned in between a first electrode 412C and a second electrode 412D of the second pair of electrodes. The second pair of electrodes may be similarly positioned around the liquid crystal material 416, which may be in a common plane with the first pair 412A; 412B, but aligned at a different angle, as depicted. The polarization selector 120 may be configured to deactivate the first pair of electrodes 412A; 412B and its corresponding electric field and to subsequently activate the second pair of electrodes 412C; 412D and its corresponding electric field. This may cause the direction of the molecules of the liquid crystal material 416 to change to be aligned with the electric field induced by the second pair of electrodes 412C; 412D. The re-alignment of the molecules may thus change the polarization angle 112 of the liquid crystal polarizer 410 to a second polarization angle 112B in analogous fashion. Further pairs of electrodes may re-align the molecules of the liquid crystal material 416 to further corresponding induced electric fields, which may correspond to further polarization angles 112 in analogous fashion.

The time for a re-alignment of the molecules for each change in direction of the electric field may vary from microseconds to milliseconds, which may depend on the strength of the induced electric field (which may depend on the voltage signal), properties of the liquid crystal material 416, and the temperature of the liquid crystal material 416, among other factors. Related properties of the apparatus 400 may be chosen to enable an especially short time of re-alignment to enable the apparatus 400 to capture more polarization information of the scene 102 in a shorter time period. This may also enable the apparatus 400 to capture shape information of faster moving objects 104 in the scene 102.

In the apparatus 400, the liquid crystal polarizer 410 is positioned upstream to the EVS 130. With at least a first and second pair of electrodes 412A; 412B; 412C; 412D configured as described above, the liquid crystal polarizer 410 may receive a partially polarized light signal reflected from an object 104 of the scene 102 and polarize the light signal in at least two predefined polarization angles 112. The electrode actuator 420 may be configured to control a voltage signal to activate a respective pair of electrodes 412 in order to select a polarization angle 112 for the incoming light signal to be polarized at that polarization angle 112. The liquid crystal polarizer 410 may continue to receive incoming light signals reflected from the object 104 while the electrode actuator 420 is deactivating and activating respective electrode pairs to switch the polarization angle 112. This may enable the EVS 130 to detect multiple sets of events 132 caused by a change in intensity associated with the respective polarization angle 112, as previously described.

In general, any number of electrodes 412 may be positioned around the liquid crystal material 416 to be activated and deactivated by the electrode actuator 420 to enable the molecules of the liquid crystal material 416 to be aligned in any number of predefined directions. With a larger number of electrodes 412, multiple combinations of electrodes 412 may be used. For example, for a large number of electrodes positioned within a common plane, two pairs of activated electrodes may form respective electric fields that are parallel. In such a configuration, four electrodes may be activated simultaneously to select a single polarization angle 112 of the liquid crystal polarizer 410. Furthermore, two or more pairs of electrodes 412 may be activated, such that their combined electric field induces an entirely new orientation for the molecules of the liquid crystal material 416, which may allow a greater number of polarization angles 112. The skilled person will appreciate that there are many possible configurations for multiple electrodes 412 to be positioned and activated to allow any number of predefined polarization angles 112 to be selected by the electrode actuator 420.

Alternatively, instead of selecting the polarization angle 112 according to multiple fixed pairs of electrodes 412, the apparatus 400 may be configured to change the respective positions of two or more electrodes 412 to enable the selection of further polarization angles 112. This may enable a greater number of predefined polarization angles 112 to be selected by the electrode actuator 420. For example, the electrode actuator 412 may be configured to be vertically or horizontally translated through space or to rotate around the liquid crystal material 416 to any number of predefined rotation angles 112. Such changes of position for the electrodes 412 may enable the electrodes 412 to induce an electric field in a new direction and may thus enable any number of predefined polarization angles 112 for the liquid crystal polarizer 410. The configuration of the electrodes 412 with a fixed and/or adjustable position may be chosen based on the detection capabilities of the EVS 130 and/or the processing capabilities of the shape estimation processor 140.

As previously described, the incoming light signal reflected from an object 104 in the scene 102 carries polarization information. By polarizing the light signal in a known polarization direction, the light intensity of the light signal may change in a specific way, which may encode the polarization information according to the polarization angle 112. The changes in intensity may be detected in the form of events 132 by the EVS 130 and, which may sent to the shape estimation processor 140 with the corresponding polarization angles 112 to extract the encoded polarization information.

In a particular embodiment, the liquid crystal polarizer 410 may comprise features to ensure that this polarization information is maintained while traveling through the entire liquid crystal layer. For example, the electrodes 412 may comprise a height similar to a height of the liquid crystal layer. In such a case, the electrodes 412 may be configured to induce an electric field within the entire liquid crystal material 416. On the other hand, if only a portion of the molecules in the liquid crystal material 416 are aligned with the electric field induced by an activated pair of electrodes 412 and other portions are not aligned, then the polarization of the light signal may be altered as it travels through the liquid crystal material 416 and polarization information may be lost. For example, this may be the case if the electrodes 412 comprise a significantly smaller height compared to the liquid crystal layer.

In the case that only a portion of the molecules in the liquid crystal material 416 may align with the induced electric field, the alignment of the rest of the molecules may be controlled by different means in a way that also maintains the polarization information. For example, the orientation of the molecules of the liquid crystal material 416 may be influenced by shaping the inner surfaces of the liquid crystal layer that are in contact with the liquid crystal material 416.

In general, molecules of a liquid crystal may orient themselves according the orientation of neighboring molecules. The molecules may flow freely to re-orient themselves based on a new surrounding environment and may maintain a crystalline structure until the surrounding environment changes again. This property may be exploited to control the orientation of the molecules through the use of externally applied electric fields, as previously described, but also by having a portion of the molecules maintain contact with a customized surface.

A surface may be manufactured with a specific pattern to align liquid crystal molecules in contact with or near the surface. In particular, the inner surface of the back plate 414B may comprise parallel gratings, grooves, or grating grooves. Gratings may be a series of parallel lines or ridges that are evenly spaced and of uniform width, while grooves may be narrow, elongated channels or trenches that are etched or patterned onto a surface. Grating grooves may comprise a combination of the properties of gratings and grooves. Grating grooves may comprise parallel grooves in a pattern of a grating, being evenly spaced and of uniform width. Grating grooves may have a smooth surface between the grooves, which may allow for a more uniform alignment of the liquid crystal molecules. Alternatively, a structure of a grating may be designed in a different way to also allow for a uniform alignment of the liquid crystal molecules.

The back plate 414B, which may comprise grating grooves, may cause the molecules of the liquid crystal material 416 in contact with or near the back plate 414B to align in a parallel direction along the grating grooves. Since liquid crystal molecules align with the neighboring liquid crystal molecules, the liquid crystal material 416 may form a pattern with a gradually changing angle of alignment between the molecules aligned with an induced electric field of activated electrodes 412 and the molecules aligned with the grating grooves of the back plate 414B. This is depicted in Fig. 4 with the molecules of the liquid crystal material 416 being aligned with the activated pair of electrodes 412A; 412B and the molecules in contact with the back plate 414B being aligned in a perpendicular direction, as depicted. While the direction of alignment is depicted on the back plate 414B, the grating grooves of the back plate 414B are not depicted. The orientation of the molecules is shown to change gradually, such that the orientation of molecules at a certain depth of the liquid crystal layer is closely, although not completely aligned with the molecules of neighboring depths. Upon deactivating one pair of electrodes 412 and activating another pair of electrodes 412, the molecules between the activated electrode pair 412 may align with the newly induced electric field, while the molecules near the back plate 414B may remain with the same orientation.

The liquid crystal polarizer 410 may be configured with the front plate 414A not comprising any such pattern, such that the incoming light signal of partially polarized light corresponding to the object 104 in the scene 102 remains unaltered. The still unaltered light signal may then be polarized (altered) at a known polarization angle 112 selected by the electrode actuator 420. This may encode the polarization information of the incoming light signal in a change of intensity according to the selected polarization angle, as previously described. After passing the portion of the liquid crystal material 416 that is aligned with the activated electrodes 412, the polarization state of the light signal may be gradually rotated according to the gradually changing angle of alignment of the liquid crystal molecules. However, while modifying a polarization state of the light signal, this may maintain the encoded polarization information, since no further changes in intensity occur. Since the molecules of the liquid crystal material 416 in contact with the back plate are parallel to its grating grooves, it may pass through back plate 414B without any change in intensity. In other words, the light signal that has been polarized by the induced electric field, and thus having changed intensity to encode information related to the polarization angle of the pair of actuators, may pass the back plate 414B without any further change in intensity.

The EVS 130 positioned behind the liquid crystal polarizer 410 may thus detect a change in intensity associated with each respective polarization angle selected by the electrode actuator 420, and the shape estimation processor 140 may extract the encoded polarization information to perform SfP processing. Embodiments using the liquid crystal polarizer 410 may efficiently record accurate data and may be configured in a variety of ways, which may have a particular advantage of not using any mechanical mechanisms for switching polarization states. This may particularly enable a higher switching rate between polarization angles 112 without sacrificing an accuracy of setting the polarization angle 112 within an acceptable range of error.

It may be desirable for embodiments to switch the polarization angle in a simpler configuration, depending on the type of scene to be analyzed for polarization information. For example, instead of re-orienting a polarizing material, as with the embodiment using the liquid crystal polarizer, other embodiments may re-direct an incoming light signal carrying the polarization information. One such embodiment will be described in relation to Fig. 5.

Fig. 5 schematically illustrates an apparatus 500 for polarization-based surface normal measurement according to a further embodiment. The polarizer 110 of the apparatus 500 may comprise a plurality of polarizer sections 512A; 512B; 512C; 512D. Each of the polarizer sections 512 may be fixed at a pre-specified polarization angle 112. Two or more polarizer sections 512 may be physically connected as part of a single unit, such as a single film. Two or more polarizer sections 512 may also be physically separate and positioned next to each other, such as the four polarizer sections depicted in Fig. 5. To select a polarization angle 112, the polarization selector 120 may be configured to direct light from the scene 102 to a corresponding polarizer section 512. For controlling the path of a light signal from the scene 102 toward the EVS 130, the polarization selector 120 of the apparatus 500 may comprise one or more reflective surfaces 522, such as the reflective surfaces 522A and 522B depicted in Fig. 5. The reflective surface may be a flat mirror. In general, the reflective surface may be a flat surface reflecting a large amount of incident light in a specular fashion.

The polarization selector 120 of the apparatus 500 may further comprise an actuation mechanism 524 to create a motion of the reflective surface 522 or an object physically connected to the reflective surface 522. The actuation mechanism 524 may be in the form of an electrostatic actuation mechanism 524A, which may rely on an attraction and/or repulsion of charged particles. For example, a force generated by electrodes may be used to move a charged body. The actuation mechanism 524 may also be in the form of an electromagnetic actuation mechanism 524B, which may use magnetic fields. For example, a force may be generated using an interaction between magnetic fields, such as magnetic fields between two or more electromagnets. The electrostatic actuation mechanism 524A may be used for actuation requiring more precise control, while the electromagnetic actuation mechanism 524B may be used for actuation requiring a larger degree of force. The polarization selector 120 may be configured to switch the orientation of the reflective surface by means of the actuation mechanism 524, such that the light signal is re-directed to a different polarizing section of the polarizer 110. The actuation mechanism 524 may combine multiple forms of each type of actuation mechanism 524A; 524B to provide a fast and precise switching between polarization angles 112 by the polarization selector 120.

In a particular embodiment, one or more of the reflective surfaces 522 may be a micro-electro- mechanical system, MEMS, mirror. The polarization selector 120 may be electrically or communicatively connected to the MEMS mirror and may control it by means of the actuation mechanism 524. The reflective surface 522 configured as a MEMS mirror may have multiple benefits for the apparatus 500, which may include a faster response time, a smaller size, a lower power consumption, a greater precision in directing light signals, and increased durability. The MEMS mirror may be configured to redirect an incoming light signal to a different polarizer section on a scale of many nanoseconds to milliseconds, depending on its design.

The apparatus 500 may comprise an array of one or more focusing lens 580. For example, the apparatus 500 may comprise a focusing lens for each polarizing section. The four polarizing sections 512A; 512B; 512C; 512D may each comprise a corresponding focusing lens 580A; 580B; 580C; 580D, as depicted. The array of focusing lens 580 may enable a more compact configuration of the apparatus 500. In particular, a light signal may be focused onto a compact area of detection of the EVS 130. The apparatus 500 may also comprise a collimator 582. The collimator 582 may be configured to receive a light signal from the scene 102 from a wide angle and to pass the received light signal in a collimated form. Such a collimated form may provide greater stability to a light beam carrying a light signal traveling through the apparatus 500 and may enable the actuation mechanism 524 to manipulate a path of the light signal more easily toward one of the polarizing sections. A collimated form of an incoming light signal may travel through the polarizing section, through the corresponding focusing lens 580, and to the detection surface of the EVS 130, as depicted. Certain embodiments of the apparatus, such as those described in Figs. 4 and 5, offer a particular advantage of mechanical stability. They may be able to switch between multiple selectable polarization angles at a particularly high switching rate. It may be desirable for embodiments to switch the polarization angle within a larger set of selectable polarization angles For example, an embodiment may enable a much larger selection of polarization angles, which may be configured to more bases of comparison to capture polarization information of the scene 102 One such embodiment will be described in relation to Fig. 6.

Fig. 6 schematically illustrates a further embodiment of the apparatus 600 for polarizationbased surface normal measurement. The apparatus 600 may comprise a polarizer gear 624 that may be configured to hold, house, or support a rotatable polarizer 610. The polarizer gear 624 may be physically connected to a powered gear 622, such that a rotation of the powered gear 622 correspondingly causes a rotation of the polarizer gear 624 and the rotatable polarizer 610. In other words, the powered gear 622 and the polarizer gear 624 may be meshed or interlocked. Optionally, the apparatus 600 may comprise further gears, which may also be meshed or interlocked with the powered gear 622 and/or the polarizer gear 624. The polarizer gear 624 and rotatable polarizer 610 may be configured to rotate about a rotational axis 618, which may be aligned with a central location of a detection surface of the EVS 130.

The polarization selector 120 and its interaction with the rotatable polarizer 610 may come in various example implementations. The polarization selector 120 may comprise an electric drive or motor, which may be connected to the powered gear 622 and cause a stable rotation of the powered gear 622, as well as the rotatable polarizer 610 based on a selected rotational speed. The rotatable polarizer 610 may maintain a stable alignment with the field of view of the EVS 130 while being rotated by the electric drive or motor. The electric drive or motor may also be physically connected to a moveable belt and may be configured to rotate the belt in a way that causes a rotation of the powered gear 622 and polarizer gear 624 in analogous fashion. In other embodiments, a rotor of the electric drive/motor may act as a rotation axis 612 of the rotatable polarizer 610. The electric drive/motor may be electronically connected to the EVS 130 or the shape estimation processor 140, such that the set of events 132 detected by the EVS 130 is recorded together with the corresponding rotation angle of the polarizer gear 624 and/or polarization angle 112 of the rotatable polarizer 610. The polarization selector 120 may be configured to select a first polarization angle 112A by rotating the rotatable polarizer 610 by means of the powered gear 622 to a first rotation state. The polarization selector 120 may further be configured to subsequently select a second polarization angle 112B by subsequently rotating the rotatable polarizer 610 to a second rotation state. In general, the rotatable polarizer 610 is configured to pass light from the scene 102 at a plurality of rotation states or rotation angles of the polarizer gear 624 and thus, a plurality of polarization angles 112 of the rotatable polarizer 610.

While in a rotating state, the polarizer gear 624 and rotatable polarizer 610 may rotate clockwise or counterclockwise, which may be caused by the powered gear rotating in the opposite or same direction. A full rotation of the rotatable polarizer 610 may span rotation angles from 0° to 360°. The first rotation angle may be any rotation angle between 0° and 360°. The second rotation angle may also be any rotation angle between 0° and 360° but different from the first rotation angle. The skilled person will appreciate that a full rotation of the rotatable polarizer 610 may span a discrete or continuous set of rotation angles and that a polarization angle of the rotatable polarizer 610 may correspond to two rotation angles of the polarization gear 624 that differ by 180°.

The rotational polarizer may take many geometrical forms. For example, the rotatable polarizer 610 may span a 2-dimensional (2D) plane. An outer perimeter of the rotatable polarizer 610 may be rotationally symmetric with respect to a rotational axis 612 perpendicular to the 2D plane spanned by the linear polarizer 610. The material of the linear polarizer 110 may also have a 3D form, which may include a curved surface. The rotational axis 612 may correspond to or may be parallel to an optical axis of the EVS 130.

The various embodiments shown in Fig. 4 to 6 each comprise features that enable a switching of polarization angles within a plurality of selectable polarization angles. This may be considered as a division of time approach since polarization information of the scene is being encoded with various polarization angles differently through time. The apparatus 100 may also be adapted to incorporate a division of focal plane approach. An embodiment incorporating both approaches will be explained in relation to Fig. 7.

Fig. 7 schematically illustrates an apparatus 700 for polarization-based surface normal measurement according to further embodiments with an enhanced division of time approach and/or division of focal plane approach. Such approaches may enable the shape estimation processor to generate more surface normal information 142 in a given time period.

The apparatus 700 may comprise a plurality of polarizers. For example, Fig. 7 depicts the plurality of polarizers as comprising four polarizers 710A; 710B; 710C; 710D. In general, the plurality of polarizers 710 may be any number of polarizers in any combination of any type of polarizer. For example, the four polarizers may be configured as liquid crystal polarizers described in Fig. 4, as depicted, or other forms of polarizers.

The plurality of polarizers 710 may be positioned before a focusing lens 732 positioned upstream to a detection surface of the EVS 130. For example, the apparatus 700 may comprise a plurality of polarizers 710 spatially separated within a common plane, such as the four polarizers 710 depicted in Fig. 7. Each of the four polarizers 710 may simultaneously receive a corresponding light signal of the scene 102, which may contain a partially polarized light signal generated as unpolarized light reflected off a polarizing object 104 in the scene 102. Since each of the four polarizers 710 are placed before the focusing lens 732, incident light corresponding to a specific object 104 in the scene 102 may be received by each polarizer 710. In such a configuration, the EVS 130 may receive light from each of the four polarizers 710 simultaneously. Furthermore, the incoming light signal may be received by each of the four polarizers 710 when each is oriented at a polarization angle that is unique amongst the four polarizers. The polarization selector 120 may be configured to control a coordinated switching of polarization angles, such that each polarizer is always switched to a polarization angle that is unique amongst the four polarizers. Such a multiplexed polarization switching (or rotating) feature may also be utilized with a division of focal plane approach. Details of configurations of the apparatus 700 related to the multiplexed polarization switching will be given below, followed by a detailed explanation of a division of focal plane approach and how these two approaches may be used together through various combinations.

Optionally, in some embodiments, each of the four polarizers 710 may have four selectable polarization angles 112. For example, each polarizer 710 may provide selectable polarization angles 112 at 0°, 45°, 90° and 135°, respectively, as depicted by the four pairs of electrodes for each liquid crystal polarizer in Fig. 7. With a single polarizer, an incoming partially polarized light signal may only be encoded using a single polarization angle at a time. However, with four polarizers 710, such as in Fig. 7, the light signal may be examined at one time based on four different polarization angles 112. In such a configuration, the light signal may be encoded with polarization information according to four polarization angles at a particular instant as opposed to one polarization angle. Optionally, this feature may be used to obtain more information related to a shape of an object at a particular instant.

In such a configuration, each of the portions of the detection surface corresponding to a different polarizer 710 may simultaneously capture different information of the scene 102. Optionally, for this feature, the apparatus 700 may further comprise a plurality of polarization selectors 720, each of which may correspond to a polarizer of the plurality of polarizers 710. For example, Fig. 7 depicts each of the plurality of polarizers 710A; 710B; 710C; 710D. with a corresponding polarization selector 720A; 720B; 720C; 720D. For example, the polarization selectors 720 may be configured as electrode actuators corresponding to a liquid crystal polarizer, as depicted, or configured with another form that is suitable for its corresponding polarizer 710. For example, the polarization selector 720 may be a reflective surface, such as a MEMS mirror, for a polarizer with multiple polarizing sections or the polarization selector 720 may be a powered gear for a rotatable polarizer.

In such a configuration, each polarization selector 720 may be optionally configured to control a switching of polarization angles for its corresponding polarizer 710. In other words, each polarization selector 720 may be physically, electrically, and/or communicatively connected to its corresponding polarizer 710 and configured to switch its corresponding polarizer 710 from a respective first polarization angle 112A to a respective second polarization angle 112B and from a respective second polarization angle 112B to a respective third polarization angle 112C, etc.

In such a configuration, each polarizer 710 and corresponding polarization selector 720 may also be configured differently in comparison to one another. For example, one polarizer may have one set of selectable polarization angles 112, while another polarizer may have a different second set of selectable polarization angles 112. Furthermore, each of the polarizers 710 may be controlled differently by its corresponding polarization selector 720. For example, polarization selectors 720 may control the corresponding polarizer 710 with a different switching speed between selectable polarizer angles 112 or with a different pattern of switching. Each polarization selector 720 may be programmed differently or programmed in a coherent fashion. In one embodiment, the apparatus 700 may comprise a master polarization selector that is communicatively connectable to each of the plurality of polarization selectors 720. For example, the master polarization selector may be configured to customize features related to polarization switching for each polarizer 710. As such, each of the polarizers 710 may be controlled independently of one another by the master polarization selector. Alternatively, the master polarization selector may control the polarizers 710 to be in sync with each other, such as having the respective polarization angle 112 switched within the same set of selectable polarization angles 112. The polarization angles 112 may be switched by the master polarization selector at the same rate and exhibit the same pattern. For example, each polarizer 710 may have polarization angles 112 switched at the same rate, but in a staggered pattern, such that each polarizer 710 at each instant has a unique polarization within the plurality of polarizers 710, which may enable greater capture of polarization information in a given time interval.

In such a configuration, the master polarization selector may be configured to change settings related to polarization switching for each polarizer 710 according to the scene 102. For example, after detecting events from the scene 102 and performing SfP measurements, the shape estimation processor 104 may provide feedback to the master polarization selector. The feedback may include instructions to re-program settings related to the polarization switching for each polarizer 710 based on its initial measurements. Such processes may be repeated to optimize the SfP measurements. In general, the apparatus 700 may be configured to cause a selection of specific polarization angles 112 for each of the plurality of polarizers 710 in way that enables the shape estimation processor to obtain more polarization information of the scene 102 in a given time span. This may also enable the apparatus 700 to capture polarization information of faster moving objects.

Some embodiments of the apparatus 700, either with a single polarizer 710 or a plurality of polarizers 710, may comprise a linearly polarizing structure 770 that may be used to provide a division of focal plane approach. Embodiments comprising the linearly polarizing structure 770 with a single polarizer 110 may be related to polarizers presented in Figs. 4 to 6 or any other form of a polarizer. The linearly polarizing structure 770 may be positioned between the single polarizer 110 and the EVS 130 (not shown). In alternative embodiments of the apparatus 700 comprising any form of a plurality of polarizers, such as the four polarizers depicted in Fig. 7, the linearly polarizing structure may be positioned between the plurality of polarizers and the EVS 130. Embodiments comprising the linearly polarizing structure 770, with either a single polarizer or a plurality of polarizers, may provide advantages in detection of polarization information to be used by the shape estimation processor 140, particularly with different features using a division of focal plane approach.

In some embodiments, the linearly polarizing structure 770 may be a linearly polarizing film 770. The linearly polarizing film 770 may be divided into multiple sections. For example, Fig. 7 depicts an embodiment with the linearly polarizing film 770 comprising four sections 770A; 770B; 770C; 770D. In some embodiments, each section of the linearly polarizing film 770 may be further divided into a number of subsections. Each of the four depicted sections 770A; 770B; 770C; 770D are shown to comprise four subsections. Each subsection may be fixed at a pre-specified polarization angle that is unique within the section, as depicted in Fig. 7. The four subsections of each section are shown to be oriented to polarize light at 0°, 45°, 90° and 135°, respectively. In general, the linearly polarizing film 770 may comprise a plurality of sections including any number of sections in total and any number of subsections within each section. The number of subsections within the plurality of sections may vary or be the same.

The linearly polarizing film 770 comprising a plurality of sections and subsections, as outlined above, may provide multiple bases of measurement. Each basis of measurement may be based on a particular pre-specified polarization angle of the linearly polarizing film 770. Using the example of the subsections above, the four bases of measurement may be based on the four respective polarization angles of 0°, 45°, 90°, and 135°. The four bases of measurement may be evenly distributed throughout the detection surface of the EVS 130. For example, all subsections oriented at 0° may provide a basis of measurement oriented at 0° for incoming light signals provided from each of the polarizers 710. Thus the linearly polarizing film may provide options related to a division of focal plane approach.

The subsections fixed at a polarization angle unique within its respective section may provide a division of focal plane approach. In other words, the detection surface (or focal plane of detection) of the EVS 130 may record polarization events 132 under a basis of four polarization angles that correspond to pixels that are evenly distributed throughout the detection surface of the EVS 130. The pixels being distributed more evenly by basis of polarization throughout the detection surface may enable obtaining more evenly spread light signal information corresponding to the scene 102 according to each basis.

The size of the sections and subsections relative to the pixels of the EVS may vary. For example, each of the four subsections of each section may uniquely correspond to (e.g. cover) a plurality of pixels of the EVS 130. Alternatively, in some embodiments, each subsection may uniquely correspond to (e.g. cover) a single pixel of the EVS 130. In such embodiments, the EVS 130 may comprise a detection surface with a pixel array 734 with each pixel corresponding to a subsection of the linearly polarizing film 770. Each corresponding subsection may be fixed at a pre-specified polarization angle that is unique within its section. Thus, each basis of measurement may have a collection of corresponding pixels that are evenly distributed throughout the detection surface of the EVS 130. In the case of one subsection uniquely corresponding (covering) a single pixel, each basis of measurement may be as evenly distributed throughout the detection surface as the resolution of the EVS 130 allows. This may provide an even more balanced division of the focal plane, which may enable smaller portions of each polarizing object 104 of the scene 102 to be individually examined under multiple bases of comparison simultaneously.

In some embodiments, the linearly polarizing film 770 may be configured as an on-chip linearly polarizing film 770. This may enable a greater precision of directing a light signal through a specific subsection to a specific corresponding pixel. In other words, this may enable each pixel of the EVS 130 under the film to selectively receive light that has been polarized only by a corresponding subsection of the linearly polarizing film 770 that is oriented at a pre-specified polarization angle. In embodiments with a pixel-wise difference of polarization angles of the linearly polarizing structure 770, neighboring pixels may receive differently polarized light simultaneously. In general, configurations enabling a pixel-wise difference of polarization angle may retain a greater amount of polarization information of the scene 102.

To enable an adequate separation of the light signal among neighboring pixels, the apparatus 700 may comprise a micro-lens array 780 located downstream to the polarizer array 710 and focusing lens 732 and upstream to the linearly polarizing film 770 and EVS 130. The microlens array 780 may be configured to focus various portions of the light signal through specific subsections of the linearly polarizing film 770. Each micro-lens of the micro-lens array 780 may focus a portion of the light signal to a corresponding subsection. In some embodiments, each micro-lens in the micro-lens array 780 may correspond to a plurality of pixels, while in other embodiments, each micro-lens of the micro-lens array 780 may correspond to a single pixel.

Furthermore, based on a division of time approach, the at least one polarizer 110 may have its polarization angle switching through time, as previously described. Each pixel of the pixel array 734 may be aligned to receive light that passes through a polarizer 710 at a predefined selectable angle (selected by the corresponding polarization selector 720) and then subsequently passes a subsection of the linearly polarizing film 770 that is fixed at a pre-specified angle. Thus, the apparatus 700 may enable a combined division of focal plane and division of time approach, which may be customized in a way to optimize computation of surface normals by the shape estimation processor 140.

In a particular embodiment, the apparatus 700 may comprise a plurality of polarizers including at least two liquid crystal polarizers 710. Each of the at least two liquid crystal polarizers may comprise a combination of features of the liquid crystal polarizer 410 described in Fig. 4. In particular, this may include each liquid crystal comprising a respective back plate with either a grating, grooves, or grating grooves, which may align the molecules of the respective liquid crystal material in a particular direction. Thus a light signal from the scene 102 may travel through the respective front plate and then be polarized by the respective liquid crystal material being aligned by an electric field induced by a pair of electrodes surrounding the liquid crystal polarizer 710. This may encode polarization information. The now altered light signal with encoded polarization information may have its polarization state rotated without loss of information, as previously described in Fig. 4, caused by the parallel grating grooves (or grating or grooves) of the transparent back plate interacting with the liquid crystal material. The altered light signal may then travel through the respective back plate of the respective liquid crystal polarizer.

Furthermore, the parallel grating grooves of each respective back plate may be oriented parallel to the pre-specified polarization angle of a corresponding subsection within the sections of the linearly polarizing structure. It is important to note that in such a configuration, regardless of which polarization angle 112 that the respective polarizer selector 120 (electrode actuator 420) selects for the respective polarizer 710 through time, that the polarization state of continually passed light signals upon passing the back plate remains at the same angle. For example, if the respective parallel grating grooves are oriented at an angle of 90°, then all light signal outputs, regardless of the selected polarization angle 112 used to encode the polarization information, will have its polarization state parallel to the parallel grating grooves at 90° upon passing the back plate. Furthermore, light signals leaving the respective back plate oriented at 90° will remain corresponding to the subsections of the plurality of sections also oriented at 90°. The subsections at 90° will pass the corresponding light signals, which are polarized at 90°, to the EVS 130 without changing the light signal.

In a particular configuration of four liquid crystal polarizers, the four respective back plates may each comprise parallel grating grooves, which may be oriented at 0°, 45°, 90°, and 135°, respectively. Thus, each of the liquid crystal polarizers may correspond to up to one quarter of the subsections of the plurality of sections. For embodiments wherein each subsection uniquely corresponds to a pixel of the EVS 130, each liquid crystal polarizer may correspond to up to a quarter of the pixels of the EVS 130.

In a simpler configuration of two liquid crystal polarizers, the two respective back plates may each comprise parallel grating grooves, which may be oriented at 0° and 90°, respectively. If each section of the linearly polarizer film 770 comprises two subsections, also oriented at 0° and 90°, then each of the liquid crystal polarizers may correspond to up to half of the subsections of the plurality of sections.

In some embodiments, the light signals leaving the back plate of each of the at least two liquid crystal polarizers may be directed to overlapping regions of the EVS 130. In particular, for such a configuration with 0° and 90°, the lights signals leaving the back plates in a polarization state of 0° or 90° are polarized in orthogonal directions. This may prevent or greatly reduce interference between the light signals of the respective liquid crystal polarizers. In particular, this may enable an enhanced division of focal plane approach, wherein every pixel of the EVS 130 may be used for every round of detection, preventing a degradation of spatial resolution that normally accompanies a division of focal plane approach.

Such overlapping regions may also be used for a configuration with four liquid crystal polarizers with respective grating grooves at 0°, 45°, 90°, and 135° with further processing adjustments by the shape estimation processor 140. Furthermore, many combinations with the aforementioned features are possible. For example, the apparatus 700 may comprise any number of liquid crystal polarizers 710, with grating grooves oriented at a particular angle, while the selectable polarization angle may be switched through time in a manner customized for each liquid crystal polarizer. In general, the aforementioned features related to liquid crystal polarizers may enable uniquely enhanced features for combining a division of time and division of focal plane approach. The division of time may be enhanced by customized polarization switching and the division of focal plane may be enhanced by preventing or reducing the degradation of spatial resolution using a linearly polarizing structure 770 and/or parallel grating grooves.

The multiple aforementioned features of the apparatus 700 relate to improvement of SfP measurements with a combination of a division of focal plane approach or division of time approach. These may done in many different combinations to provide a wide range of possible configurations for capturing polarization information of the scene 102, particularly in the case that the scene 102 includes fast moving objects 104. Such configurations may allow more information to be captured for each round of event detection. In this case, a division of focal plane approach may enable slower switching speeds of polarization angles. In other words, incorporating a division of focal plane approach may relax the requirements of using a division of time approach. Such features may be particularly advantageous for embodiments that are limited in switching speed between polarization angles, such as by a mechanical instability or limits within electrical instrumentation.

In addition to the division of focal plane approach, it may also be advantageous by further enhancing the division of time approach.

Fig. 8 schematically illustrates an apparatus 800 for polarization-based surface normal measurement according to a further embodiment. The apparatus 800 may comprise a 2^nd EVS 830. The 2^nd EVS 830, as depicted, is not behind any polarizer or any device that changes or manipulates light reflecting from the scene 102. As such, the 2^nd EVS may be configured to directly detect changes in light intensity of the scene 102. For example, the 2^nd EVS may be configured to detect changes in absolute light intensity or relative light intensity, without changes in polarization. In particular, the 2^nd EVS 830 may be configured to detect a motion of one or more objects 104 in the scene 102 or one or more areas of motion in the scene 102. The motion in the scene 102 may be relative to the background or relative to another static or moving object in the scene 102. The motion may be in any direction with respect to the 2^nd EVS 830, which is detected when the motion is in an area within a field of view of the 2^nd EVS 830. Events produced by the 2^nd EVS 830 may be tuned to detect temporal changes in the brightness of the scene 102. These motion events 832 may effectively locate moving objects in the scene 102, which are associated with regions of interest.

The polarization selector 120 of the apparatus 800 may further comprise a controlling circuit 826 that may be connected to the EVS 830. The controlling circuit 826 may be configured to control a switching speed between the first polarization angle 112A and the second polarization angle 112B of the polarizer 110 based on (e.g. in proportionality to) information obtained by the 2^nd EVS 830 related to a motion or an area of motion in the scene 102. The 2^nd EVS 830, like the EVS 130 behind the polarizer 110, may obtain information of the scene 102 in the form of events. Since an EVS is not configured to record events of a completely static scene and the events may refer directly to a motion within the scene 102, the events recorded by the 2^nd EVS may be referred to as motion events 832.

The controlling circuit 826 may be configured to monitor how many motion events 832 are generated by the 2^nd EVS 830 in a predefined time interval and to control the switching speed between the first and second polarization angles 112A; 112B of the polarizer 110 based on (e.g. in proportionality to) the number of motion events 832 in the predefined time interval. The switching of polarization angles 112 may include the switching mechanisms outlined in the previous embodiments, such as switching activation of electrodes 412, redirecting a path of the light signal to a different polarizing section 610, and rotating a rotatable polarizer 710, among other possible polarization switching mechanisms.

For this purpose, the controlling circuit 826 of the polarization selector 120 may be coupled with a controllable actuator configured to cause the switching of polarization angles 112 from a first to a second polarization angle 112 A; 112B. The controlling circuit 826 may be configured to control a switching speed between various polarization angles 112 at a rate directly proportional to the number of motion events 832 detected by the 2^nd EVS 830 in such a predefined time interval. The controlling circuit 826 may be configured to increase the switching speed between polarization angles 112 when an amount of motion detected by the 2^nd EVS 830 increases and to decrease the switching speed when an amount of motion detected by the 2^nd EVS 830 decreases. The controlling circuit 826 may be configured to apply any mathematical function to determine the switching speed. For example, polarization selector 120 may be configured to use a mathematical function to receive an input of a number of motion events 832 recorded within one or more units of the predefined time interval and to generate an output of a desired switching speed of the polarizer 110.

In a particular configuration, the controlling circuit 826 of the apparatus 800 may be coupled to the powered gear 722 depicted in Fig. 7 to rotate the powered gear 722 at a chosen rotational speed. As previously described, the powered gear 722 may be in interface contact with the polarization gear 724 housing the polarizer 110, which may cause the polarizer 110 to rotate based on (e.g. in proportionality to) a rotation of the powered gear 722. The controlling circuit 826 may be configured to control the rotational speed of the powered gear 722, and thus a rotational speed of the polarizer 110. The controlling circuit 320 may be configured to increase or decrease a rotational speed of the linear polarizer 110 in direct proportionality to an increase or decrease in the number of events generated by the 2^nd EVS 830 within the corresponding unit of the predefined time interval, respectively. In other words, the controlling circuit 320 may be configured to increase the rotational speed of the linear polarizer 110 when an amount of motion detected by the second EVS increases and to decrease the rotational speed of the linear polarizer 110 when the amount of motion detected by the second EVS decreases.

This may be applied analogously to a switching speed. The controlling circuit 826 may be configured to control the switching speed of a mechanism used by the polarization selector 120. For example, the polarization selector 120 may be configured to change the switching speed for switching a direction of a light signal to various polarizer sections 610 or to change the switching speed of the deactivation and activation of respective electrode pairs in a liquid crystal polarizer. The switching speed may analogously be adjusted according to the number of events detected by the 2^nd EVS 830 in a predefined time interval.

In a particular embodiment, the switching speed can be adapted to a switching pattern including multiple switching speeds, depending on different types of motion in the scene 102, such as a non-continuous motion or a staggered motion of one or more objects 104 in the scene 102. The switching speed may also be proportional to an average of the number motion events 832 calculated for multiple sets of motion events 832. The average may be calculated with weighting factors for specific sets of motion events 832.

For embodiments comprising multiple polarizers 710, the master polarization selector may comprise a controlling circuit that may control a switching speed of polarization angles of all polarizers 710. The controlling circuit may be configured to provide an equal or customized change in switching speed for each polarizer 710 proportional to the detected motion events 832.

The shape estimation processor 140 may comprise processing units related to the processing of the motion events 832 detected by the 2^nd EVS 830. For example, the shape estimation processor may comprise a motion event processing unit. The motion event processing unit may read the motion events 832, extract information therefrom, and provide the information to the controlling circuit 826 to control the switching speed. The shape estimation processor 140 may be configured to perform SfP measurements according to a first set of polarization events 132 corresponding to a first time instance of detection by the EVS 130, as well as a first set of motion events 832 detected by the 2^nd EVS 830 corresponding to the same time instance. Both the polarization events 132 and motion events 832 of the same instance may be associated with a first polarization angle of the polarizer 110. In addition, the shape estimation processor may further be configured to perform SfP measurements according to an analogous second set of polarization events 132 and motion events 832 detected at a subsequent second time instance, each associated with a second polarization angle of the polarizer 110.

The detection of motion events 832 may be used to perform multiple measurement iterations for SfP measurements. In particular, if a polarizing object 104 in the scene 102 is moving, the apparatus 800 may perform multiple rounds of adapting the switching speed between the selectable polarization angles 112 so that optimal polarization information of the scene 102 may be obtained.

Fig- 9 illustrates a flow chart of an exemplary method 900 for polarization-based surface normal measurement applying the 2^nd EVS 830. The method 900 may include defining 910 a set of polarization angles 112 and a switching rate between polarization angles 112 to be used for a full scan. The method 900 may further include detecting 920 polarization events 132 by an event-based vision sensor, EVS 130, in the full scan related to changes in intensity, each caused by a change in polarization angle 112. The method may further include estimating 930 a shape of one or more objects 104 based on the detected events 132. Furthermore, the method 900 may include examining 940 motion events 832 detected by a second EVS 830 to refine a contour estimation and may further include deciding a new switching rate for a following full scan based thereon. The method 900 may include a repeat of previous method portions 910, 920, 930, and 940 adapted to the new switching rate. The method 900 may further include processing 950 data for a refined surface normal estimation.

The method 900 may enable the shape estimation processor 140 to obtain a greater amount of polarization information of the scene in the form of polarization events 132 and corresponding polarization angles 112 at an optimal rate. While a faster switching rate between different polarization angles 112 may be desired, it may also be the case that a switching may at some point be too fast, based on the particular components of the apparatus 100 or in further embodiments. A switching speed that is too fast may cause an inaccuracy of the recorded data. For example, the polarizer 110 may not be set to its predefined polarization angle 112 within an acceptable range of error, or the EVS 130 may not be able to record events accurately at such a high rate. The motion events 832 provided by the 2^nd EVS 830 enable the apparatus 100 to minimize the use of faster switching rates between polarization angle 112 only to cases for recording polarization data related to fast moving objects 104 in the scene 102. In general, the method 900 may increase an operational efficiency of recording data by the EVS 130 and processing data by the shape estimation processor 140.

The shape estimation processor 140 in particular may comprise features that enable an efficient processing of the polarization information recorded in the form of events by the EVS 130. Since the polarization events 132 (and motion events 832) are only related to changes in intensity, they do not capture properties of the light signal related to wavelength and absolute or relative brightness of the scene 102. Thus, it may be desirable to maximize a rate of capturing polarization information and an efficiency of processing such data. For this, the shape estimation processor may comprise features related to machine learning.

Fig 10 schematically illustrates an apparatus 1000 for polarization-based surface normal measurement according to a further embodiment. In the apparatus 1000, the shape estimation processor 140 may comprise a trained machine-learning network 144 configured to predict one or more surface normals 142 of the scene 102 based on a plurality of predefined angles of the polarizer and respective events associated with the plurality of predefined angles 112 of the polarizer 110.

The trained machine-learning network 144 may be configured to accept input including polarization information. The polarization information may include one or more sets of polarization-based events 132, each corresponding to a polarization angle 112 of the polarizer 110. The polarization-based events 132 may be one or more 2-dimensional pixel arrays, each depicting a map of ON and OFF-events, with each pixel array associated with a polarization angle 112 of the polarizer 110. The trained machine learning network 142 may accept an input including a surface normal estimation or a refined surface normal estimation after one or more rounds of method steps 910 to 940. It may accept ambiguous normal maps depicting surface normal information 142 of one or more objects 104 in the scene and analyzed by the shape estimation processor 140. The ambiguous normal maps may be a 2-dimensional pixel array, wherein a pixel is either encoded with a surface normal vector 210, or the pixel is not encoded with information. The surface normal vector 210 may have a corresponding zenith angle 220 and azimuth angle 230 encoded. The ambiguous normal maps may include ambiguous solutions to a diffuse dominant or specular dominant model of reflection. The surface normal information 142 may include two physical solutions for a zenith angle 220 in the specular dominant model.

The trained machine-learning network 144 may be trained how to combine the polarization information with one or more ambiguous normal maps. The polarization information and one or more ambiguous maps may be provided as input to a prediction model, which may be configured to predict one or more surface normals of the scene 102. The predicted surface normal may be based on the polarization information and one or more ambiguous normal maps.

The trained machine-learning network 144 may be a convolutional neural network (CNN), which may comprise a surface normal reconstruction network. The surface normal reconstruction network may comprise a convolutional encoder, a downsampling unit, an upsampling unit, and a decoder to output machine-learning network estimated normals for one or more objects. The CNN may comprise one or more convolutional layers, which may perform a convolution, or a linear operation that includes a weighted multiplication on input data. The weighted multiplication may be performed by a 2-dimensional array of weights on a 2-dimensional array of data.

The convolutional encoder may be configured to extract high-level features of one or more objects from the polarization-based information and ambiguous normal maps. The encoder may provide data related to extracted high-level features of one or more objects to a downsampling unit. The downsampling unit may perform instance normalization, contrast normalization, intensity normalization, or batch normalization. Normalization may include re-centering or re-scaling of image layer inputs. The downsampling unit may generate an output, which may be provided to an upsampling unit. The upsampling unit may apply a resizing of one or more images, which may be based on surface normal information 142 or ambiguous normal maps, and a spatially adaptive normalization (SPADE). The SPADE may be a conditional normalization method of a normalization technique for semantic image synthesis with an input semantic layout. A semantic image may be an image partitioned into regions labeled with a category, which may delineate meaningful objects or features. The SPADE may minimize the loss of semantic information from one or more input layers. The upsampling unit may generate an output, which may include one or more estimated normal vectors and, which may be provided to a final layer or a decoder. The final layer or decoder may generate an output image of one or more objects based on the reconstructed surface normal information 142. The estimated surface normals may be normalized to a unit length.

The trained machine-learning network 144 may be configured to use prior knowledge related to shape or other physical properties of objects 104 in the scene 102 and/or statistical shape knowledge based on a training with a real or synthetic dataset. It may be trained how to combine both polarization information, ambiguous normal maps, and statistical shape knowledge to effectively perform shape estimation. Other prior knowledge related to objects 104 in the scene 102 may include statistical knowledge related to object speed and motion patterns.

The machine-learning network 144 may be configured to implement a supervised learning algorithm, which may be used for a training of the machine-learning network 144. The machine-learning network 144 may be trained with various training data. The training data may be labeled, including labeled inputs and outputs. The training data may comprise polarization information, including one or more sets of polarization-based events 132 and corresponding polarization angles 112, simulated ambiguous normal maps, and ground truth surface normal data. The ambiguous normal maps may include solutions to a diffuse model and solutions to a specular model.

The training data may comprise real data and/or simulated data. The real and simulated data may comprise one or more sets of events. The one or more sets of events may be depicted as 2-dimensional pixel arrays of ON and OFF-events. Real data may provide polarization-based images, images from a standard stereo camera setup, ground truth camera poses, depth maps derived from a LiDAR, or optic flow maps.

Simulated data may be in the form of events. It may be produced without a continuous representation of a visual signal by a sampling of the frames of the visual signal at a high framerate synchronously, and then performing linear interpolation to reconstruct a piecewise linear approximation of the continuous visual signal. Frames may be sample uniformly or adaptively, based on predicted dynamics of the visual signal. The adaptive sampling may be performed according to changes in intensity of a light signal and pixel displacement, which may be caused by a motion of an object. A generation of data including adaptive sampling may generate event-information that may be used to train the machine-learning network 144.

The supervised learning algorithm may perform data mining, including classification to assign the data into specific categories, and regression, including linear regression models, logistic regression models, and polynomial regression models. The machine-learning network 144 may be configured to learn to minimize the difference between the reconstructed depth computer from polarization-based events 132 and ground truth surface normal data.

The various embodiments of the apparatus 100 present many features to enable capturing polarization information of the scene 102 and performing SfP measurements. In particular, features related to the polarizer 110, the polarization selector 120, the EVS 130, and the shape estimation processor 140 may each be configured in various combinations to enable the polarization information of the scene 102 to be efficiently detected and processed. As such, the apparatus 100 may make calculations related to surface normals of the scene 102 and depict objects 104 therein, which may include fast moving objects 104. This may also be done without information or with reduced information related to wavelength or brightness of the scene 102. As such, the features of the apparatus 100 may improve technological applications making use of depth modeling and 3D modeling, particularly related to non-Lambertian surfaces.

The following examples pertain to further embodiments:

(1) An apparatus for shape measurement of a scene, the apparatus comprising at least one polarizer having a plurality of selectable polarization angles; a polarization selector configured to select a first polarization angle of the polarizer to cause the polarizer to pass light from the scene at the first polarization angle and to subsequently select a second polarization angle to cause the polarizer to pass light from the scene at the second polarization angle; an eventbased vision sensor, EVS, configured to detect a first set of events associated with the passed light of the first polarization angle of the polarizer and to subsequently detect a second set of events associated with the passed light of the second polarization angle of the polarizer; and a shape estimation processor configured to compute surface normal information of the scene based on the first and second set of events and the corresponding first and second polarization angles of the polarizer.

(2) The apparatus of (1), wherein the polarizer comprises a liquid crystal layer including a liquid crystal material; and wherein the polarization selector comprises at least a first pair of electrodes and a second pair of electrodes, and wherein the polarization selector is configured to select the first polarization angle by activating the first pair of electrodes and to select the second polarization angle by activating the second pair of electrodes.

(3) The apparatus of (2), wherein the liquid crystal layer is positioned between a first electrode and a second electrode of the respective pair of electrodes.

(4) The apparatus of (2) or (3), wherein the polarizer comprises a front plate and a back plate, wherein the polarizer is configured to pass a light signal of the scene through the front plate, subsequently through the liquid crystal material of the liquid crystal layer, and subsequently through the back plate.

(5) The apparatus of (4), wherein the back plate comprises parallel grating grooves. (6) The apparatus of (1), wherein the polarizer comprises a plurality of polarizer sections, including a first polarizer section fixed to pass light from the scene at the first polarization angle and a second polarizer section fixed to pass light from the scene at the second polarization angle and wherein the polarization selector is configured to select the first polarization angle by directing light from the scene to the first polarizer section and to select the second polarization angle by directing light from the scene to the second polarizer section.

(7) The apparatus of (6), wherein the polarization selector comprises a reflective surface and an electric drive configured to control an orientation of the reflective surface.

(8) The apparatus of (7), wherein the polarization selector comprises an electromagnetic actuation mechanism or an electrostatic actuation mechanism to control an orientation of the reflective surface and/or wherein the reflective surface is a MEMS mirror.

(9) The apparatus of (1), wherein the polarizer is a rotatable polarizer and the polarization selector comprises a powered gear for rotating the rotatable polarizer, wherein the polarization selector is configured to select the first polarization angle by rotating the rotatable polarizer to a first rotation angle and to select the second polarization angle by rotating the rotatable polarizer to a second rotation angle.

(10) The apparatus of (1) to (9) comprising a plurality of polarizers and a plurality of polarization selectors, wherein each polarization selector corresponds to a polarizer, wherein each polarization selector is configured to sequentially select multiple polarization angles for the corresponding polarizer.

(11) The apparatus of claim (1) to (10), further comprising a linearly polarizing structure positioned between the at least one polarizer and the EVS.

(12) The apparatus of (11), wherein the linearly polarizing structure is in the form of an on- chip linearly polarizing film.

(13) The apparatus of claim (11) or (12), wherein the linearly polarizing structure comprises a plurality of sections, each comprising a plurality of subsections, wherein each subsection is fixed to pass light at a pre-specified polarization angle that is unique within the respective section.

(14) The apparatus of (13), wherein each subsection uniquely corresponds to a pixel of the EVS.

(15) The apparatus of (13), wherein the at least one polarizer comprises a plurality of at least two liquid crystal polarizers comprising a respective liquid crystal material, a respective front plate upstream to the respective liquid crystal material, and a respective back plate downstream to the respective liquid crystal material, and a respective polarization selector for each of the at least two liquid crystal polarizers, each comprising a respective set of multiple electrode pairs, wherein the respective polarization selector is configured to switch between the multiple selectable polarization angles by sequential activation of the respective set of multiple electrode pairs to sequentially cause the respective liquid crystal material to polarize an incoming light signal at the selected polarization angle; wherein each liquid crystal polarizer is configured to pass a respective light signal of the scene through a respective front plate, subsequently through the respective liquid crystal material, and subsequently through a respective back plate; and wherein each respective back plate comprises parallel grating grooves oriented parallel to the pre-specified polarization angle of a corresponding subsection within each section of the plurality of sections.

(16) The apparatus of (1) to (15), further comprising a second EVS configured to detect one or more areas of motion in the scene, and wherein the polarization selector comprises a controlling circuit configured to control a switching speed of the polarizer between the selectable polarization angles based on information from the one or more areas of motion detected by the second EVS.

(17) The apparatus of (16), wherein the controlling circuit is configured to control the switching speed in proportionality to a number of events generated by the second EVS in a predefined time interval.

(18) The apparatus of (1) to (17), wherein the shape estimation processor is configured to compute an orientation of one or more surface normals of a portion of the scene based on a change in intensity of passed light between the first and second polarization angles of the polarizer that triggered the EVS to output an event associated with the portion of the scene.

(19) The apparatus of (1) to (18), wherein the shape estimation processor comprises a trained machine-learning model configured to predict one or more surface normals of the scene based on a plurality of pre-specified polarization angles of the polarizer and respective events associated with the plurality of pre-specified polarization angles of the polarizer.

(20) A method for shape measurement of a scene, the method comprising: providing a polarizer with a plurality of selectable polarization angles; selecting a first polarization angle of the polarizer to cause the polarizer to pass light from the scene at the first polarization angle; detecting, with an EVS, a first set of events associated with the passed light of the first polarization angle of the polarizer; selecting a second polarization angle to cause the polarizer to pass light from the scene at the second polarization angle; detecting, with the EVS, a second set of events associated with the passed light of the second polarization angle of the polarizer; computing surface normal information of the scene based on the first and second set of events and the corresponding first and second polarization angles of the polarizer.

The aspects and features described in relation to a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the further example.

Examples may further be or relate to a (computer) program including a program code to execute one or more of the above methods when the program is executed on a computer, processor or other programmable hardware component. Thus, steps, operations or processes of different ones of the methods described above may also be executed by programmed computers, processors or other programmable hardware components. Examples may also cover program storage devices, such as digital data storage media, which are machine-, processor- or computer-readable and encode and/or contain machine-executable, processor-executable or computer-executable programs and instructions. Program storage devices may include or be digital storage devices, magnetic storage media such as magnetic disks and magnetic tapes, hard disk drives, or optically readable digital data storage media, for example. Other examples may also include computers, processors, control units, (field) programmable logic arrays ((F)PLAs), (field) programmable gate arrays ((F)PGAs), graphics processor units (GPU), application-specific integrated circuits (ASICs), integrated circuits (ICs) or system-on-a-chip (SoCs) systems programmed to execute the steps of the methods described above.

It is further understood that the disclosure of several steps, processes, operations or functions disclosed in the description or claims shall not be construed to imply that these operations are necessarily dependent on the order described, unless explicitly stated in the individual case or necessary for technical reasons. Therefore, the previous description does not limit the execution of several steps or functions to a certain order. Furthermore, in further examples, a single step, function, process or operation may include and/or be broken up into several sub-steps, - functions, -processes or -operations.

If some aspects have been described in relation to a device or system, these aspects should also be understood as a description of the corresponding method. For example, a block, device or functional aspect of the device or system may correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described in relation to a method shall also be understood as a description of a corresponding block, a corresponding element, a property or a functional feature of a corresponding device or a corresponding system.

The following claims are hereby incorporated in the detailed description, wherein each claim may stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent or independent claim. Such combinations are hereby explicitly proposed, unless it is stated in the individual case that a particular combination is not intended. Furthermore, features of a claim should also be included for any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

Claims

1. An apparatus for shape measurement of a scene, the apparatus comprising: at least one polarizer having a plurality of selectable polarization angles; a polarization selector configured to select a first polarization angle of the polarizer to cause the polarizer to pass light from the scene at the first polarization angle and to subsequently select a second polarization angle to cause the polarizer to pass light from the scene at the second polarization angle; an event-based vision sensor, EVS, configured to detect a first set of events associated with the passed light of the first polarization angle of the polarizer and to subsequently detect a second set of events associated with the passed light of the second polarization angle of the polarizer; and a shape estimation processor configured to compute surface normal information of the scene based on the first and second set of events and the corresponding first and second polarization angles of the polarizer.

2. The apparatus of claim 1, wherein the polarizer comprises a liquid crystal layer including a liquid crystal material; and wherein the polarization selector comprises at least a first pair of electrodes and a second pair of electrodes, and wherein the polarization selector is configured to select the first polarization angle by activating the first pair of electrodes and to select the second polarization angle by activating the second pair of electrodes.

3. The apparatus of claim 2, wherein the liquid crystal layer is positioned between a first electrode and a second electrode of the respective pair of electrodes.

4. The apparatus of claim 2, wherein the polarizer comprises a front plate and a back plate, wherein the polarizer is configured to pass a light signal of the scene through the front plate, subsequently through the liquid crystal material of the liquid crystal layer, and subsequently through the back plate.

5. The apparatus of claim 4, wherein the back plate comprises parallel grating grooves.

6. The apparatus of claim 1, wherein the polarizer comprises a plurality of polarizer sections, including a first polarizer section fixed to pass light from the scene at the first polarization angle and a second polarizer section fixed to pass light from the scene at the second polarization angle and wherein the polarization selector is configured to select the first polarization angle by directing light from the scene to the first polarizer section and to select the second polarization angle by directing light from the scene to the second polarizer section.

7. The apparatus of claim 6, wherein the polarization selector comprises a reflective surface and an electric drive configured to control an orientation of the reflective surface.

8. The apparatus of claim 7, wherein the polarization selector comprises an electromagnetic actuation mechanism or an electrostatic actuation mechanism to control an orientation of the reflective surface and/or wherein the reflective surface is a MEMS mirror.

9. The apparatus of claim 1, comprising a plurality of polarizers and a plurality of polarization selectors, wherein each polarization selector corresponds to a polarizer, wherein each polarization selector is configured to sequentially select multiple polarization angles for the corresponding polarizer.

10. The apparatus of claim 1, further comprising a linearly polarizing structure positioned between the at least one polarizer and the EVS.

11. The apparatus of claim 10, wherein the linearly polarizing structure is in the form of an on-chip linearly polarizing film.

12. The apparatus of claim 10, wherein the linearly polarizing structure comprises a plurality of sections, each comprising a plurality of subsections, wherein each subsection is fixed to pass light at a pre-specified polarization angle that is unique within the respective section.

13. The apparatus of claim 12, wherein each subsection uniquely corresponds to a pixel of the EVS.

14. The apparatus of claim 12, wherein the at least one polarizer comprises a plurality of at least two liquid crystal polarizers comprising a respective liquid crystal material, a respective front plate upstream to the respective liquid crystal material, and a respective back plate downstream to the respective liquid crystal material, and a respective polarization selector for each of the at least two liquid crystal polarizers, each comprising a respective set of multiple electrode pairs, wherein the respective polarization selector is configured to switch between the multiple selectable polarization angles by sequential activation of the respective set of multiple electrode pairs to sequentially cause the respective liquid crystal material to polarize an incoming light signal at the selected polarization angle; wherein each liquid crystal polarizer is configured to pass a respective light signal of the scene through a respective front plate, subsequently through the respective liquid crystal material, and subsequently through a respective back plate; and wherein each respective back plate comprises parallel grating grooves oriented parallel to the pre-specified polarization angle of a corresponding subsection within each section of the plurality of sections.

15. The apparatus of claim 1, further comprising a second EVS configured to detect one or more areas of motion in the scene, and wherein the polarization selector comprises a controlling circuit configured to control a switching speed of the polarizer between the selectable polarization angles based on information from the one or more areas of motion detected by the second EVS.

16. The apparatus of claim 15, wherein the controlling circuit is configured to control the switching speed in proportionality to a number of events generated by the second EVS in a predefined time interval.

17. The apparatus of claim 1, wherein the shape estimation processor is configured to compute an orientation of one or more surface normals of a portion of the scene based on a change in intensity of passed light between the first and second polarization angles of the polarizer that triggered the EVS to output an event associated with the portion of the scene.

18. The apparatus of claim 1, wherein the shape estimation processor comprises a trained machine-learning model configured to predict one or more surface normals of the scene based on a plurality of pre-specified polarization angles of the polarizer and respective events associated with the plurality of pre-specified polarization angles of the polarizer.

19. A method for shape measurement of a scene, the method comprising providing a polarizer with a plurality of selectable polarization angles; selecting a first polarization angle of the polarizer to cause the polarizer to pass light from the scene at the first polarization angle; detecting, with an EVS, a first set of events associated with the passed light of the first polarization angle of the polarizer; selecting a second polarization angle to cause the polarizer to pass light from the scene at the second polarization angle; detecting, with the EVS, a second set of events associated with the passed light of the second polarization angle of the polarizer; computing surface normal information of the scene based on the first and second set of events and the corresponding first and second polarization angles of the polarizer.