WO2023175162A1

WO2023175162A1 - Device and method for detecting an object above a detection surface

Info

Publication number: WO2023175162A1
Application number: PCT/EP2023/056928
Authority: WO
Inventors: Etienne SANTOUL; Joshua TUCKEY
Original assignee: Embodme
Priority date: 2022-03-18
Filing date: 2023-03-17
Publication date: 2023-09-21

Abstract

The invention relates to an electronic device (1) comprising a detection surface, in particular for detecting an object (10) in the near field (2), comprising: a substrate; a plurality of photoemitters separated from one another on the substrate; a plurality of photodetectors, at least some of these photodetectors being disposed between the photoemitters on the substrate and computation means for determining the distance between the object and the detection surface.

Description

Description Title: DEVICE AND METHOD FOR DETECTING AN OBJECT ABOVE A DETECTION SURFACE

Field of the invention

The invention relates to an electronic device for detecting an object in the near field in front of a detection surface. The invention also relates to a method for detecting such an object.

State of the art

Today there are devices for detecting an object or several objects based on the use of optics.

For example, document FR3103341 describes the use of a device such as a camera or depth camera to detect a user's hand or the recognition of a hand gesture.

A disadvantage of this type of device is that the detection cone or field of view of such a camera is narrow and defined by its optical lens. Consequently, this type of device is not suitable for detecting an object in the near field, that is to say at almost zero height relative to the detection device.

To resolve this drawback, document EP2843509 describes a device comprising a peripheral border of a screen making it possible to estimate the coordinates of an object above the screen.

However, a drawback of this device is that it is not precise enough to accurately determine the height coordinate of the object to be detected (or its distance from the screen).

Another disadvantage is that this device, due to the location of the photodetectors on the periphery and their low detection angle, does not make it possible to detect an object close to the center of the screen.

The object of the invention is therefore to propose a device and a method making it possible to precisely detect objects in the field close to a detection surface.

Summary of the invention According to a first aspect, the invention relates to an electronic device comprising:

■ a detection surface, in particular near field detection comprising: o optionally a substrate; o a plurality of photoemitters, the photoemitters being separated from each other, for example on said substrate; o a plurality of photodetectors, at least part of these photodetectors each being arranged between two photoemitters, for example on the substrate;

■ a first acquisition unit connected to the plurality of photodetectors and to the plurality of photoemitters comprising means adapted for carrying out the following steps: o generating an emission by controlling the ignition of the photoemitters; o the acquisition of a light intensity map according to the position and the light intensity received by each photodetector;

■ a second unit for detecting an object placed in front of the detection surface comprising means adapted for determining a distance between said object and said detection surface.

In one embodiment, the second detection unit comprises means adapted for determining a distance between said object and said detection surface by executing the following steps:

■ selecting a plurality of point spread functions, each point spread function being characteristic of a predetermined height;

■ carrying out a deconvolution of the light intensity mapping acquired by each selected point spreading function;

■ for each deconvolution, the calculation of a coherence score;

■ determining the distance of said object with the detection surface from the predetermined height of the function spreading of the point which made it possible to obtain the deconvolution having obtained the best coherence score.

■ the selection of a plurality of point spread functions, each function being characteristic of a predetermined height;

■ for each function of the plurality of functions, calculating a coherence score from the light intensity mapping,

■ determining the distance of said object with the detection surface from the predetermined height of the function having made it possible to obtain the best coherence score.

In one embodiment, the calculation of the consistency score includes:

■ carrying out a deconvolution of the light intensity mapping acquired by each selected function;

■ for each deconvolution, the calculation of a coherence score.

An advantage of such an electronic device is to allow the detection of an object, without being hampered by the detection angle. It is therefore possible to detect an object in the near field or at short distance from a large surface, which is not possible with traditional optics or depth cameras.

In one embodiment, calculating a coherence score includes applying a transform of the deconvolution of the acquired light intensity map and/or the acquired light intensity map. The second unit may include means adapted for the application of such a transform. In one embodiment, applying a transform includes applying a Fourier transform or a Laplace transform.

In one embodiment, the second unit also comprises means for executing a step of estimating the shape of the object in a plane parallel to the plane of the detection surface from the light intensity mapping acquired and/or from the deconvolution having obtained the best score.

In one embodiment, the second unit also comprises means for executing a step of determining the spatial coordinates of the object in a plane parallel to the plane of the detection surface, for example from the deconvolution having obtained the best score .

In one embodiment, the detection surface includes a display screen. Preferably, the display screen is formed by the plurality of pixels and the photoemitters and photodetectors are distributed between said pixels. In one embodiment, the pixels of the plurality of pixels each comprise at least one visible light source, preferably RGB light sources, and in which the photoemitters are designed to emit infrared light and the photodetectors are designed to detect an infrared light. light intensity in the infrared region.

In one embodiment, the first unit comprises means for executing the step of acquiring a light intensity map so that said acquisition step includes a step of measuring, by each photodetector, the light intensity emitted by the photoemitters and reflected by at least one object placed in front of the detection surface.

In one embodiment, the electronic device comprises a covering layer covering the plurality of photoemitters and the plurality of photodetectors, said covering layer being transparent to the rays emitted by the photoemitters.

In one embodiment, the coherence score is calculated based on a sharpness criterion of the image obtained by the deconvolution of the acquired light intensity mapping.

The invention also relates to a method of using an electronic device comprising the following steps:

■ generating an emission by controlling the ignition of the photoemitters; ■ the acquisition of a light intensity map as a function of the position and the light intensity received by each photodetector;

■ for each deconvolution, the calculation of a coherence score;

■ determining the distance of said object with the detection surface from the predetermined height of the point spreading function which made it possible to obtain the deconvolution having obtained the best coherence score.

According to a second aspect, the invention relates to an electronic device comprising:

■ a detection surface, in particular for the detection of an object in the near field comprising: o optionally a substrate; o a plurality of photoemitters separated from each other, for example on said substrate; o a plurality of photodetectors, at least part of these photodetectors being arranged between said photoemitters, for example on the substrate;

■ a first unit for acquiring a light intensity map connected to the plurality of photoemitters and to the plurality of photodetectors comprising means adapted to execute the following steps: o generating a first emission pattern by the controlling the ignition of a first set of photoemitters of the detection surface; o the acquisition of a first light intensity map as a function of the position and the light intensity received by each photodetector during the generation of the first emission pattern. In one embodiment, the device comprises a second unit for generating a point cloud of said object placed in front of the detection surface comprising means for executing a construction function making it possible to generate a final point cloud of said object from the first emission pattern and the first acquired light intensity mapping.

In one embodiment, the device comprises a second unit for generating at least one point of said object placed in front of the detection surface comprising means for executing a construction function making it possible to generate at least one end point of said object from the first emission pattern and the first acquired light intensity mapping.

In one embodiment, the first unit and/or the second unit comprises means for carrying out the following steps:

• the generation of a second emission pattern by controlling the ignition of a second set of photoemitters different from the first set;

• the acquisition of a second light intensity map as a function of the position and the light intensity received by each photodetector of the detection surface during the generation of the second emission pattern;

In one embodiment, the generation of at least one end point of said object is carried out from the first emission pattern, the first acquired light intensity mapping, the second emission pattern and the second light intensity mapping. light intensity.

In one embodiment, the at least one point comprises a point cloud, and the at least one end point comprises a end point cloud.

The invention also relates to a method implemented by a computer for generating a point cloud of an object placed in front of the detection surface of an electronic device according to the invention comprising the following steps:

■ generating a first emission pattern by controlling the ignition of a first set of photoemitters of the detection surface; ■ the acquisition of a first light intensity map as a function of the position and the light intensity received by each photodetector during the generation of the first emission pattern;

■ the execution of a construction function making it possible to generate a final point cloud of said object from the first emission pattern and the first acquired light intensity map.

An advantage of such an electronic device and such a method is to authorize the detection and reconstruction of a cloud of points of an object, without being hampered by the detection angle. It is therefore possible to detect an object in the near field or at short distance from a large surface, which is not possible with traditional optics or depth cameras.

In one embodiment, the execution of the construction function comprises the execution of the following steps:

■ generating a first image by carrying out a simulation of light rays emitted by the photoemitters of the first set and reflected by an object defined by a cloud of candidate points;

■ a first comparison of the first light intensity map acquired with the first image generated;

■ minimization of the result of the first comparison by varying the candidate point cloud;

■ determining a cloud of points representative of said object based on the minimization.

In one embodiment, the generation of the end point cloud comprises:

■ the generation of a second emission pattern by controlling the ignition of a second set of photoemitters different from the first set;

■ the acquisition of a second light intensity map as a function of the position and the light intensity received by each photodetector during the generation of the second emission pattern. In one embodiment, the generation of the point cloud of said object is carried out from the first emission pattern and the first acquired light intensity mapping as well as the second emission pattern and the second intensity mapping bright.

In one embodiment, the implementation of the construction function comprises:

■ generating a second image by carrying out a simulation of light rays emitted by the photodetectors of the second set and reflected by an object defined by the cloud of candidate points;

■ a second comparison of the second light intensity map acquired with the second image generated.

In one embodiment, the minimization step comprises minimization of the result of the second comparison and the first comparison by varying the candidate point cloud of said object.

In one embodiment, the minimization step is implemented by a gradient descent method.

In one embodiment, the step of generating an image comprises generating a virtual object defined by a candidate point cloud in a virtual environment comprising a simulation of a surface similar to the detection surface.

In one embodiment, the construction function is a learning function, implemented by a neural network trained to generate a point cloud of an object from one or more pairs of patterns and intensity maps light acquired.

In one embodiment, the electronic device comprises a memory comprising a library (or a database) of acquired light intensity maps, each associated with an emission pattern and a point cloud of an object, the construction function being capable of generating said point cloud by comparison with the maps of said library.

In one embodiment, the second unit comprises means for receiving at least one distance information between said object and the detection surface and in which the point cloud is generated by the construction function from said distance information. In one embodiment, the first set of photodetectors comprises at least a first zone of the substrate in which all the photodetectors are turned on.

In one embodiment, the second set of the second pattern is selected based on the result of the first comparison.

In one embodiment, the detection surface comprises a display screen formed by a plurality of pixels and in which the plurality of photoemitters and photodetectors are dispersed between and/or in said pixels.

In one embodiment, the device comprises means for implementing the method according to the first aspect of the invention.

According to a third aspect, the invention relates to a movement recognition system comprising an electronic device according to the invention comprising a movement recognition unit comprising means for executing the acquisition of a plurality of end points and/or or point clouds of the object placed in front of the detection surface as a function of time and the determination of at least one movement parameter from a rotation of the object and/or a direction of translation of the object detected from the end points and/or point clouds acquired as a function of time.

According to another aspect, the invention relates to a method of displaying a user interface on the screen of an electronic device. Said method comprises determining a distance between the detection surface 2 of the electronic device and an object; providing a database in which a plurality of different user interfaces are recorded, each user interface being associated with a predetermined distance different from the others. The user interface to be displayed is selected by comparison between the determined distance and the predetermined distances associated with the pre-recorded user interfaces. The method finally includes displaying the selected user interface.

The invention also relates to a computer program product comprising instructions which cause the electronic device according to the invention to execute the steps of a method or methods according to the invention. The invention relates to a computer-readable medium on which the computer program according to the invention is recorded. Brief description of the figures

Other characteristics and advantages of the invention will emerge on reading the detailed description which follows, with reference to the appended figures, which illustrate:

Fig. 1: a schematic view of an electronic device according to one embodiment of the invention comprising a detection surface whose light rays emitted by the photoemitters are reflected by the hand of a user towards said detection surface.

Fig. 2: a schematic sectional view of the detection surface comprising a plurality of photoemitters emitting light rays and photodetectors for detecting the rays reflected by the object above the detection surface.

Fig. 3: an example of an array of pixels of the detection surface according to an embodiment in which each pixel comprises sources of red, blue, green light as well as a photoemitter and a photodetector capable of detecting the rays emitted by the photoemitter.

Fig. 4: an example of an array of pixels of the detection surface according to an embodiment in which each pixel comprises red, blue, green light sources as well as a photoemitter or a photodetector.

Fig. 5: a representation of a light intensity map acquired by the photodetector network according to one embodiment of the invention.

Fig. 6: an example of deconvolution of the light intensity mapping according to Figure 5 from different point spread functions. Each point spreading function being characteristic of a predetermined height of the object and different from the other point spreading functions.

Fig. 7: a flowchart representing a method for determining the distance between an object located in front of the detection surface and said detection surface according to one embodiment of the invention.

Fig. 8A: a flowchart representing a method for generating a point cloud representing an object detected in front of the detection surface according to one embodiment of the invention.

Fig. 8B: a flowchart representing a method for generating a point cloud representing an object detected in front of the detection surface according to an embodiment of the invention in which several acquisitions of Light intensity maps are made by modifying the emission pattern.

Fig. 9: a flowchart representing an example of carrying out the minimization step of the method according to Figure 8B.

Fig. 10: a flowchart representing a method for generating a movement parameter of an object detected by the detection surface according to one embodiment of the invention.

Fig. 11: a representation of several emission patterns by the activation of a set of photoemitters.

Fig. 12: a schematic representation of a calculation system of the electronic device according to one embodiment of the invention.

Fig. 13A to Fig. 131: examples of gestures that can be detected by the electronic device according to one embodiment of the invention.

Fig. 14: a schematic view of an electronic device according to one embodiment of the invention in which a user interface is selected as a function of the distance between the object and the detection surface to be displayed on the screen of the electronic device .

Fig. 15: a representation of the second spreading function of the point which made it possible to obtain the deconvolution D2 of Figure 6.

Definitions

By “photodetector” is meant any type of structural means making it possible to detect a light intensity and to generate a signal as a function of said detected intensity.

By “photoemitter” we mean any type of means capable of emitting light radiation when activated, for example by applying a current or a voltage.

By “unit”, we mean any type of software and/or hardware means allowing the execution of a method or algorithm.

By “substantially” followed by a numerical value means said numerical value +/- 5% or 10%. When the term “substantially” designates an angle, this term must be understood as plus or minus 1°, 5° or 10°. The terms “normal”, “parallel” and “perpendicular” implicitly designate angles of 90°, 0° and 90° respectively.

By “plane of the detection surface” is meant the plane formed by the detection surface. When the detection surface is curved, the plane of the detection surface must be understood as the tangent at a point of said detection surface.

By “an object located in front of the detection surface”, we mean an object located in the area adjacent to the detection surface illuminated by the photodetectors when the latter are turned on. We can also simply hear an object in the light beam generated by the photoemitters of the detection surface when they are turned on or activated.

By “luminous”, we mean any signal in the visible range, in the infrared (near or far) or ultraviolet.

Description of the invention

Detection surface

According to a first aspect, the invention relates to an electronic device comprising a near field detection surface. An embodiment of such a device is shown in Figure 1.

Such an electronic device 1 comprises a detection surface 2. The detection surface 2 can extend over one face of the electronic device 1.

Detection surface 2 designates a flat or curved surface making it possible to detect an object in a volume extending in the direction normal to said detection surface 2.

The detection surface 2 preferably designates a substrate whose first two dimensions (length and width) are significantly larger, for example at least 3 times larger than the third dimension (thickness) or preferably 5 times larger or 10 times bigger.

The detection surface 2 comprises at least one substrate 32 on which a plurality of photodetectors and photoemitters are arranged.

The photodetectors 33 and the photoemitters 31 can be arranged on the same side of the substrate 32. The photodetectors 33 and the photoemitters 31 can be arranged so as to emit or detect a light beam from the same side of the substrate 32.

In an embodiment not shown, the substrate comprises at least two layers. A first layer comprising the photoemitters 31 and a second layer comprising photodetectors 33 or vice versa.

Each photodetector 33 of the second layer is preferably arranged between two photoemitters 31 of the first layer or vice versa.

Preferably, the matrix of the first layer is transparent to the radiation emitted by the photoemitters 33. Thus, the light rays emitted by the photoemitters can, after reflection, pass through the first layer to be detected by the photodetectors. In another example, the first layer includes openings allowing the passage of radiation to the photodetectors of the second layer.

In another alternative embodiment not shown, the detection surface 2 comprises a first portion on which the photodetectors are arranged and a second portion, framing the first portion on which the photoemitters are arranged.

The photoemitters 31 are designed to emit photons by electroluminescence effect. The photoemitters 31 are preferably designed to emit light rays in a range of predefined wavelengths.

Preferably, the photoemitters 31 are designed to emit infrared light radiation.

The photodetectors 33 are designed to detect a light ray emitted by the plurality of photoemitters 31. The photodetectors 33 are designed to transform the light absorbed in a range of wavelengths into a measurable quantity such as an electric current or an electric voltage .

The photodetectors include, for example, optical fiber sensors or preferably photodiodes. The photodetectors 33 may include infrared light photodetectors or infrared light phototransistors.

Preferably, the photoemitters are designed so that their cone of diffusion of emitted light is as weak as possible. Likewise, each photoemitter is arranged on the substrate 32 so as to emit a diffusion cone whose axis is perpendicular or substantially perpendicular to the detection surface 2. An advantage is to obtain a plurality of parallel light rays and whose surface area of the spreading zone on an object 10 is as small as possible.

In one embodiment, the photodetectors are designed to have the lowest possible detection cone.

The angle formed by the diffusion cone of one or more photoemitters may be different from the angle of the detection cone of one or more photodetectors. In one mode, the angle formed by the diffusion cone of one or more photoemitters is greater than the angle of the detection cone of one or more photodetectors. By “angle formed by the diffusion cone”, we mean the angular distribution function of the light intensity of diffusion of light radiation.

Likewise, the term “detection cone” means the relative radial sensitivity of a detector, that is to say the difference in intensity received by a detector as a function of the angular position of the source.

In one embodiment, the device comprises means allowing the activation and extinction of each photoemitter 31 individually. Each photoemitter 31 can therefore be activated (to emit light radiation) or turned off independently of the activation of the other photoemitters 31.

The electronic device 1 further comprises means for measuring the signal emitted by each photodetector.

Preferably, the photodetectors and the photoemitters are regularly distributed along the two directions of space (x,y) of the detection surface 2.

Said two directions of space relate to the detection surface 2, particularly in the case of a non-planar detection surface, for example having a curvature.

Each photodetector is preferably arranged between at least two photoemitters in the two directions of space. In one example, the photodetectors and the photoemitters are regularly distributed in two orthogonal directions of the plane of the detection surface 2.

The detection surface 2 preferably comprises a plate or a transparent layer arranged to cover the plurality of photoemitters 31 and photodetectors. By “transparent” we mean transparent to the light rays emitted by the photoemitters. Such a plate advantageously makes it possible to protect the physical integrity of the components of the detection surface 2 while allowing outgoing and incoming light rays to pass. In one example, the covering layer is transparent to infrared rays. In one example, the covering layer is transparent to light rays emitted by the photoemitters and to rays in the visible range such as a quartz plate.

Display screen

In one embodiment, the detection surface 2 comprises a display screen 3. The display screen 3 comprises a plurality of light sources arranged on the substrate.

The light sources preferentially form pixels to form the display screen. Each pixel preferably includes RGB light sources. The light sources are preferably RGB light-emitting diodes.

In one embodiment, the photoemitters and/or photodetectors are arranged, preferably regularly arranged, between the pixels of the detection surface. In an example illustrated in Figure 3, each pixel includes a photodetector and a photoemitter. In another example illustrated in Figure 4, each pixel comprises a photodetector or a photoemitter. The photoemitters and photodetectors are arranged so as to have a photoemitter every two pixels.

An advantage of integrating photoemitters 31 and photodetectors within or between the pixels of a display screen is to take advantage of the connection tracks of the light sources of the screen to connect the photoemitters and photodetectors. Thus, manufacturing costs are reduced and it is not necessary to increase the size of the device to add peripheral detection elements.

In a first example, the photoemitters and the photodetectors are distributed regularly on the substrate, for example by forming a matrix. In a second example, the photoemitters and the photodetectors are distributed on the substrate randomly. This may in particular be the case when the photoemitters and/or photodetectors are designed by printing an ink forming organic photodiodes on said substrate. In this case, it remains important to know the spatial coordinates of each photodetector 33 in the plane of the detection surface 2.

It is obvious that other methods of distribution can be imagined by those skilled in the art and that the scope of protection is not limited to the examples cited in this application.

In a preferred embodiment, the photoemitters emit light rays in a range of wavelengths distinct from the range of wavelengths of the light rays emitted by the light sources of the screen. An advantage is to be able to carry out light intensity maps as described below without having pollution from the RGB light sources of the screen 3. In an alternative embodiment, at least one of the light sources RGB light can be the photoemitter 31, in this case, the photodetector 33 is designed to detect a light intensity emitted by said photoemitter 31.

Software means

The device according to the invention also comprises software and/or hardware means for implementing the method(s) according to the invention described below. These hardware means may include calculation units, processors, memories and are described in more detail in the present description.

Acquisition of a light intensity map

According to a first aspect, the invention relates to a method 200 for determining the distance between an object 10 placed in front of the detection surface 2 and said detection surface 2. Said method 200 can be implemented by the electronic device 1 according to the 'invention.

By “object”, we mean one or more objects of a scene defined in a volume extending from the detection surface in a direction normal to said detection surface 2. As explained below, an “object” preferably comprises a or more hands of a user. In a embodiment, the term “object” designates a scene comprising a plurality of objects in said volume extending from the detection surface in a direction perpendicular to said detection surface.

The method comprises a step MOT1 of emitting light 22 from the photoemitters 31 of the detection surface 2. By "emission" is meant that at least part of the photoemitters 31 emits light or light rays 22. This Emission is produced by the activation of the photoemitters 31. This activation can be caused by means of controlling the photoemitters 31. These control means advantageously make it possible to remotely activate all or part of the plurality of photoemitters 31, for example by applying a signal to the terminals of said photoemitters 31.

Preferably, this emission step MOT1 comprises the activation of all the photoemitters of the plurality of photoemitters of the detection surface 2. The plurality of photoemitters thus emits light rays 22 which can be schematized by a diffusion cone whose axis has a direction normal or substantially normal (or perpendicular) to the plane of the detection surface 2. Preferably, all the photoemitters are turned on at the same time during this step, very preferably at the same intensity.

When an object 10 is present in a volume extending from the detection surface in a direction normal to said detection surface, this object 10 is illuminated by the rays emitted 22 by the photoemitters. The object 10 then reflects part of these emitted rays 22.

The reflected rays 21 are returned towards the detection surface 2.

The reflected rays 21 will illuminate the detection surface 2 and the photodetectors.

The ACQU1 acquisition step is carried out by measuring the light intensity received by each photodetector 33 of the detection surface 2. Advantageously, said measurement is carried out at the same time as the emission step M0T1 or at the same time. time as the emission of the rays 22 by the photoemitters 31. Thus, the measurement carried out by the acquisition step comprises the detection of the light intensity reflected 21 by an object 10 placed in front of the detection surface 2. The measurement by each photodetector 33 of the light intensity received makes it possible to construct a light intensity map 4. An example of such a map 4 is illustrated in Figure 5. In one embodiment, a light intensity map comprises a two-dimensional representation representing the intensity of the light detected by each photodetector 33 as a function of its position on the detection surface 2.

Preferably, the color at a given coordinate of said light intensity map is a function of the light intensity measured by a photodetector arranged at a coordinate on the detection surface 2 representative of the given coordinate.

Said image is constructed according to the light intensity received by each photodetector and according to its position on the detection surface. In particular, the color of a pixel of said mapping (for example a position on a gray scale) is a function of the intensity received by the photodetector whose position on the detection surface corresponds to the position of the pixel on the mapping.

An example of such acquired light intensity mapping is shown in Figure 5.

Such acquisition includes the generation of a first pattern by controlling the ignition of the photoemitters.

Stitch spread function

The point spread function (also called “point spread function” in English or “PSF”) is a mathematical function describing the response of an imaging system to a point source of light. This function is representative of the spreading of a light point. To simplify, the projection on a wall of a light source appears blurry and more diffuse than the source itself. This is due to this natural spreading of light.

In our example, the object 10, when illuminated by the light rays 22 emitted by the photoemitters, can be considered as a light source emitting the reflected light rays 21.

As illustrated in Figure 5, the acquired light intensity mapping 4, due to the point spreading function appears blurry and diffuse. There deduction of the shape of the object is difficult or even impossible to achieve from this acquired light intensity mapping 4.

Now, the point spread function can be known. It is notably a function of properties and photodetectors, for example the angle of the cone and detection.

The point spread function is also a function of the height of the light source. In other words, the point spread function is different depending on the height of the object position. By “height of the object”, we will mean in the remainder of the description the distance between the object 10 and the detection surface 2.

The spreading function of the point at a given height being known, a deconvolution operation makes it possible to reverse the process to find, from the light intensity mapping 4, an image representative of the light source and therefore of the object 10 to detect.

Selecting point spread functions

The method therefore comprises a selection SEL1 of at least two, preferably a plurality of spreading functions of the point PSF1, PSF2, PSF3. Each selected point spreading function is characteristic of a predetermined height of the object 10 with the detection surface 2.

For example, a first spreading function of the point PSF1 can be characteristic of a height (distance between the object 10 and the detection surface 2) of 5 cm.

Deconvoi utions of light intensity mapping

In one step, a DEC1 deconvolution of the light intensity map 4 is carried out with respect to each spreading function of the selected point PSF1, PSF2, PSF3.

A plurality of deconvolved images D1, D2, D3 are therefore generated. Each deconvolved image (also called deconvolution) is therefore a representation of the light source assuming that its distance from the detection surface 2 is the height characterizing the point spreading function used to obtain said deconvolution.

An example of deconvolutions D1, D2, D3 obtained are illustrated in Figure 6. Consistency score

A step of calculating CALC1 of a coherence score C1, C2, C3 is then carried out.

For each deconvolved image D1, D2, D3, a calculation of a coherence score is carried out. Preferably, this coherence score is representative of the coherence of the deconvolved image. By evaluating the coherence of each deconvolved image and selecting the most coherent image, we can deduce which point spread function is probably the fairest and therefore, at what distance from the detection surface 2 is the more likely object 10.

Once the scores C1, C2, C3 have been calculated, the method includes the DET1 determination of the best of the calculated scores.

In one embodiment, the characteristic height of the point spread function used to generate the best-scoring deconvolved image is selected as the distance Z1 between the object 10 and the detection surface 2.

In one mode of execution, the distance Z1 is determined from the predetermined height of the spreading function of the point which made it possible to obtain the deconvolution having obtained the best coherence score. For example, the distance Z1 can be determined between two characteristic heights of the two deconvolutions having reached the two best scores.

In another example, a continuous coherence score profile can be generated from the coherence scores of each deconvolution and the characteristic height of the point spread functions used for that deconvolution. The distance Z1 can then be determined from the local extremum of said coherence score profile.

In a simplified example, the image of the light intensity mapping illustrated in Figure 5 is deconvolved:

- by a first function PSF1, characteristic of a first height to obtain the first deconvolved image D1 illustrated in Figure 6;

- by a second function PSF2, characteristic of a second height to obtain the second deconvolved image D2 illustrated in Figure 6; - by a third function PSF3, characteristic of a third height to obtain the third deconvolved image D3 illustrated in Figure 6.

Among these three deconvolved images D1, D2, D3, the second image D2 is the most coherent. Indeed, sharp edges are more coherent than a diffuse object. We can therefore deduce that the shape of the object 10 is substantially similar to that of the light spot of the second image D2 and/or that the object 10 is at a distance from the detection surface equal to the second height (characteristic of the second spreading function of point PSF2).

The second spreading function of the point PSF2 is illustrated in Figure 15.

Criteria used for calculating the consistency score

Different criteria can be used to calculate said consistency score.

In one embodiment, the calculation of the coherence score includes a calculation depending on the sharpness of the edges of the light spot obtained on the deconvolved image D1, D2, D3. In one embodiment, the consistency score includes the calculation of a variance or any other statistical calculation method.

In one embodiment, any other method of calculating a coherence score can be imagined. The variance score can be an indicator of image quality. The quality indicator can, for example, be generated by a classifier. The classifier is configured to receive a deconvolved image as input and to generate a quality indicator as output. Preferably, said generated quality indicator is associated with said deconvolved image. The classifier can be implemented by a neural network trained from light intensity maps and/or deconvolved images, each associated with a quality indicator. Examples of neural network implementations are indicated later in the description.

The consistency score and/or the quality indicator is representative of a probability that the deconvolved image may be similar to a light intensity map. Using Fourier Transform

According to one embodiment, the CALC1 calculation step of a coherence score for each deconvolved image comprises the TRF application of a Fourier transform.

The coherence score C1, C2, C3, for each deconvolution is then calculated from the Fourier transform T1, T2, T3 of each deconvolution. Such a Fourier transform of a point spread function is also called an “optical transfer function”. It makes it possible to determine the distribution of light energy in image space and/or to determine the distribution of light energy in a space of spatial frequencies.

In another alternative or cumulative embodiment, the deconvolution step DEC1 comprises the TRF application of a Fourier transform of the light intensity mapping 4. Each deconvolution is then carried out from the Fourier transform of the said light intensity mapping.

The application of a Fourier transform advantageously simplifies the calculations of the coherence score. Indeed, the spectral domain of an image highlights changes in light intensities. In particular, the amplitude of the optical transfer function (also called "modulation transfer function" advantageously makes it possible to characterize the fineness of the details of the light spot of the object. The optical transfer function advantageously makes it possible to highlight the spatial frequency distribution in the deconvolved image. Thus, the wider the spatial frequency distribution, the more sharp edges the corresponding deconvolved image includes.

A Fourier transform of a deconvolved image can take the appearance of a 2D map whose intensities are a function of the frequency distribution or can take the appearance of a graph comprising 2 abscissa axes representing the two x axes, y of the space of the light intensity mapping and frequencies on the ordinate.

In one example, the coherence score is calculated from a Fourier transform peak width at a predetermined peak height (e.g. peak width at half height or peak width at base) or a combination of several peak widths at different heights predetermined. In another example, the coherence score can be calculated from the Fourier transform amplitudes.

Other types of transforms can be used to replace the Fourier transform for calculating the consistency score. For example, the Laplace transform can be used to calculate the consistency score.

Alternatives for determining object height

In one embodiment, another type of function can be used in place of the point spread function.

In another embodiment, instead of the step of carrying out a deconvolution of the acquired light intensity mapping, the coherence score is calculated directly from the selected function and the light intensity mapping.

In another embodiment, the height determination is calculated directly from the acquired light intensity mapping.

For example, determining the distance of said object with the detection surface includes calculating a gradient map from the acquired light intensity map.

The calculation of a gradient map may include, for each pixel of the acquired light intensity map, the calculation of a spatial gradient value. Preferably, the calculation of said spatial gradient of a pixel is a function of a differential between the value of the intensity of said pixel with the values of the intensity of successive neighboring pixels of said pixel.

In one example, the value of the gradient of said pixel is a function of a sum between the absolute value of a differential between said pixel and its two closest neighbors along a first axis and the absolute value of a differential between said pixel and its two nearest neighbors along a second axis perpendicular to the first axis.

The value of said gradient is thus calculated for each pixel of the light intensity mapping, thus defining a gradient mapping. It is understood that the example described involves the calculation of a gradient value for each pixel but may alternatively include the calculation of a gradient value for each unit of the light intensity mapping. Said unit comprising a pixel grouping. An advantage is to allow greater speed of calculation at the expense of the definition of the gradient mapping.

From the gradient mapping, an extremum is determined, for example by applying a filter or an extremum search function.

In one example, the extremum may include the pixel of the gradient map for which the calculated gradient value is the highest.

In one embodiment, the distance Z1 of the object 10 with the detection surface is determined from the extremum of the gradient mapping and/or from a function taking into account the value of the gradient for each pixel and the position of said pixels on the detection surface.

Preferably, the spatial coordinates in the plane of the detection surface are determined from the coordinates of the extremum determined on the gradient mapping.

Determining the shape and coordinates of the object in the plane of the detection surface

Once the deconvolution having obtained the best score has been identified, it is possible to determine from said image, certain additional information on the object 10 to be detected. For example, the shape of the object and/or its spatial coordinates in the plane of the detection surface can be estimated from the deconvolution having obtained the best score (in particular from respectively the shape and/or spatial coordinates of the light spot).

The first method according to the invention thus makes it possible, from a detection surface as described above, to determine the height (and optionally the shape and/or spatial position) of an object 10 placed in front of said detection surface 2.

The electronic device 1 according to the invention comprises software and/or hardware means for implementing the method according to the invention. The software resources may include processors, calculation units, memories (in particular non-transitory memories). Presence of two objects

In one embodiment, the method includes detecting a second object placed in front of the detection surface.

To this end, the method can include the detection on the acquired light intensity map of a second object, for example by the detection of several local extrema. The method may include a segmentation of the acquired light intensity mapping and for each segmentation, the generation of a distance Z1 for each object as described previously.

Method for generating a point cloud of the object

According to a second aspect, the invention relates to a second method 300 for generating a point cloud NP1 representing an object 10 located in front of the detection surface 2 of the device 1 according to the invention.

Emission patterns and acquisition of a light intensity map

The second method 300 comprises the generation of a first emission pattern P1, in particular by controlling the ignition of a first set of photoemitters of the detection surface.

Examples of several sets of photoemitters 31 are illustrated on the figure 11.

The first set of photoemitters includes a selection of photoemitters. The first set of photoemitters may include the photoemitters located in a portion of the predefined detection surface. In a first example, the emission pattern P1 comprises the activation of a set 34 of photoemitters forming approximately half of the detection surface 2. In a second example, the emission pattern P1 comprises the activation of a set of photodetectors 31 forming a plurality of zones 35, 36 separated from each other. Said zones can be separated from each other by zones whose photoemitters are not activated. Said zones are preferably separated from each other by a distance substantially equal to one of the dimensions of said zone. Preferably, said zones form geometric shapes such as polygons or quadrilaterals. An advantage of such patterns is to allow the object 10 to be illuminated in different ways. In particular, object 10 will be more or less illuminated on its different portions, thus increasing the information on object 10 in the light intensity maps.

Preferably, the photoemitters of the detection surface 2 other than those of the first set defined by the emission pattern are not activated or remain off during the generation of the first emission pattern P1 or during the ignition of the set of photoemitters according to the first emission pattern.

In one embodiment, an emission pattern may include turning off all photoemitters. Such an emission pattern advantageously makes it possible to measure the surrounding noise detectable by the photodetectors. In one embodiment, the light intensity map obtained is used as a filter for processing the light intensity maps subsequently obtained. Such filtering advantageously makes it possible to obtain a better signal-to-noise ratio.

In one embodiment, the electronic device 1 comprises a memory, preferably a non-transient memory. A library or a database comprising a plurality of different patterns P1, P2 is recorded on said memory.

In one embodiment, the memory is connected to a processor and/or to means for controlling the activation of the photoemitters. The means for controlling the activation (or ignition) of the photoemitters is configured to activate the photoemitters according to the selected emission pattern.

The second method includes an acquisition ACQU1 of a light intensity map similar to that of the first method 200. Preferably, the acquisition is carried out simultaneously with the switching on of the photoemitters 31 according to the first emission pattern P1.

Preferably, the acquired light intensity map 41 is associated with a label comprising indications making it possible to identify the first emission pattern P1 generated to acquire said light intensity map 41. Building function

In a first mode, the second method 300 further comprises the generation of a point cloud NP1 of the object 10 located in front of the detection surface 2. Said point cloud is generated from said acquired light intensity mapping 41 and from the first emission pattern P1 implemented for the acquisition of said mapping 41. The generated point cloud NP1 is a representation of said object 10 placed in front of the detection surface.

In a second cumulative mode or alternative to the first mode, the second method 300 further comprises the generation of at least one point of the object 10 located in front of the detection surface 2. Said at least one point is generated from said acquired light intensity mapping 41 and from the first emission pattern P1 implemented for the acquisition of said mapping 41. The at least one point generated, in particular its coordinates is a representation of the position of said object 10 placed in front the detection surface in space.

This step of generating a cloud of points NP1 or at least one point is implemented by a construction function FCT1. The FCT1 function is designed to receive as input at least one light intensity map 41 and an emission pattern P1 associated with said map 41. The FCT1 function is designed to, at output, generate a cloud of points NP1 representing the object 10 placed in front of the detection surface 2 during the acquisition step ACQU1 or at least one point representative of the position of the object 10 placed in front of the detection surface 2 during the acquisition step ACQU1. Preferably, the construction function generates a cloud of points NP1 or at least one point of the object 10 in a reference frame comprising the detection surface 2. An advantage is to reproduce a cloud of points representative of the object 10 or at least a point representative of the position of the object 10 and comprising distance and orientation information relative to the detection plane.

The construction function FCT1 is preferably implemented by a computer of the electronic device 1 according to the invention. A memory, connected to said computer, may include code information to execute said construction function FCT1 when it is implemented by the computer or by the electronic device 1. In one embodiment, the construction function generates a cloud of points NP1 for each set of light intensity maps acquired, preferably in real time.

Plurality of emission reasons

According to one embodiment, the second method 300 comprises the acquisition ACQU2 of a second light intensity map 42 by the same detection surface as that used for the acquisition of the first light intensity map, more particularly by the same photodetectors as those used for the acquisition of the first light intensity map.

The second method then comprises an MOT2 generation of a second emission pattern P2 by controlling the ignition of a second set of photoemitters different from the first set of the first emission pattern.

The acquisition ACQU2 of the second light intensity map 42 is carried out simultaneously with the generation of the second emission pattern MOT2 to make it possible to acquire the light intensity emitted by the second pattern P2 and reflected by the object 10 for which we wish to generate a cloud of points NP1.

Said second light intensity mapping 42 can be associated with the second emission pattern P2 different from the first pattern P1. Alternatively, the second light intensity map 42 is associated with a label comprising identification data of the second emission pattern P2.

In one embodiment, the method comprises the acquisition of a plurality of light intensity maps, each carried out simultaneously with the generation of an emission pattern by controlling the ignition of a set of photoemitters of the detection surface different from the sets characterizing the other emission patterns.

Said second acquisition ACQU2 is carried out within a very short period of time from the first acquisition ACQU1.

Preferably, the second acquisition ACQU2 and the first acquisition ACQU1 are both carried out in a time period of less than 5 ms, very preferably less than 500 ps. Preferably, the plurality of acquired light intensity maps forming a set of light intensity maps are all acquired in a time period of less than 10 ms, very preferably less than 4 ms.

The advantage is to obtain two light intensity maps or a set of light intensity maps of the same object 10 which has not had time to move. Thus, the invention advantageously makes it possible to generate a plurality of point clouds NP1 of the object over time. The comparison of successive point clouds makes it possible to follow the movement of said object as explained below.

These real-time light intensity maps acquired according to different emission patterns can be grouped into a set of light intensity maps. A set of light intensity maps comprises two or a plurality of light intensity maps acquired in a very short time interval (less than 10 ms or less than 4 ms) and from different emission patterns P1, P2 . This set of light intensity maps advantageously allows the construction function FCT1 described below to generate a cloud of points NP1 of the object 10 in a more reliable and more precise manner.

Indeed, taking several light intensity maps with different emission patterns makes it possible to provide more information on the object 10 to be detected. The resolution and reliability of the generated NP1 point cloud are therefore advantageously improved. Likewise, it is thus possible, from two light maps acquired from two different emission patterns, to determine by the construction function at least one point determining the position of the object in space with respect to the detection surface.

In one embodiment, the construction function generates a cloud of NP1 points for each set of light intensity maps acquired, preferably in real time.

Simulation and generation of an image

Different modes of execution of the construction function FCT1 are now described. For the sake of clarity, these different modes are called first, second and third FCT1 construction function. The first construction function FCT1 comprises the determination of a cloud of points NP1 by a function for minimizing images obtained from the simulation of the light rays reflected by the object 10.

The first construction function then includes the selection of a first candidate point cloud NPO. This first candidate point cloud can represent a first estimate of the object 10.

This first NPO candidate point cloud is used to carry out an acquisition simulation of a light intensity map.

A simulation of emission of light rays by the first set of photodetectors of the first emission pattern P1 is carried out.

The candidate point cloud NPO is used to simulate a candidate virtual object. The SIM1 simulation thus comprises the simulation of the light emitted by a detection surface according to the first emission pattern and reflected by a surface formed by said cloud of NPO candidate points. The simulation finally includes the determination of the light intensity detected by each photodetector, in particular the light intensity resulting from the rays reflected by the object 10.

A first image 411 is then produced. The first image is preferably produced as a function of the position and the light intensity received by each photodetector during the simulation of a first emission pattern.

In one embodiment, the simulation includes simulating a virtual environment including a virtual sensing surface similar to the sensing surface of the electronic device. In particular, the virtual detection surface reproduces the arrangement of the photoemitters and photodetectors of the detection surface of the electronic device 1. Each photoemitter and each photodetector of the detection surface of the electronic device 1 has an equivalent on the virtual detection surface.

Preferably, the optical properties and the detection properties of the photoemitters and photodetectors of the detection surface of the electronic device 1 are reproduced by the virtual detection surface.

The production of the first image 411 is therefore, in other words, a simulation of a light map which would have been generated if the first candidate point cloud NPO is a faithful reproduction of the object 10 to be detected.

The SIM1 simulation therefore preferably includes a simulation of light emission by the photodetectors of the first set, the reflection of these light rays on the first candidate point cloud NPO, and the GEN1 generation of the first image 411 from the reflected light rays simulated and the simulation of their detection by the photodetectors.

In the embodiment where a second light intensity map 42 is acquired from a second emission pattern P2, a second image 421 is generated in the same way on the basis of a SIM2 simulation from said second emission pattern P2 and the same first candidate point cloud.

In the embodiment where a plurality of light intensity maps are generated based on a plurality of light patterns, a plurality of images are generated, each of these images is generated in the same manner based on a simulation from a different emission pattern.

Comparison of generated image and mapping

The first construction function FCT1 optionally includes a comparison step.

The COMP1 comparison includes the comparison of the generated image 411 with the acquired light intensity map 41. In particular, the comparison COMP1 includes the comparison of the image generated 411 from the first emission pattern P1 used to acquire the first light intensity map 421.

In one embodiment, the comparison COMP1 comprises the generation of a comparison indicator V1. The comparison indicator V1 is generated according to said comparison operation. In one embodiment, the comparison comprises the comparison of the intensities of each pixel of the generated image 421 with that of the corresponding pixels of the acquired light intensity map 41.

In the embodiment where a second light intensity map 42 is acquired, a second comparison COMP2 is carried out between said second light intensity map 42 and the second image generated from the second emission pattern P2. A second comparison indicator V2 is thus generated. Alternatively, a single comparison indicator V0 is generated as a function of the first V1 and the second comparison V2.

In the case where a plurality of light maps is acquired in the time period described above, the first construction function FCT1 may include a plurality of comparisons. Each comparison comprises the comparison of an acquired light intensity map with an image generated from the same emission pattern as that used for the acquisition of said light intensity map.

Minimization of the comparison result by varying the point cloud

The result V1, V2 of the comparison step(s) is then used to carry out a step of minimizing said result. Said result may include the comparison indicator(s).

When the result V1 of the comparison is less than or greater than a predetermined target value, the generated image(s) 422, 421 are considered sufficiently close to the acquired light intensity maps and the candidate point cloud is considered as being sufficiently close to reality. In this case, the construction function FCT1 generates a point cloud NP1 of said object 10 equal to the candidate point cloud NPO. In an alternative mode, the generated point cloud NP1 is a function of the candidate point cloud NPO and the result of the comparison V1.

When the result of the comparison does not reach a predetermined target value, a step of minimizing the result V1 of the comparison is preferably carried out.

A mode of execution of the minimization step MIN1 is now described. In this mode, the minimization includes the selection of a new candidate point cloud Anp. A new SIM11 simulation step and GEN11 generation of an image 412 are carried out in the same way as described previously, this time on the basis of the new candidate point cloud Anp. A new comparison step COMP11 is then carried out by comparing said new image 412 generated and the light intensity map 41 acquired with the same emission pattern P1.

A new result of the comparison V11 is generated from the new comparison step COMP11.

If the new result of the V11 comparison still does not reach the predetermined target value, a new loop (selection of a new candidate point cloud, simulation, generation of a new image, comparison between the new image and the mapping of acquired light intensity) can be generated.

In one execution mode, the selection of a new candidate point cloud Anp depends on the result of the previous comparison.

The step of minimizing the comparison result is carried out by varying the candidate point cloud NPO, Anp.

Preferably, the minimization step includes a gradient descent step. The selection of a new candidate point cloud Anp can be carried out in such a way that the variation between the new candidate point cloud Anp and the previous candidate point cloud NPO is a function of the difference between the result of the comparison and the predetermined target value.

The gradient descent method therefore iteratively generates new candidate point clouds Anp to minimize the result of the comparison function between the image generated from a candidate point cloud and the light intensity mapping acquired.

In the mode where two or more light intensity maps are acquired, the gradient descent method therefore iteratively generates new candidate point clouds Anp to minimize the result of the comparison function between each generated image and the light intensity mapping acquired from the same emission pattern.

Once the result of the comparison function reaches a predetermined target value or once the comparison function reaches an extremum, the point cloud NP1 of the object 10 is generated according to the candidate point cloud for which the comparison function has respectively reached the target value or said extremum.

In one execution mode, the first NPO candidate point cloud is selected from a pre-recorded point cloud. According to another mode, the first predetermined point cloud is generated from a (preferably the last) point cloud NP1 of the object generated by the second method or by the construction function FCT1 previously, preferably during the period of previous time. Indeed, if the object 10 has moved only slightly between the two successive acquisitions or between two successive periods of time, we can consider that the last cloud of points NP1 can serve as a basis for the generation of the new cloud of points NP2 .

In another mode of execution, the first cloud of candidate points NPO is generated according to the shape of the object 10 and/or its spatial coordinates determined by the first method according to the invention.

Neural network

An alternative mode of execution of the construction function FCT1 is now described, here called second construction function in which said second construction function is a learning function or is implemented by means of such a learning function.

The learning function is preferentially trained by a series of light intensity maps. In one example, the learning function was configured from learning comprising the submission of a plurality of light intensity maps (optionally acquired by a device similar to the electronic device according to the invention), each intensity map light being associated with a first label representing a cloud of points representing the object 10 having made it possible to obtain said light intensity mapping. Preferably, the label may also include information characteristic of the acquisition method which made it possible to obtain the associated light intensity mapping. For example, the first label may include information representing the emission pattern used for the acquisition of said light intensity map. This information can alternatively be included in a second label associated with each light intensity map. In one embodiment, the step of generating a point cloud NP1 of the object 10 comprises the generation of a correspondence score for a plurality of candidate point clouds. The match score may include a probability that the candidate point cloud is compatible with the acquired light intensity mapping. The execution of the second construction function then includes the selection of the candidate point cloud for which the correspondence score is the best.

In one embodiment, the learning function generates a cloud of points NP1 for each set of light intensity maps acquired, preferably in real time.

In one embodiment, the second construction function is implemented by means of a neural network trained by a series of sets of labeled light intensity maps. The advantage of such a second construction function is that it allows the generation of a cloud of NP1 points of the object from training using labeled light intensity maps.

In one example, the neural network is a convolutional neural network, called CNN or Convet.

According to one embodiment, the configuration of the CNN neural network may include:

■ Convolutions or neural network layers comprising a plurality of multiplications of matrices comprising weighting coefficients obtained from a learning process;

■ non-linear operations.

According to one embodiment, the configuration of the CNN neural network includes as input(s) light intensity maps acquired or received or stored in a memory. Preferably, each light intensity map acquired or received is associated with the emission pattern used for its acquisition.

The CNN neural network can include convolutions in its first layers and then layers of fully connected neurons, called “fully connected layers” at the end of the model. In the latter case, these are neurons connected to all the neurons of the previous layer and connected to all those of the next layer. The convolution layers may include a scan of an input matrix producing a series of matrix calculations. The other layers of the neural network generally include matrix calculations based on the size of the input matrix.

According to one example, each convolution includes a matrix product between an input matrix and a weight matrix and the consideration of an additional bias.

Applying layered processing within the CNN neural network involves applying a series of matrix multiplications which are followed by a non-linear function to produce an output of said layer. The succession of these operations defines the depth of the neural network.

According to an exemplary embodiment, the neural network is a multi-layer perceptron, known by the acronym MLP and in English terminology by “multi-layers perceptron”. According to one example, the neural network may be a network equivalent to the MLP.

According to another example, the neural network is a recurrent neural network. Preferably, the architecture of the neural network is a long-short-term memory (known by the acronym “LSTM” for “Long Short-Therm Memory” in English). This type of architecture has feedback connections. An LSTM unit may include a cell, an input gate, an output gate, and a forget gate. The three gates regulate the flow of information into and out of said cell. According to one example, the neural network can be an equivalent to LSTM.

According to another example, the neural network is a convolutional neural network of the “U-NET” type, that is to say a network based on an entirely conventional architecture. This type of neural network is particularly advantageous for performing precise image segmentation and for operating with less training data.

In one embodiment, a cell of the neural network receives first input data and generates first output data. The first input and output data are then recorded in a memory (preferably a non-transient memory). When new input data is received, new output data is generated. This new output data can be generated by function of the new input data and the data recorded in the memory (i.e. the first input data and previous output data). By “cell” we also mean a layer or a block of a neural network.

In one embodiment, each time output data is generated, said output data and the input data used to generate them are time-stamped before being recorded in the memory.

The recurrent neural network, upon receiving new input data, generates new output data based on the new input data and based on the recorded data and its timestamp.

An advantage is to improve the speed and precision of the output data (for example, of the generated NP1 point cloud). Indeed, when generating a point cloud in real time, we can assume that the shape and position of object 10 are very close to those of the previous iteration. Such a neural network advantageously improves the calculation speed and the precision of the generated point cloud NP1.

In one embodiment, each acquired light intensity map is time-stamped. The timestamp can be used by the build function to generate an NP1 point cloud with a generated point cloud whose timestamp is closest to the timestamp of the acquired light intensity mapping.

Using a library

The third construction function includes the use of a database of acquired light intensity maps. Preferably, said database is recorded on a computer-readable medium such as a memory, preferably a non-transient memory. In one embodiment, each light intensity map in the database is associated with a label comprising a candidate point cloud having been used for the acquisition of said associated light intensity map. Preferably, each light intensity map in the database is associated with a second label comprising information representing the emission pattern used for the acquisition of said associated light intensity map. The implementation of the third construction function includes receiving the acquired light intensity map 41 and its associated emission pattern P1 and/or receiving the set of acquired light intensity maps.

The implementation of the third function may include a comparison of the acquired light intensity mapping with the light intensity maps of the database associated with the same emission pattern as the acquired light intensity mapping. A match score can be calculated for each comparison made and the light intensity map from the database with the best match score can be selected. The point cloud NP1 can then be generated according to the candidate point cloud associated with said light intensity mapping of the selected database.

In one embodiment, the light intensity maps acquired from the database are grouped into groups of light intensity maps. Preferably, each group includes light intensity maps sharing the same candidate point cloud.

When a set of light intensity maps is acquired, said set can be compared with sets of light intensity maps in the database. Preferably, each acquired light intensity map is compared with the light intensity map of the group in the database associated with the same emission pattern as the acquired light intensity map.

Preferably, a correspondence score is calculated for each light intensity mapping group in the database. The correspondence score can be a function of the comparison of each acquired light intensity map. The group of light intensity maps from the database can then be selected.

The point cloud NP1 can then be generated according to the candidate point cloud associated with the group of light intensity maps of the selected database. It is understood that each mode of the construction function described above can be adapted to generate at least one point representative of the position of the object 10.

Successive generations of a point cloud of the object

According to another aspect, the invention relates to a method for tracking the movement of an object 10 placed in front of the detection surface 2.

Said tracking method comprises a first generation of a cloud of points NP1 of said object 10 in front of the detection surface 2. Preferably, at least one of the methods described above is used for the acquisition of such a cloud of points NP1 .

The method comprises a second generation of a second cloud of points NP2 of said object 10. Preferably, at least one of the methods described above is used for the acquisition of such a cloud of points NP2.

The first point cloud NP1 is preferably generated from at least one light intensity map acquired on the first date. The second point cloud NP2 is preferably generated from at least one light intensity map acquired on a second date different from the first date, preferably later than the first date.

In a particular example, each point cloud NP1, NP2 is generated from a set of at least two light maps, each acquired from a different emission pattern.

Preferably, the method comprises the acquisition of light intensity maps continuously and the generation of point clouds NP1, NP2 in real time from said acquired light intensity maps.

By “continuously” we mean that the acquisition frequency and the point cloud generation frequency of the object 10 are sufficiently high to authorize the tracking of the movement of said object. Preferably, the method comprises the acquisition of a light intensity map or a set of light intensity maps as described above at a frequency greater than or equal to 100 per second. Point cloud processing

From the at least two generated point clouds of the object 10 or from a plurality of successive point clouds of the object, a movement parameter IC can be generated.

In one embodiment, a calculator of the electronic device 1 comprises software means for implementing a point cloud processing step NP1, NP2 of the object 10.

In a simplified embodiment of the invention, a calculator of the electronic device makes it possible to detect simple movements and/or simple positions and/or speeds of movement of the hand. This is the case for simple movements, for example of an arm going from left to right or up and down in relation to detection surface 2.

In an enhanced embodiment of the invention, the calculator of the electronic device is configured to detect hand postures, finger movements or complex gestures comprising a sequence of linked movements. The enriched embodiment may also include detection according to the simplified mode. The two embodiments can be combined.

According to one embodiment, the processing of the generated point cloud NP1, NP2 results in the generation of an image comprising at least points of interest of the user.

In one embodiment, the electronic device comprises a module for processing the generated point clouds. The processing module generates at least a second image or a second point cloud comprising points of interest or regions of interest extracted from the generated point cloud. This could be, for example, the end of a limb such as the ends of the fingers, points of articulation, shape contours, etc.

Generating a movement parameter

From the at least two generated point clouds of the object 10 or from a plurality of successive point clouds of the object, a movement parameter can be generated 400.

In one embodiment, the determination of a movement parameter IC comprises the detection of at least one type of movement of a user from the point clouds NP1, NP2 of the object 10 generated. Preferably, the detection of a type of movement of a user comprises a detection of a movement of the hand of a user. The detection of a movement is carried out from the point clouds NP1, NP2 of the object 10 generated by the electronic device 1.

By "from the generated point clouds NP1, NP2 of the object 10", we include here the raw point clouds NP1, NP2 as they were generated by the electronic device 1 as well as 3D images generated by the processing of these point clouds, for example from the generation of points of interest. We also include any two-dimensional image or depth map.

Preferably, the different types of movement are listed in a library. The computer can then carry out a fitting operation or an analytical regression operation to determine a particular type of movement from the captured images.

Figures 13A to 131 illustrate examples of types of hand movements. These types of movements are recognizable by a computer. The calculator generates a movement parameter IC from the type of movement detected.

Examples of types of hand movement may include a closed fist wrist rotation movement (Figure 13B), a rotation of the hand along a longitudinal axis of the forearm (Figure 13E), along an axis a perpendicular to the longitudinal axis of the forearm (Figure 13D and Figure 13F). Another example of a type of movement may include a transverse movement of the hand (Figures 131, 13H and 13G) relative to the detection surface 2, that is to say a movement having a non-zero component in a parallel direction at the plane of the detection surface 2.

The type of movement may also depend on the position and/or movement of the finger joints. For example, the type of movement may be different if the wrist rotation gesture is performed with an open hand or a closed hand. Certain types of movements can be associated with known gestures of personalities in the musical world or in the audiovisual world. Another example of the type of movement illustrated in Figure 13A may include closing the fingers of the hand with the fingers extended. Another example type of movement may include an oscillation of the hand so as to reproduce the movement of a wave, as shown in Figure 13C.

The type of movement can also depend on the direction of the gesture. For example, a translation movement gesture of the hand can be discriminated as a function of the plane and/or the direction of translation. For example in Figure 13G, the translation is carried out along the axis perpendicular to the palm of the hand. Figure 13H and Figure 131 illustrate a translation movement of the hand in the same plane parallel to the plane of the detection surface. One in the direction perpendicular to the longitudinal direction of the fingers (figure 13H) and the other in a direction parallel to the longitudinal direction of the fingers (figure 131). In another example, wrist rotation can be detected in-plane.

Preferably, the movement parameter IC also includes the determination of the speed and/or amplitude of the movement.

Preferably, the movement parameter IC is determined at least from the difference between two clouds of points acquired successively. In another embodiment, the movement parameter IC is determined at least from the difference between points of interest and/or regions of interest between at least two point clouds generated successively.

In an alternative embodiment, the movement parameter IC is calculated from the coordinates of the object 10 calculated by the method according to the first aspect of the invention, that is to say from its spatial coordinates X1 , Y1, Z1 relative to the detection surface 2. Preferably, the movement parameter IC is calculated from the variation of the spatial coordinates X1, Y1, Z1 of the object 10 between at least two acquisitions.

In another embodiment, the movement parameter IC is calculated from a single point cloud NP1. In this case, the movement parameter is calculated from the overall shape of said cloud of points. For example, a particular predetermined arrangement of the fingers relative to the palm of the hand can be detected from the points of the point cloud NP1 and the movement parameter is then calculated based on the detected arrangement. In another embodiment, the movement parameter IC is calculated directly from the estimate of the shape of the object 10 determined.

The shape of the object 10 can be determined from at least one acquired light intensity map. For example, light intensity mapping segmentation is implemented to identify the overall shape of the object. For example, the segmentation of the acquired light intensity mapping generates the position of at least a first finger in relation to the palm of the hand or in relation to a second finger. The movement parameter IC is then generated as a function of said generated relative position.

Using the motion setting on a display device

As seen previously, the electronic device may include a display screen.

In one embodiment, a movement parameter IC can control the display of said screen.

In the first example, the screen display includes the display of a pointer and the movement of said pointer is controlled from the detected movement parameters. For example, the device can be configured so that the detection of a type of predetermined movement of the hand causes a movement of the pointer or the selection of a digital object on which the pointer is positioned.

In a second example, the electronic device can be configured so that the detection of a predetermined movement parameter triggers the control of an enlargement (“zoom”) or shrinkage (“zoom out”) function of a digital object. displayed on the screen.

An advantage is to allow control of the screen display interface through hand movements without the need to touch the screen, thus providing a real alternative to traditional touch screens.

In a third example illustrated in Figure 14, the electronic device can be configured to display on a screen 3 a user interface S3, S2, S1. A user interface may include control areas 140 (such as virtual buttons) which, if selected, activate the launch of a pre-recorded program (launching an application, turning off the device, increasing the volume , ... ). Preferably, a plurality of user interfaces S1, S2, S3 are pre-recorded in a memory of the electronic device 1. In one embodiment, a user interface S2 is selected to be displayed on the screen 3 of the electronic device.

The selection of a user interface S2 among the plurality of interfaces is carried out as a function of the movement parameter IC and/or as a function of the determined distance Z1 between the detection surface 2 and the object 10. In the example illustrated in Figure 14, each pre-recorded user interface S1, S2, S3 is associated with a predetermined distance M1, M2, M3. The determined distance Z1 between the detection surface 2 and the object 10 is compared to said predetermined distances M1, M2, M3. For example, a user interface whose associated predetermined distance is closest to distance Z1 is selected. Once selected, the S2 user interface is displayed on screen 3 of the electronic device.

In another embodiment, each user interface is associated with a predetermined distance and a predetermined movement parameter. In this case, the selection of an interface is carried out as a function of the determined distance Z1 between the detection surface 2 and the object 10 and as a function of the movement parameter generated by the method according to the invention described previously.

A first advantage is to allow control of the screen display interface by hand movements without the need to touch the screen, thus providing a real alternative to traditional touch screens. A second advantage is to allow the user to navigate more quickly and easily between different user interfaces by simply varying the distance between their finger and the detection surface.

According to another aspect, the invention relates to a method of displaying a user interface on the screen of an electronic device according to the invention. Said method comprises determining a distance Z1 between the detection surface 2 of the electronic device and an object 10; providing a database in which a plurality of different user interfaces are recorded, each user interface being associated with a predetermined distance different from the others. The user interface to be displayed is selected by comparison between the determined distance Z1 and the predetermined distances associated with the user interfaces pre-recorded. The method finally includes displaying the selected user interface.

In one embodiment, the method further comprises determining a movement parameter IC and each pre-recorded user interface is associated with a combination of a predetermined distance and a movement parameter. In this case, the user interface to be displayed is selected by comparison between the determined distance Z1 and the movement parameter IC generated according to the invention and the combination associated with each pre-recorded user interface.

In one embodiment, the movement parameter may be replaced by a pose parameter. The pose parameter can be determined from a generated NP1 point cloud. The pose parameter can be generated from the shape of the NP1 point cloud and/or its orientation. An advantage is to be able to provide the electronic device with control elements by the shape of the hand placed above the detection surface, for example by closing or opening the fist or by pointing one or more fingers or by varying the direction of a finger relative to the plane of the detection surface 2.

Implementation of the processes according to the invention by the electronic device

The device according to the invention comprises at least one calculation unit. The at least one calculation unit 11, 12, 13, 14 is in communication with the photodetectors 31 and/or with the photoemitters. This communication advantageously allows the calculation unit to process the signals generated by the photodetectors 33 to process the measurements made by these photodetectors. This communication also advantageously makes it possible to control the activation of the photoemitters 31. By activation is meant the switching on or off of a photoemitter, that is to say the emission or not of light by the photodetector. In one embodiment, at least one calculation unit 11, 12, 13, 14 is connected to the means for controlling the photoemitters and/or is connected to means for recording the signals generated by the photodetectors. In one embodiment, each calculation unit includes a calculator. In another embodiment, the electronic device comprises one or more computers comprising the calculation units 11, 12, 13, 14. The electronic device 1 may include a calculation system 100, an example of which is illustrated in Figure 12. The calculation system may include the different units 11, 12, 13, 14 described. To this end, the calculation system may comprise one or more computers 11, 12, 13, 14 interacting with a computer program product, in particular to implement any of the methods according to the invention described above.

The system 100 is a computer, for example, a microcomputer, a computer network, an electronic component, a tablet, a smartphone or a personal digital assistant (PDA).

The system 100 comprises a calculation module 110. This calculation module 110 comprises, for example, one or more processors capable of interpreting instructions in the form of a computer program, a programmable logic circuit, such as an application-specific integrated circuit (ASIC ), an in situ programmable gate array (FPGA), a programmable logic device (PLD) and/or programmable logic arrays (PLA), a system on chip (SOC), an electronic card in which steps of the method according to The invention is implemented in hardware elements. The processing may be executed by a processor, simultaneously or sequentially, or by another method, by one or more processors.

The calculation module 110 comprises a data processing module 111 for carrying out calculations, a memory 112, operationally coupled to the data processing circuit 111, a computer-readable medium 114 and possibly a reader 113 adapted to read the computer-readable medium. computer 114.

The system 100 also includes an input device, an output device, and a communication device.

Each function of the system 100 is executed by causing the data processing module 111 to read a predetermined program on hardware such as the memory 112 such that the data processing module 111 executes calculations, controls communications carried out by the electronic device 1 and reading and/or writing data in the memory 112 and the computer readable medium 114. - A7 -

The method(s) according to the invention are executed on a single computer or on a system distributed between several computers (in particular via the use of cloud computing).

The memory 112 is a computer-readable recording medium, and can be configured with, for example, at least one of the following elements: a read-only memory (ROM), a read-only memory erasable and programmable (EPROM, from English Erasable Programmable Read-Only Memory), a programmable and electrically erasable read-only memory (EEPROM, from English Electrically Erasable Programmable Read-Only Memory), a random access memory (RAM, from English Random Access Memory) and another suitable storage medium. Memory 112 can include an operating system and load programs according to the invention. It includes registers suitable for recording parameter variables created and modified during the execution of the aforementioned programs.

The program product may include computer readable recording media 114 which is a tangible device, not being a transient signal per se, may be configured with, for example, at least one of the following: removable media, such as for example, in a non-limiting manner, a magneto-optical disc (for example, a read-only compact disc (CD-ROM, from English Compact Disc Read-Only Memory), a versatile digital disc or DVD (from English Digital Versatile Disc), a removable disk, a hard disk drive, a smart card, a flash memory device (e.g., a card, a key), a magnetic tape, a database, a server, and another suitable storage medium.

Alternatively, program instructions are taken from an external source and downloaded over a network. This is particularly the case for applications. In this case, the computer program product comprises a computer-readable data carrier on which the program instructions are stored or a data carrier signal on which the program instructions are encoded.

A computer program product includes computer readable media 114 containing instructions which, when executed by circuit 110, cause the device electronic 1 to implement the steps of one or more methods according to the invention.

The form of the program instructions is, for example, a source code form, a computer executable form or any intermediate form between a source code and a computer executable form, such as the form resulting from the conversion of the source code via a interpreter, assembler, compiler, linker or locator. Alternatively, the program instructions are microcode, firmware instructions, state definition data, integrated circuit configuration data (e.g. VHDL), or object code. Program instructions are written in any combination of one or more programming languages, for example, object-oriented programming language (C++, JAVA, Python), procedural programming language (C language for example).

The system 100 further comprises a user interface 120 comprising an input device 121 and an output device 122.

The user interface 120 includes an input device 121 to allow the user to enter data or commands so as to be able to interact with the programs according to the invention. The input device 121 includes, for example, a keyboard or a pointing interface, such as a mouse, an optical pencil, a touchpad, a remote control, a voice recognition device, a haptic device.

The output device 122 is designed to return information to a user, sensory or electrical, such as, for example, visually or audibly. The output device 122 may include the screen of the electronic device 1. The output interface 122 includes, for example, a graphical interface. The output interface 122 can be the input device, for example, in the case of a touchscreen tablet.

The set of at least one communication device allows communication between the elements of the system 100 and possibly between at least one element of the system and a device external to the system 100. This communication device can establish a physical link between elements of the system. system 100 and/or between an element of the system 100 and a device external to the system 100 and/or a remote (wireless) communication link between elements of the system 100 and/or between an element of the system and a device external to the system 100.

Claims

CLAIMS Electronic device (1) comprising:

■ a detection surface (2), in particular near field detection comprising: o a plurality of photoemitters (31), the photoemitters being separated from each other; o a plurality of photodetectors (33), at least a part of these photodetectors being arranged between said photoemitters (31);

■ a first acquisition unit (11) connected to the plurality of photodetectors (33) and the plurality of photoemitters (31) comprising means adapted for carrying out the following steps: o the generation of an emission (M0T1) by controlling the ignition of the photoemitters (31); o the acquisition (ACQ1) of a light intensity map (4) as a function of the position and the light intensity received by each photodetector;

■ a second detection unit (12) of an object (10) placed in front of the detection surface (2) comprising means adapted for determining a distance (Z1) between said object (10) and said detection surface (2). Device according to claim 1, characterized in that the second detection unit comprises means adapted for determining a distance (Z1) between said object (10) and said detection surface (2) by carrying out the following steps : o the selection (SEL1) of a plurality of point spreading functions (PSF1, PSF2, PSF3), each function being characteristic of a predetermined height; o for each function of the plurality of functions, calculating (CALC1) a coherence score (C1, C2, C3) from the light intensity mapping; o determining the distance (Z1) of said object (10) with the detection surface (2) from the predetermined height of the function that made it possible to obtain the best consistency score. Device according to claim 2, characterized in that the calculation of a coherence score comprises: o carrying out (DEC1) a deconvolution (D1, D2, D3) of the light intensity mapping acquired (4) by each selected function; o for each deconvolution, the calculation (CALC1) of a coherence score (C1, C2, C3). Device according to claim 3, characterized in that the calculation of a coherence score comprises the application (TRF) of a transform of the acquired light intensity mapping (4) and/or of the deconvolution of the mapping d acquired light intensity (4). Device according to claim 4 characterized in that the application (TRF) of a transform comprises the application of a Fourier transform or a Laplace transform. Device according to claim 2 characterized in that the second unit (12) also comprises means for executing a step of estimating the shape of the object (10) in a plane parallel to the plane of the detection surface (2) from the acquired light intensity mapping and/or from the deconvolution having obtained the best score. Device according to one of the preceding claims, characterized in that the second unit (12) also comprises means for executing a step of determining the spatial coordinates (X1, Y1) of the object (10) in a plane parallel to the plane of the detection surface (2). Device according to one of the preceding claims, characterized in that the detection surface (2) comprises a display screen formed by the plurality of pixels (5) and the photoemitters (31) and photodetectors (33) are distributed between said pixels (5).

9. Device according to claim 8, in which the pixels of the plurality of pixels (5) each comprise at least one visible light source (R, G, B) and in which the photoemitters (31) are designed to emit light infrared and the photodetectors (33) are designed to detect light intensity in the infrared range.

10. Device according to one of the preceding claims characterized in that the first unit (11) comprises means for executing the acquisition step (ACQU1) of a light intensity map (4) so that said acquisition step comprises a step of measuring, by each photodetector (33), the light intensity emitted (21) by the photoemitters (31) and reflected (22) by at least one object (10) placed in front of the surface detection (2).

11. Device according to one of the preceding claims, characterized in that further comprising a covering layer (25) covering the plurality of photoemitters (31) and the plurality of photodetectors (33), said covering layer (25) being transparent to rays emitted by the photoemitters (31).

12. Movement recognition system comprising an electronic device (1) according to claim 7, characterized in that it comprises a movement recognition unit (14) comprising means for carrying out the following steps:

■ the acquisition over time of a plurality of spatial coordinates (X1, Y1, Z1) of the object (10) placed in front of the detection surface (2) as a function of time;

■ the determination (PAR) of at least one movement parameter (IC) from:

■ a rotation of the object and/or a direction of translation of the object detected from a variation of the spatial coordinates (X1, Y1, Z1) acquired as a function of time; Or

■ the estimation of the overall shape of the object (10) determined from a segmentation of the acquired light intensity mapping.electronic positive (1) comprising

■ a detection surface, in particular for the detection of an object in the near field (2) comprising: o a plurality of photoemitters (31), the photoemitters being separated from each other; o a plurality of photodetectors (33), at least part of these photodetectors being arranged between said photoemitters (31);

■ a first acquisition unit (11) of a light intensity map (4) connected to the plurality of photoemitters (31) and to the plurality of photodetectors (33) comprising means adapted to execute the following steps: o the generation (MOT 1) of a first emission pattern (P1) by controlling the ignition of a first set (34) of photoemitters (31) of the detection surface (2); o the acquisition (ACQU1) of a first light intensity map (41) as a function of the position and the light intensity received by each photodetector (33) during the generation of the first emission pattern;

■ a second unit (13) for generating at least one point of said object placed in front of the detection surface comprising means for executing a construction function (FCT1) making it possible to generate at least one end point (NP1) of said object (10); characterized in that the first unit (11) and/or the second unit (13) comprises means for executing the following steps: o the generation (MOT2) of a second emission pattern by controlling the ignition of a second set of photoemitters (35, 36, 37) different from the first set (34); o the acquisition (ACQU2) of a second light intensity map as a function of the position and the light intensity received by each photodetector of the detection surface during the generation of the second emission pattern; and in which the generation of at least one end point (NP1) of said object (10) is carried out from the first emission pattern (P1), the first acquired light intensity mapping (41), the second pattern d emission (P2) and the second light intensity map (42).

14. Device according to claim 13, wherein the at least one point comprises a cloud of points, and the at least one final point (NP1) comprises a cloud of final points.

15. Device according to claim 14, characterized in that the execution of the construction function (FCT1) comprises the execution of the following steps: o the generation (GEN1) of a first image (411) by carrying out a simulation ( SIM1) of light rays emitted by the photoemitters (31) of the first set (34) and reflected by an object defined by a cloud of candidate points (NPO); o a first comparison (COMP1) of the first acquired light intensity map (41) with the first generated image (411); o a minimization (MIN1) of the result (V1) of the first comparison by varying the candidate point cloud (NPO); o determining a cloud of points (NP1) representative of said object (10) based on the minimization (MIN1).

16. Device according to claim 14 and claim 15, characterized in that the second unit (13) comprises means for carrying out the following steps: ■ the generation (GEN2) of a second image (421) by carrying out a simulation of light rays emitted by the photodetectors (31) of the second set and reflected by an object defined by the cloud of candidate points (NPO);

■ a second comparison (COMP2) of the second acquired light intensity map (42) with the second generated image (421); the minimization step (MIN1) comprising a minimization of the result of the second comparison (V2) and the first comparison (V1) by varying the candidate point cloud (NPO) of said object.

17. Electronic device according to one of claims 14 to 16, characterized in that the minimization step (MIN1) is implemented by a gradient descent method.

18. Electronic device according to one of claims 14 to 17, characterized in that the step of generating an image (421, 411) comprises the generation of a virtual object defined by a candidate point cloud (NPO) in a virtual environment comprising a simulation of a surface similar to the detection surface (2).

19. Device according to claim 14, characterized in that the construction function (FCT1) is a learning function, implemented by a neural network trained to generate a point cloud (NP1) of an object (10) to from one or more combinations each comprising a pattern (P1, P2) and an acquired light intensity map (41, 42).

20. Device according to claim 14, characterized in that it comprises a memory comprising a library of acquired light intensity maps, each associated with an emission pattern and a point cloud of an object, the function of construction (FCT1) being capable of generating said point cloud (NP1) by comparison with the maps of said library. Electronic device according to claim 14, characterized in that the second unit (13) comprises means for receiving at least one distance information (Z1) between said object (10) and the detection surface (2) and in which the cloud point (NP1) is generated by the construction function (FCT1) from said distance information (Z1). Motion recognition system comprising an electronic device (1) according to one of claims 14 to 21, characterized in that it comprises a motion recognition unit (14) comprising means for executing the following steps:

■ the acquisition of a plurality of end points (NP1, NP2) of the object (10) placed in front of the detection surface (2) as a function of time;

■ the determination (PAR) of at least one movement parameter (IC) from

■ a rotation of the object and/or a direction of translation of the object detected from the point clouds (NP1, NP2) acquired as a function of time; Or

■ the estimation of the overall shape of the object (10) determined from a segmentation of the acquired light intensity mapping.