KR102635069B1 - Single-pixel sensor - Google Patents

Abstract

This disclosure relates to performing facial recognition using a single-pixel sensor that measures the temporal signature of light pulses reflected from a target face. Due to sensor position and depth differences between different parts of the subject's face, reflections of short-lived illumination pulses from different parts of the subject's face will return to the sensor at different times, and thus Provides a time-based one-dimensional signature. By analyzing the reflection signature using neural networks or principal component analysis (PCA), recognition of the object can be obtained. Additionally, the same system can also be used to recognize or determine any other objects of known shape besides faces, for example products manufactured on a production line.

Description

Single-pixel sensor {SINGLE-PIXEL SENSOR}

This disclosure relates to a single-pixel sensor that measures the time signature of diffuse light pulses reflected from an object. Due to sensor location and depth differences between different parts of the object, reflections of the short-duration illumination pulse return to the sensor at different times, thus providing a unique time-based one-dimensional signature for each individual object.

Facial recognition systems typically use visual acquisition and recognition techniques to identify or authenticate a person by analyzing certain facial features (such as the relative positions of the nose, eyes, chin, and others) and comparing them against a reference or sample in a database. Use algorithms. They are most commonly implemented using image or video feeds along with a computing system that runs identification algorithms. They can be used standalone or in conjunction with other biometrics such as retina/iris and fingerprint scanning. The trend in smartphones is to add facial recognition as an alternative or authentication function in addition to personal identification number (PIN) codes and/or fingerprint. These systems often use two-dimensional (2D) cameras to acquire facial information, but 2D-only systems tend to fail with still images. Even when using added levels of security (eg, the requirement to blink one eye during recognition), these systems are usually not robust or secure enough to replace other biometric sensors.

This disclosure relates to performing facial recognition using a single-pixel sensor that measures the temporal signature of diffuse light pulses reflected from a subject's face. In some examples, a few such sensors may be used together, for example, to obtain different perspectives of the scene to be measured. Due to depth differences between the sensor location(s) and different parts of the subject's face, reflections of short-term illumination pulses from different parts of the subject's face will return to the sensor at different times, thus Provides a unique time-based one-dimensional signature for the target. By analyzing the reflection signature using algorithms such as principal component analysis (PCA) or artificial neural networks (ANN), recognition of the object can be obtained. Additionally, the same system can also be used to recognize or determine any other objects of known shape besides faces, for example products manufactured on a production line.

In view of the above, an example of the present disclosure provides a sensor for detecting an object, the sensor comprising: a light source arranged to emit short-duration pulses of light to illuminate the face of the object, 1 indicating attributes of the face of the object; - a light detector arranged to detect light from light pulses reflected from the face of said object to generate a dimensional time-based reflected signal, and to receive signals representative of the reflected signal and rely thereon to generate a time-based reflection signal for said object. and a processor arranged to generate a reflection signature. With this arrangement, the one-dimensional time-based reflection signal is used to allow a reflection signature to be generated, which can then be used in many applications, for example to recognize or authenticate an object.

For example, a recognition processor is provided, wherein the recognition processor is arranged to receive the time-based reflection signature and recognize the object in dependence on the reflection signature. Automated recognition of objects can be useful in many scenarios, including on production lines, and for security purposes.

In one example, the recognition processor uses machine learning techniques to recognize the object based on a time-based reflection signature. For example, the machine learning techniques may include at least one of a) PCA, and/or b) one or more neural networks. Other artificial intelligence techniques that can perform automated or assisted recognition of objects may also be used.

In one example, a reference channel feature is also provided, wherein the reference channel feature is arranged to receive at least a portion of a short-duration light pulse directly from the light source to provide a reference signal. The processor may then also be arranged to normalize the reflected signal depending on the reference signal to take into account undesirable characteristics in the illumination pulse. This arrangement allows irregularities in the illumination pulse to be automatically compensated, thus improving the detection and recognition performance of the sensor.

In a more detailed example, the light source may have a ringing characteristic, and the reference channel component includes a reference photodetector arranged to detect the ringing characteristic. The processor may then also be arranged to receive a signal generated by a reference photodetector in response to the ringing characteristic and normalize the reference signal to remove artifacts caused by the ringing.

In one example, a two-dimensional space-based recognition system may also be provided in combination with a one-dimensional (1D) sensor. The two-dimensional space-based recognition system may be arranged to capture a two-dimensional image of the face of the object and rely on it for recognition of the object. The recognition processor may then rely on recognition of the object using both the two-dimensional image and the one-dimensional time-based reflection signal to generate an output signal indicative of successful recognition of the object. With this arrangement, the 2D sensor and the 1D sensor can work together synergistically, such that recognition of an object occurs using the 2D sensor, and the 1D sensor ensures that the recognized scene also contains sufficient depth; , therefore it is not simply a photo or other 2D image of the object to be recognized.

In one example, the object is a human subject, and the object's face is the human subject's face. In this regard, examples of the present disclosure may be designed particularly for recognizing faces of human subjects and for the purpose of automatically unlocking portable electronic devices, such as mobile phones or tablet devices, among others.

In an example, the recognition processor may store object-specific one-dimensional time-based signal trace data (or a corresponding mathematical template) against which the time-based reflection signature (or a corresponding template) is compared to recognize the object. there is. In particular, the object-specific time-based signal trace data may include respective sets of samples of each object-specific time-based signal trace captured during a training phase, wherein the recognition processor is configured to generate signal traces to match. to interpolate between sets of samples (e.g., sampled data points).

From another aspect, further examples of the present disclosure provide a method of operating a single-pixel sensor: illuminating an object with a short-duration pulse of light, light reflected from the entire illuminated face of the object illuminated by the pulse of light. Detecting light from a light pulse reflected from the object at a single-pixel sensor to obtain a one-dimensional reflection time trace representing, stored representations of one-dimensional reflection time traces obtained from known objects and said Comparing one-dimensional reflection time traces, and then identifying or authenticating a stored time trace as an object matching the obtained time trace.

In one example, the identification includes using machine learning techniques to recognize the object based on the obtained time trace. For example, the machine learning techniques may include at least one of a) PCA and/or b) one or more neural networks.

In a further example, the method includes receiving at least a portion of a short-duration optical pulse directly from the light source to provide a reference signal, and relying on the reference signal to take into account undesirable characteristics in the short-duration optical pulse to determine the reflection time. It may further include normalizing the trace. In particular, the short-duration optical pulse may have a ringing characteristic, and the method may further include detecting the ringing characteristic and normalizing the reflection time trace to remove artifacts caused by the ringing. .

In a further example, the method may also be used in combination with a 2D sensor to allow 2D sensor confirmation of findings from a 1D sensor, or vice versa. In particular, in one example, the method may further comprise capturing a two-dimensional image of the object's face and relying thereon for recognition of the object. Successful recognition of the object can be achieved by relying on recognition of the object using both 2D images and 1D time traces.

In one example, the method further includes storing object-specific one-dimensional time-based signal trace data against which the one-dimensional time trace is compared to recognize the object. In particular, in a more detailed example, the object-specific time-based signal trace data comprises a respective set of samples of each object-specific time-based signal traces captured during a training phase, and the method determines the signal to match. It further includes interpolating between sets of samples (e.g., sampled data points) to reproduce traces.

From a further aspect, another example of the present disclosure provides a sensor system comprising: a single-pixel sensor arranged to capture a one-dimensional reflection trace corresponding to light reflected from a target user's face from a short-duration illumination pulse; a two-dimensional image sensor arranged to capture a two-dimensional image of the target user's face; and one or more devices arranged to control a device of which the sensor system forms a part to identify the target user in dependence on the captured two-dimensional image and the captured one-dimensional reflection trace, and to operate in dependence on the identification. Includes processors. With this arrangement, the data from the one-dimensional sensor is used to ensure that facial recognition is made based on two-dimensional image data, for example the two-dimensional sensor system is not spoofed by a photo or other image of the target user. It can be used to confirm that it is not.

In further examples of the present disclosure, a plurality of single-pixel sensors may be provided arranged to capture respective one-dimensional reflection traces corresponding to light reflected from the target user's face, and one or more processors may also be provided for each It is arranged to identify the target user relying on one-dimensional reflection traces. Providing a plurality of these single-pixel sensors allows the sensors to have a slightly different field of view of the target user, thus increasing the accuracy of identification or recognition of the user, and/or the time required to capture appropriate reflection trace samples. The amount should be reduced.

Additional features, examples, and advantages of the present disclosure will be apparent from the following description and the appended claims.

In order to provide a more complete understanding of the present disclosure and its features and advantages, reference is made to the following description, taken together with the accompanying drawings, in which like reference numerals indicate like parts:
1 is a diagram illustrating the concept of operation of an example of the present disclosure;
Figure 2 is a graph illustrating signals derived from the example of Figure 1;
3 is a block diagram of an exemplary system of the present disclosure;
4 is a flow diagram illustrating operations or portions of an example system of the present disclosure;
5 is a further flow diagram illustrating the operation of portions of an example system of the present disclosure;
6 is a flow diagram illustrating operations or portions of an example system of the present disclosure;
7 is a system block diagram of a further example of the present disclosure;
Figure 8 is a graph illustrating relevant photodetector responses used in examples of the present disclosure;
9, 10, and 11 are simulated responses and associated decision forms in a theoretical example of the present disclosure.
12 is a simulated bandwidth limited pulse response showing as-received and normalized versions of the response illustrating example operation of the present disclosure.
13 is a simulated response characteristic from an example of the present disclosure.
Figure 14 is the actual response characteristic, after digitization, from an example of the present disclosure illustrating the characterization of response curves;
Figure 15 is a flow diagram illustrating the recognition process used in examples of this disclosure; and
Figure 16 is a table listing use cases for examples of the present disclosure.

This disclosure relates to a new type of sensor for detecting the shape of objects, such as human faces. The sensor comprises a single photodetector operating as a single-pixel sensor, with an illumination flash operating to temporarily illuminate the object to be detected with short diffuse illumination pulses. The sensor then records the temporal reflection waveforms returning to the sensor from the entire portion of the illuminated object, and the waveforms (or corresponding mathematical templates) are then stored corresponding to the objects to be recognized to allow object recognition to occur. It can be compared to signature waveforms (or templates). A single-pixel sensor does not scan across the object, taking samples from small portions (e.g., a raster scan across the object), but instead takes samples from different parts of the object at different depths to the sensor. Note in particular that reflections are received from the entire illuminated face of the object illuminated by the flash, depending on the temporal difference in the arrival time of the reflected light. The object can then be characterized and recognized by looking at the received temporal waveform and identifying small changes therein due to the different relative depths (relative to the sensor) of the reflective parts of the object.

More specifically, examples of the present disclosure utilize a single photodetector (photodiode (PD), avalanche photodiode (APD), silicon photomultiplier (SiPM), or single-photon avalanche diode (SPAD)). It is based on analyzing the time response from a diffuse short laser pulse reflected from a received object (such as a face). The received electrical response is the convolution of the laser pulse with a vector containing depths within the field of view (FOV). Although it is often not possible to reconstruct the target object, at least with a single fixed exposure, the signal provides a signature that can be used with algorithms to determine the object and perform facial recognition. Such systems can, in some instances, operate with 2D imagers, offering the potential advantages of smaller, less expensive, and lower power consumption implementations compared to 3D imagers, such as structured light imagers.

1 and 2 illustrate the principles of operation of examples of the present disclosure in more detail. Referring first to Figure 1, imagine that a light emitter is provided with a co-located light sensor right next to it. The co-location of the emitter and sensor causes light to reflect back from the object to the sensor at an angle alpha (α) that is approximately equal to the angle of incidence of the rays from the emitter. Next consider an example object with a surface that provides multiple depths D ₁ to D ₅ to light incident on it from an emitter. The emitter and sensor are at a reference depth (D _REF ) from the rear surface of the object, with the exemplary depths all representing surface reliefs (D ₁ to D ₅ ) of the object that are measurably smaller than D _REF .

In operation, the emitter generates short pulses of diffuse light, typically on the order of nanoseconds, but better results are obtained with illumination as short as possible, ideally measured on the order of picoseconds. The light pulse illuminates the surfaces of the object facing the emitter, and the light reflects back from these surfaces to a co-located sensor. As shown in Figure 2, light from different surfaces with different relative depths to the emitter returns to the sensor at different times. Within Figure 2, the light pulse emitted by the emitter is shown on the top trace, and the bottom trace then shows the time it takes for the sensor to detect light reflected from the light pulse from different depth surfaces of the object. Thus, for example, light from a surface that is closer to the emitter (D ₁ ) is reflected at time TD1, while light from a surface farther from the emitter (D ₂ ) is reflected at time TD2. ) is reflected back to the sensor. Similarly, for the light reflected from surfaces D ₃ , D ₄ , and D ₅ , this is then received back at the sensor at times TD3, TD4, and TD5 respectively. The time trace of the reflected light therefore provides a characteristic signal representing the different depth surfaces of the object, which can then act as a signature time trace of the reflected light that is characteristic of the object. As shown in Figure 2, with the theoretical infinite bandwidth of the sensor, separate pulses from each surface at different depths can then be discriminated within the trace. However, in real life, the sensor will have limited bandwidth, resulting in significant rise and fall times for the signal generated by the sensor in response to incident light, with the result that the signature trace will be closer to that shown by the dashed line in Figure 2. cause However, these signature traces are still characteristic of the object's shape and can be used for object characterization and recognition. Object characterization and identification can be performed by storing data corresponding to bandwidth-limited signature traces of multiple objects and then comparing the stored signature traces to measured traces from the illumination.

An exemplary sensor system of the present disclosure will now be described with respect to FIGS. 3-6. In this regard, FIG. 3 illustrates an example of a single-pixel object recognizer (SPOR) of the present disclosure, in this example shown for performing face recognition (and therefore often referred to herein as a single-pixel face recognizer (SPFR)). Although a block diagram illustrates one example arrangement, it should be understood that other object shape determination and recognition may be handled by the system described in other examples. Additionally, in this example, the single-pixel sensor is integrated with a 2D image sensor, such as a camera, but it should be understood that in other examples, the single-pixel sensor may be used by itself.

As shown in Figure 3, in this example the SPOR (SPFR) system is formed by three main sub-systems:

1. Weird Signature Processor 1, which measures depth changes for the illuminated part of the face;

2. 2D camera 2, which associates a visual image of the face; and

3. Application processor 3, which processes the depth signature and visual cues to authenticate the user.

The depth signature processor (1) operation is as follows. The SPFR processor (1.11) is an illumination driver (1.1) that drives a laser diode (LD) or vertical-cavity surface-emitting laser (VCSEL) (1.2) with the shortest possible pulse and optimal power level to cover the desired area. The operation is initiated by transmitting an activation signal to . It also requires ensuring good optical signal integrity. In its simplest form, the lighting driver (1.1) can be a gallium nitride field-effect transistor (GaN-FET) with passive current limitation, while more sophisticated implementations may be LD or vertical-cavity surface-emitting laser (VCSEL) peak activating. It may be a high-speed, high-current digital-to-analog converter (DAC) capable of selecting current. Because of the fast di/dt, the lighting driver (1.1) and LD/VCSEL are ideally co-packaged or use direct junction to minimize parasitic inductance.

The optical pulse generated by the LD/VCSEL (1.2) is split into two paths, the first path (b1) is generated by the waveguide (1.13), detected by the photodetector (1.7) and trans-impedance amplifier ( The signal amplified by TIA (1.8) will be used as a reference. The second path b2 is directed to the light diffuser 1.3, which emits light at an angle sufficient to cover the surface face area 4.1. A light diffuser (1.3) can be integrated into the LD/VCSEL or the emission characteristics are such that a diffuser is not required.

The light reflected from the illuminated area (4.1) is captured by a photodetector (1.5) such as PD, APD and amplified by a TIA (1.6), the reference and measured signals are muxed (1.9) and the SPFR processor (1.11). ) is then digitized by a high-speed analog-to-digital converter (ADC) 1.10 (e.g., a high-speed ADC).

The digitized signal is finally processed by the SPFR processor 1.11, which stores samples of the waveforms in dedicated memory 1.12. Because of signal variability, the signature waveform may be acquired multiple times and filtered before final processing. Final processing involves calculating the object nominal distance, normalizing the reference and measured digitized signals, temporal tuning, and de-convolution. The result is a depth signature for a specific object.

For additional security, the SPFR processor 1.11 may also encrypt the depth signature data transmitted to the application processor 3.

The operation of the 2D camera 2 is as follows. The high-resolution imaging sensor (2.1) streams the captured video information and transmits it to the application processor (3) via 2D imager processing and control (2.4). In most cases, this element is part of a camera module already available in computing systems (smartphones, tablets, laptops, etc.). In other cases, this may be a dedicated imager, such as a complementary metal-oxide (CMOS) camera, which may be red green blue (RGB) and/or infrared (IR) sensitive.

The 2D camera 2 has an image stabilizing and You can have accurate shaking. The 2D camera system may also have a white and/or IR illuminator (2.6) and driver (2.7) used in low light conditions. For additional security, 2.4 encrypts the video data transmitted to the application processor 3.

The operation of the application processor 3 is to perform the following three functions:

ㆍControlling face authentication process and learning

· Collecting samples of the depth signature from the depth signature processor (1), and 2D snapshots from the 2D camera (2)

ㆍ Performing object identification or authentication using algorithms (e.g., machine learning algorithms)

Figures 4, 5, and 6 provide further details of the operation of software executing on the application processor 3 to assist in doing this.

Figure 4 illustrates the process undertaken by the facial recognition algorithm and application processor to learn a new object. First, in 4.2, the program operates to cause a smartphone or the like on which the system is deployed to display on-screen visual cues on its display to enable targeted facial recognition. Then, when the target user looks at the camera, in 4.4, facial data in the form of a 2D image is captured with the 2D camera. Software and facial recognition algorithms within the application processor then use known image recognition techniques to identify key features within the 2D image, such as eyebrows, nose, mouth, pupils, etc. For example, pupils can be easily determined from images by color binarizing the image to find small black circles. Next, at 4.8, the display of the smartphone or other device with which the system is integrated is updated to illustrate the target detection area and detection features matching the depth of detection signature sensor area. Prompts are then shown at 4.10 to the user to place their head in the required position by telling them whether to look straight at the screen, raise or lower their head, or move it from left to right. . Once the user is properly viewing the 2D sensor and depth signature sensor, in 4.12 2D image data and depth data are computed for the assumed head position. At 4.14, an evaluation is then undertaken as to whether all desired head positions required for data to be captured have been taken, and processing proceeds again to 4.10, where the user is prompted to assume a slightly different head position. This loop of operations 4.10, 4.12, and 4.14 is repeated until all desired head positions for the user have been captured.

At this point, therefore, the application processor has captured, through the 2D camera and the depth signature processor, both image data relating to the user's face as well as depth signature data relating to the user's face, and a plurality of both sets of captured data. The sets correspond to slightly different angles and positions of the head. This data can then be subjected to a parameter extraction process to determine the key parameters of the data and these parameters, once identified in 4.18, can then be encrypted and stored in a mathematical template in 4.20. If no valid parameters can be extracted, the process fails and a message is displayed to the user. Parameter extraction from captured time traces to generate user depth signature traces for use by the application processor for matching purposes is described in more detail later with respect to FIG. 14 .

Assuming that the learning process of Figure 4 has been undertaken and that data corresponding to the particular user detected is then stored within the application processor, Figure 5 shows a full user authentication process using the system of Figure 3 to identify or authenticate the user. Illustrate the process. The first part of Figure 5 is to determine whether virtually any facial data has been learned by the system, for example using the process of Figure 4. If no facial data or facial template is stored in the application processor, it is necessary to use an alternative authentication mode, such as a PIN code in 5.16, which is then determined to match in 5.18. However, if the application processor stores facial data for the user (e.g., in the form of 2D images and depth signature time traces), in 5.4 the processor stores a visual cue on its screen to enable targeted facial recognition. Control devices such as smartphones to display images. These prompts may be the same visual cues used in 4.2 of the learning process in Figure 4 to allow the user to correctly position his face with respect to the device. For example, the device may display on its screen an image of the user overlaid with fiducial marks, so that the user can then display the device so that certain parts of his face, eyes, nose, or ears are overlaid with the marks. should place his head against. In other examples, however, displaying of visual cues may not be performed, and instead the process continues with image data that the device may capture at any time.

If the target face is properly oriented with respect to the capture device, facial image data is then captured using the 2D camera in 5.6. This image data is then fed to an application processor, and a 2D face matching algorithm, which may be conventional, is then used to attempt to match the face. If the 2D face matching algorithm cannot return a positive result, i.e. cannot recognize the user's face, processing proceeds to 5.14, where an evaluation is made on the number of attempts the user attempts to perform recognition. If the maximum number of retries has been obtained, the alternative authentication mechanisms of 5.16 and 5.18 are then used and if these cannot authenticate the user, the authentication process fails. Conversely, if an alternative authentication method, such as a PIN code, can authenticate the user, the authentication process returns that the user has been authenticated.

Returning to 5.8, however, if the 2D image captured by the 2D camera matches the user's face, then it is necessary to determine whether the user's facial depth signature also matches. 5.10, therefore, the facial depth signature is then captured using the depth signature processor 1, and the resulting time-based waveform is then fed to the application processor and the depth signatures (or corresponding mathematical templates) stored therein. ) matches. If the application processor can match the captured facial depth signature with the stored depth signature for the user (above a certain matching threshold), then the user has been authenticated by both the 2D image capture and the depth signature processor and can therefore be authenticated as if they were real. . As previously discussed, the advantage of performing both 2D face matching using a 2D image from the camera, and face depth signature matching, is that it is then possible to fool the 2D image face matching process by simply displaying a photo of the user with the camera. Since this is not possible, greater security can be obtained. Instead, in order to authenticate a user, there must be a real user with a three-dimensional face, because depth signature information, captured simultaneously with the user's 2D image, must also be matched. Alternatively, a life-like mask of the user could also be used to attempt to spoof the system, but it will be appreciated that this is considerably more difficult to reproduce than just a photo.

The depth signature capture process 5.10 of Figure 5 is shown in more detail in Figure 6. In particular, Figure 6 illustrates the steps performed by the depth signature processor 1 when obtaining a depth signature signal trace to transmit to the application processor for recognition. The steps in Figure 6 are controlled by the SPFR processor 1.11, which controls other components of the depth signature processor as appropriate.

Turning to Figure 6, first at 6.2, the acquisition memory 1.12 is removed, and then the first step to be performed by the depth signature processor is to obtain baseline measurements of the response of the laser diode or VCSEL to its drive signal. will be. The reference measurement mode is selected in 6.4 and involves transmitting illumination pulses from the illumination driver (1.1) to a laser diode or VCSEL (1.2) and sampling using a high-speed ADC (1.10). Using a reference detector (1.7), the light generated by the laser diode or VCSEL is fed through a trans-impedance amplifier (1.8) to a high-speed ADC through a multiplexer (MUX) (1.9). That is, at 6.6, the illumination pulse is transmitted through the illumination driver (1.1) to the laser diode (1.2), and the resulting light pulse from the laser diode is, at 6.8, detected by the reference detector (1.7) and the high-speed ADC (1.10). is sampled through TIA (1.8) and MUX (1.9). In 6.10, the SPFR processor (1.1) then extracts useful samples from the information received from the ADC and updates the reference waveform memory. It then performs an evaluation as to whether the required number of reference waveforms have been obtained and whether additional reference waveforms are required, i.e. the number of reference waveforms (R _MAX ) has not been met and then the processing goes back to 6.6 and further A reference waveform is acquired. This process repeats until the desired number of reference waveforms (R _MAX ) are obtained and stored in the reference waveform memory.

Regarding the need for reference waveforms, optical signal distortion may occur due to hardware limitations when transmitting optical pulses, and if such distortion occurs, it can help to normalize it. For example, a reference waveform is shown in Figure 8. Here, the reference waveform 82 shows that the response of the laser diode or VCSEL to its drive pulses is non-linear, exhibiting large ringing. This is due to the current di/dt changing rapidly with time within the laser diode or VCSEL due to drive pulses and circuit parasitics within the laser diode. Ringing in the response of a laser diode or VCSEL to its driving pulse is indicated by changes in the light output by the laser diode or VCSEL during the pulse duration, which is then later collected by the photodetector (1.5). It is necessary to record this ringing at the laser diode or VCSEL output as a reference signal so that it can be used to normalize the reflected optical signal. If this normalization were not performed, differences in light amplitude and pulse shape caused by system non-linearities could be mistaken for being caused by features of the illuminated object, e.g. the target user's face, and thus the target object. It would be nearly impossible to use the light reflected from the illumination pulse to recognize or characterize.

Returning to Figure 6, once the reference waveforms of the laser diode or VCSEL response have been collected, it is then possible to begin collecting reflected waveforms from the target user 4. Therefore, at 6.14 the signal measurement mode is selected by the SPFR processor, at 6.16 illumination pulses are sent from the illumination driver (1.1) to the laser diode or VCSEL (1.2) and the ADC starts sampling again. At this time, however, the ADC samples the signals received from the light detector 1.5 via the trans-impedance amplifier 1.6 and the MUX 1.9, and the sampled signals are therefore in response to illuminating light pulses from the subject's face. Shows a time trace of the output of photodetector 1.5 in response to reflected light. At 6.18, the SPFR processor collects waveforms from the photodetector (1.5) via TIA (1.6), MUX (.19), and ADC (1.10), and then at 6.20, extracts useful samples and stores them in signal waveform memory (1.12). ) is updated. The SPFR processor also calculates the minimum distance required for successful detection, measures time-of-flight information, and then performs an evaluation of whether the number of acquired signal waveforms is equal to the desired number (S _MAX ), and if not, more Control the depth signature processor circuit to repeat the process to acquire many samples. However, once the required number of signal waveforms have been acquired, the SPFR processor 1.11 controls the circuit to stop collecting the signal waveforms and then proceed to analyze the waveforms collected and stored in memory 1.12. This is done starting from 6.24.

More specifically, in 6.24 the SPFR processor calculates from stored waveforms whether the waveform contains sufficient characterization data, i.e., whether the target is close enough. In this regard, the distance to the user is calculated by measuring the time of flight, which is the time difference between the rising edge of the reference waveform and the rising edge of the received pulse. If it is determined that the stored waveforms indicate that the distance is out of range, at 6.28 a message is displayed to the target user indicating that he should get closer to the device, after which signal collection can be repeated. Conversely, if the reference waveforms indicate that the target is in range, at 6.30 and 6.32 the reference waveform and the received signal waveforms are normalized, and at 6.34 they are time-tuned. 6.36, the reference waveform is then de-convolved from the signal waveform, which results in removing the contribution of system non-linearity (or artifacts such as the previously described ringing response) of the laser diode or VCSEL from the received signal. has 12 illustrates example simulated waveforms, with the top trace 1202 of FIG. 12 illustrating example rectangular pulse responses for simulated surfaces with different depth differences as steps, while the bottom trace 1204 The same dates are shown but normalized to remove the effects of artifacts from the laser diode or VCSEL.

Processing then proceeds to 6.38 where verification checks are performed on the de-convolved data, and if the data appears to be valid, it is then encrypted in 6.42 and then sent to the application processor in 6.44 for matching to the user. do. Conversely, if the data cannot be verified, at 6.40 the application processor is notified that the data is invalid, and the matching process is not performed.

With the above process, the depth signature processor can therefore capture time trace waveform characteristics of the shape of the target user's face from which light pulses are reflected. To accommodate artifacts within the laser diode, these pulses are then matched to a reference waveform captured from the laser diode such that the shape of the waveforms depends almost solely on the shape of the object from which the light pulses reflect, rather than on errors introduced within the circuit. It is normalized for As such, the time-based waveform traces can then be used by the application processor as recognition signals for the target user.

To validate the concept, a test setup was built based on off-the-shelf product components and instrumentation. Figure 7 shows a block diagram for this setup.

As shown in Figure 7, LD pulse generation is provided by a digital signal generator 712 that generates pulses as short as 1 ns. This signal is applied to the LD board 710, which contains a laser diode and a driver operating at 850 nm or other near infrared (NIR) frequencies. The maximum LD peak current is set by a resistor and can be controlled by the voltage applied to the LD anode, and in our tests this voltage is set to 18V, corresponding to an LD peak current of 8A (~5W optical peak power). Because of the fast di/dt and the inevitable circuit parasitics, it was not possible to generate clean rectangular pulses, so the LD drive signal had large ringing as shown in Figure 8 and discussed above.

The light exiting the LD module 708/710 is collimated by lens 706, and the collimated beams are 50/50 separated by 704, with one beam acquired at CH1 of high-speed digital oscilloscope 714. It is directed to a high-speed photodiode DET08CL (716) to be used as a reference signal. The other beam from splitter 704 passes through a 20°diffuser 705 set to cover a portion of the target user's face located ˜30 cm from the emitter. The illumination light then reflects across the subject's face.

Light reflected from the object is detected by the high-speed APD 718 and the amplified signal from the high-speed APD 708 is acquired on CH2 of the high-speed digital oscilloscope 714. The scope acquires reference and measured waveforms through a PC program that processes the data by normalizing, time-tuning and de-convolving the acquired signal, and storing the results as well as the original waveform data in a database for post-processing. It is connected to GPIB on a laptop computer. Normalization, temporal tuning and de-convolution are as previously described for Figure 7. The processing performed to recognize the target user from the obtained waveform traces is then described below.

Figure 15 shows at a high level the process performed by the application processor 3 to recognize or characterize an object 4 from reflection time traces. At 15.2, time series data is received from the SPFR processor 1.11, and then at 15.4, the application processor combines previously stored time series signature data (or templates) for the claimed target user attempting to be authenticated, or alternatively, the time series data. It operates to compare time series data (or corresponding templates) with all target users for which signature traces are stored. Once a match is made to the stored time series data, in 15.6, the user subject to whom the match was made is authenticated. Also, when used with a 2D image sensor, such as a camera module, in 15.8, the application processor may also initiate 2D image recognition, and the processor may then (above the set threshold) authenticates the user. Once the user is authenticated, the application processor outputs a verification/identification signal to its host device. Receipt of this verification or identification signal by the host device may then cause the device to perform an operation, for example, to be unlocked to allow use of the device by a user.

As for the time trace matching that needs to be performed by the application processor, as mentioned, this is performed on a one-dimensional time trace signal captured by a PD, APD or SPAD. To better understand this, a mathematical model was developed to help understand the complexity of the signal response reflected from an object with variable depths. To simplify modeling, the object surface was partitioned into m×n elements, and the reflectance and scattering profiles of each element were assumed to be identical (R = 0.5, reflectance). Additionally, since the reflection amplitude difference was caused by the difference angle for each element, the distance difference caused by the difference angle of each element was considered. The simulation of illumination is then also considered with variable pulse-width and bandwidth from an ideal source or actual optical waveform measured in a test setup. The end result is the integration of the response from all elements with respect to amplitude and time. Exemplary idealized (non-bandwidth limited, picosecond pulse) results are shown in Figures 9-11, while bandwidth limited and normalized results from nanosecond pulses are shown in Figures 12 and 13.

Briefly referring first to the idealized simulations of Figures 9-11, although they do not represent actual responses, they illustrate the theoretical discrimination capabilities of the technique. Figure 9, for reference, illustrates the theoretical response of a flat surface. From the left plot, it will be appreciated that a flat response with an unbounded bandwidth detector and idealized picosecond illumination pulses gives a single reflection peak, as would be expected. Figure 10 then illustrates the idealized surface with a 5 cm step difference, from which it can be seen that a 5 cm depth difference results in two separate output pulses separated by 0.3 ns in the time trace. Figure 11 shows that a smaller depth difference of only 0.5 cm steps, but again in the idealized simulation, two peaks are detectable in the trace, representing steps. In practice, however, due to bandwidth limitations in PD, a 5 cm step in Figure 10 may be detectable, but a 0.5 cm step would normally exceed the resolution of PD, due to bandwidth limitations that cause peaks to merge.

However, although the above applies to idealized simulations, in actual use there is no need to detect the difference in respective depths as shown for the idealized picosecond pulse case. Instead, reflections from different spots with different depths merge to form one single waveform. This is shown in Figure 13, where a multi-faceted surface was simulated with a more feasible 5 ns pulse. As will be seen, from a multi-faceted surface, the different reflection traces effectively blend together into a single trace, which can then be used as the unique time trace for the object.

The next issue then arises as to how to characterize the time trace. In this regard, the issue is that the differences in the waveforms for different objects are small and will reflect most of the amplitude, which itself is not stable for a real system. Again, since the difference is very small, the requirements for the digitizer will be very high. However, a 3 GHz sampling rate should provide enough data points, and in a real scenario, the reflectance and scattering profiles of different spots will be different, which should provide more difference. Figure 14 illustrates a sampled and interpolated trace, where the trace in Figure 13 was sampled at 3 GHz (to provide 3 samples per ns). Exemplary sample points are therefore shown at 1404, 1406, 1408, 1410, 1412, 1414, and 1416, and the waveform is recreated for matching purposes in the application processor 3 by linear interpolation on a sample point by sample point basis. Of course, in other examples, different interpolation techniques may be used to recreate the waveform for matching, taking sample points as input.

Considering the above, and returning to Figure 15, as mentioned in 15.6, the application processor may add time traces (expressed as a series of samples, as described above) to the stored time traces for specific registered users. ) is matched. As to how this matching is performed, proof of concept systems have implemented machine learning algorithms for user identification, and in particular two special types of matching algorithms have been used:

1. ANN matching algorithm - maps time series data directly to input neurons, one or more hidden layers, and an output layer to provide Go/NoGo authentication or identification of individuals from the data set.

2. Partial least squares discriminant analysis (PLS-DA) algorithm - Before implementing regression, PCA is used to reduce the dimensionality of time series data. Provides output similar to the ANN example.

When tested on a sample size of 32 people, initial results show that the system can easily distinguish between real people vs. 2D images, with neural network matching having slightly lower PLS-DA matching, and ~~ for individuals in the dataset. It shows that a 99% recognition rate is achieved. Of course, in other examples, different matching algorithms may also be used, as will be known by the intended reader.

In summary, examples of the present disclosure improve the recognition rate of a concomitant 2D system by up to 100x, especially in challenging conditions (e.g., poor lighting, cosmetic changes, etc.), or single-pixel modules with the potential to be used as a standalone system. Provides a sensor system. Moreover, when used with a 2-D sensor, the addition of 3D depth sensing capabilities of a single-pixel sensor system makes the entire system more vulnerable to simple spoofing (e.g., sending a photo or video feed of a user object instead of a real person). by using ). Additionally, single-pixel sensor systems are much simpler and less expensive than other 3D imaging systems, such as structured light or time-of-flight scanners.

Various modifications may be made to the above examples to provide additional examples. In one further example, a plurality of single-pixel sensors may be provided and arranged in a one-dimensional line, or a two-dimensional array. The advantage of providing a plurality of these sensors is not that it then becomes possible to image an object using them, but that multiple measurements can then be obtained simultaneously from the same illumination pulse. This will reduce the number of calibration and measurement illumination pulses required for operation. Additionally, each single-pixel sensor of the plurality of sensors will have a slightly different FOV of the target user due to the slightly different ray paths the light will take from the lighting diode to the user and then back to the different sensors in the array. This will mean that each single-pixel sensor in the array will capture a slightly different one-dimensional time trace representing the light reflected back from its own FOV. The set of synchronous time traces so obtained for each separate illumination flash provides additional signature characteristics of the target user or object, which set can be used to enhance discrimination or recognition of the object.

In a further example, if the lighting diode or VCSEL has a clean and predictable response to its drive signal (i.e., does not exhibit ringing or other unpredictable artifacts), it then includes a reference channel component, or It may not be necessary to perform the baseline normalization steps described in the examples. Excluding these features when possible will result in a simpler, and therefore less expensive, implementation.

In some examples, devices may include means for implementing/performing any one of the methods described herein.

It is also important to note that all specifications, dimensions, and relationships outlined herein (eg, number of processors, logical operations, etc.) are provided for purposes of illustration and teaching only. Such information may be changed substantially without departing from the spirit of the disclosure, or the scope of the appended claims (if any) or examples set forth herein. The specifications apply to non-limiting examples only and, therefore, they should be construed as such. In the foregoing description, example embodiments have been described with reference to specific processors and/or component arrangements. Various modifications and changes may be made to these embodiments without departing from the scope of the appended claims (if any) or the examples described herein. The description and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.

Note that, with the number of examples provided herein, the interaction may be described in terms of two, three, four, or more electrical components or parts. However, this was done for clarity and example purposes only. It should be understood that the system may be integrated in any suitable manner. Following similar design alternatives, any of the illustrated components, modules, blocks, and elements of the figures may be combined into a variety of possible configurations, all of which are expressly within the broad scope of this disclosure. . In certain cases, it may be easier to describe one or more of the functions of a given set of flows by referring to only a limited number of electrical elements. It should be understood that the electrical circuits of the drawings and their teachings are readily scalable and can accommodate multiple components, as well as more complex/sophisticated arrangements and configurations. Accordingly, the examples provided should not limit the scope or suppress the broad teachings of electrical circuits as potentially applicable to countless other architectures.

In this specification, “one embodiment”, “exemplary embodiment”, “an embodiment”, “another embodiment”, “some embodiments”, “various embodiments”, “other embodiments”, References to various features (e.g., elements, structures, modules, components, steps, operations, features, etc.) included in an “alternative embodiment” or the like refer to any such features being disclosed herein. is included in one or more embodiments of, but is intended to mean that they are or are not necessarily combined in the same embodiments. It is also important to note that the functions described herein illustrate only some of the possible functions that can be implemented by or within the systems/circuits illustrated in the figures. Some of these operations may be deleted or eliminated as appropriate, or these operations may be significantly modified or altered without departing from the scope of the present disclosure. Additionally, the timing of these operations can vary significantly. The previous operational flows were provided for example and discussion purposes. Considerable flexibility is provided by the embodiments described herein in that any suitable arrangements, chronological sequences, configurations, and timing mechanisms may be provided without departing from the teachings of the present disclosure. Numerous other changes, substitutions, variations, modifications, and modifications may occur to those skilled in the art and the present disclosure does not fall within the scope of the appended claims (if any) or the examples set forth herein. It is intended to include all such changes, substitutions, variations, modifications, and modifications. Note that any optional features of the device described above may also be implemented with respect to the method or process described herein and details in the examples may be used elsewhere in one or more embodiments.

Various further modifications, whether by addition, deletion, or substitution, may be made to the above-mentioned examples to provide further examples, any and all of which are intended to be encompassed by the appended claims. .

The claims herein have been presented in a single dependent format suitable for filing at the USPTO, but for purposes of those jurisdictions permitting multiple dependent claims, each claim shall be synonymous with any prior claim of the same type unless expressly technically impracticable. It will be understood that reliance may be placed on the claims.

Claims (20)

In a sensor for detecting an object,
a light source arranged to emit light pulses to illuminate the face of the object, the light pulses having a duration of less than 5 nanoseconds;
a diffuser that spreads the light pulse across the face of the object;
a light detector arranged to detect light from the light pulse reflected from the object's face to generate a one-dimensional time-based reflection signal representative of attributes of the object's face;
a recognition processor arranged to receive a time-based reflection signature and recognize the object according to the time-based reflection signature, wherein the recognition processor is configured to: receive an object-specific reflection signature compared to the time-based reflection signature to recognize the object; stores 3D time-based signal trace data; and
a processor arranged to receive signals representative of the one-dimensional time-based reflection signal and generate a time-based reflection signature for the object in accordance with the signals,
The object-specific one-dimensional time-based signal trace data includes each of a set of samples of each object-specific time-based signal trace to be captured during a training phase, and the recognition processor determines the object-specific time-based signal trace to be matched. A sensor for detecting an object, interpolating between the set of samples to reproduce underlying signal traces. In claim 1,
wherein the recognition processor uses machine learning techniques to recognize the object based on the time-based reflection signature. In claim 2,
The above machine learning techniques are:
a) principal component analysis; and
b) one or more neural networks
A sensor for detecting an object, comprising at least one of: In claim 1,
further comprising a reference channel arrangement arranged to receive at least a portion of the optical pulse directly from the light source to provide a reference signal, wherein the processor further operates according to the reference signal to take into account undesirable characteristics in the optical pulse. A sensor for detecting an object, arranged to normalize the one-dimensional time-based reflection signal. In claim 4,
The light source has a ringing characteristic, the reference channel component includes a reference photodetector arranged to detect the ringing characteristic, and the processor further provides a reference photodetector configured to detect the ringing characteristic. A sensor for detecting an object, arranged to receive a signal and normalize the one-dimensional time-based reflected signal to remove artifacts caused in the signal by the ringing. In claim 1,
further comprising a two-dimensional space-based recognition system arranged to capture a two-dimensional image of the face of the object and undertake recognition of the object according to the two-dimensional image, wherein the recognition processor is configured to: and generating an output signal indicative of successful recognition of the object upon recognition of the object using both the one-dimensional time-based reflection signals. In claim 1,
A sensor for detecting an object, wherein the object is a human subject, and the face of the object is the face of the human subject. In claim 1,
wherein the recognition processor also stores a mathematical template that is compared to a template corresponding to the time-based reflection signature to recognize the object. In a sensor for detecting an object,
a light source arranged to emit light pulses to illuminate the face of the object, the light pulses having a duration of less than 5 nanoseconds;
a diffuser that spreads the light pulse across the face of the object;
a light detector arranged to detect light from the light pulse reflected from the object's face to generate a one-dimensional time-based reflection signal representative of attributes of the object's face;
a processor arranged to receive signals representative of the one-dimensional time-based reflection signal and generate a time-based reflection signature for the object in accordance with the signals; and
a reference channel arrangement arranged to receive at least a portion of the optical pulse directly from the light source to provide a reference signal, wherein the processor further provides a reference signal for taking into account undesirable characteristics in the optical pulse; arranged to normalize the dimensional time-based reflected signal, including:
The light source has a ringing characteristic, the reference channel component includes a reference photodetector arranged to detect the ringing characteristic, and the processor further receives a signal generated by the reference photodetector in response to the ringing characteristic. and normalizing the one-dimensional time-based reflected signal to remove artifacts caused in the signal by the ringing. In claim 9,
A sensor for detecting an object, further comprising a recognition processor arranged to receive the time-based reflection signature and recognize the object according to the time-based reflection signature. In claim 10,
wherein the recognition processor uses machine learning techniques to recognize the object based on the time-based reflection signature. In claim 11,
The above machine learning techniques are:
a) principal component analysis; and
b) one or more neural networks
A sensor for detecting an object, comprising at least one of: In claim 10,
further comprising a two-dimensional space-based recognition system arranged to capture a two-dimensional image of the face of the object and undertake recognition of the object according to the two-dimensional image, wherein the recognition processor is configured to: and generating an output signal indicative of successful recognition of the object upon recognition of the object using both the one-dimensional time-based reflection signals. In claim 10,
wherein the recognition processor stores object-specific one-dimensional time-based signal trace data or a corresponding mathematical template that is compared to the time-based reflection signature or a corresponding template to recognize the object. sensor for. In claim 14,
The object-specific one-dimensional time-based signal trace data includes each set of samples of each object-specific time-based signal trace captured during a training phase, and the recognition processor determines the object-specific time-based signal trace to be matched. A sensor for detecting an object, interpolating between the set of samples to reproduce underlying signal traces. In claim 9,
A sensor for detecting an object, wherein the object is a human subject, and the face of the object is the face of the human subject. delete delete delete delete