WO2023125329A1

WO2023125329A1 - Living-body fingerprint identification system and living-body identification method thereof

Info

Publication number: WO2023125329A1
Application number: PCT/CN2022/141653
Authority: WO
Inventors: 李丽; 武振华; 王宇; 黄志雷
Original assignee: 北京与光科技有限公司
Priority date: 2021-12-31
Filing date: 2022-12-23
Publication date: 2023-07-06
Also published as: CN116380241A; CN116386093A

Abstract

The present application relates to a living-body fingerprint recognition system, a living-body recognition method thereof, and spectroscope thereof; the living-body fingerprint recognition system comprises: a screen, which is provided with a plurality of slits and which is used for receiving incident light generated after a light source projects light to a finger to be detected for reflection, and modulating said incident light; a photosensitive module, said photosensitive module being located at the lower end of the screen and comprising: an image sensor, used for receiving modulated incident light so as to obtain spectral information of the incident light, and processing said spectral information to obtain living-body information and image information of the fingerprint.

Description

Living body fingerprint identification system and living body identification method thereof

technical field

The present application relates to the technical field of spectral imaging, and more specifically, to a living body fingerprint identification system and a living body identification method thereof.

Background technique

Various types of biometric systems are increasingly used to provide greater security and/or enhanced user convenience. For example, fingerprint sensing systems have been widely used in various terminal devices, such as consumer smartphones, due to their small size, high performance, and high user acceptance. At present, there are many kinds of fingerprint sensing systems in the market, such as sensing systems based on capacitive fingerprint modules, sensing systems based on optical fingerprint modules, etc. Although the above-mentioned types of fingerprint sensing systems can realize unlocking, they are still being used After the fingerprint identification of the mobile terminal is unlocked, criminals can steal the user's fingerprint to make a fake fingerprint to crack the user's security system, which instead increases the probability of the mobile terminal's fingerprint password being discovered, and has caused a great impact on the information security of the mobile terminal. threat.

Therefore, there is a need for security hardening of traditional biometric systems, for example, by liveness detection to protect biometric systems from attacks that exploit spoofed body parts, such as spoofed fingerprints. Many solutions for liveness detection exist, for example, hardware-based methods to find material properties, pulse detection by oximetry, software-based methods to look for spurious artifacts in acquired fingerprint images to spoof and view fine-scale textures Methods.

However, the existing living fingerprint identification schemes have certain defects, such as too complex structure, complex identification operation or high fingerprint cost, etc. Therefore, a simple and reliable fingerprint identification scheme is urgently needed to realize living fingerprint identification.

Contents of the invention

An advantage of the present application is that it provides a living body fingerprint identification system and its living body identification method, wherein the living body fingerprint identification system can use the modulation effect of diffraction and/or interference brought about by the slit structure of the screen to obtain spectrum Information, through the spectrum information to realize the identification of living body.

According to one aspect of the present application, a living fingerprint identification system is provided, which includes:

The screen has a plurality of slit units arranged periodically, used to receive the incident light generated when the light projected by the light source is reflected on the finger to be tested, and modulate the incident light, and the slit units have a corresponding transmission spectrum curve;

A photosensitive module, the photosensitive module is located at the lower end of the screen, and includes:

The image sensor is configured to receive the modulated incident light, obtain spectral information of the incident light, and process the spectral information.

In the living fingerprint identification system according to the present application, each slit unit includes at least one slit and/or small hole.

In the living fingerprint identification system according to the present application, the slits of the plurality of slit units are arranged in the same manner.

In the living fingerprint identification system according to the present application, the shape and/or structure and/or size of the multiple slit units are consistent.

In the living fingerprint identification system according to the present application, any one of the slit units and its two adjacent slit units define an area whose sum of two vectors is equal to the dot product of the two vectors, and the area of the area is After the pattern is translated by an integer number of displacements of the vectors along the vector directions corresponding to the two vectors within the range of the periodic area, the slits in this area coincide with the slits in the area after the translation, wherein, The periodic area is an area formed by a plurality of slit units arranged in a periodic manner.

In the living fingerprint identification system according to the present application, a plurality of the slit units are distributed in the whole range of the screen.

In the living fingerprint identification system according to the present application, a plurality of the slit units are distributed in a local area of the screen.

In the living fingerprint identification system according to the present application, a plurality of the slit units are distributed within a testing area of the screen, and the testing area corresponds to the photosensitive module.

In the living fingerprint identification system according to the present application, the screen includes a glass cover and a light emitting unit located at the lower end of the glass cover.

In the living fingerprint identification system according to the present application, the photosensitive module includes an optical assembly, the optical assembly includes an aperture and at least one lens, and the optical assembly is located on the photosensitive path of the image sensor.

In the living fingerprint identification system according to the present application, the photosensitive module includes a light filtering structure, and the light filtering structure is located on the photosensitive path of the image sensor.

In the living fingerprint identification system according to the present application, the image sensor includes black and white pixels.

According to another aspect of the present application, the present application provides a living body identification method, which includes:

determining spectral pixels and non-spectral pixels of an image sensor;

Cast mixed light to the object being shot;

Obtaining image data of the object being shot through the non-spectral pixels, and obtaining spectral data of the object being shot through the spectral pixels; and

performing liveness detection and object recognition based on the spectral data;

Among them, the spectral pixels and non-spectral pixels of the image sensor are determined, including:

Place a low refractive index material or a test piece with high reflectivity on the glass cover of the screen;

The first light-emitting unit of the control screen emits the first pre-standard light;

receiving first spectral response data corresponding to the first light-emitting unit through an image sensor;

Extracting the physical pixels in the image sensor corresponding to the response data exceeding the preset value in the first spectral response data, and corresponding to the response data exceeding the preset value in the first spectral response data The position of the physical pixel of is recorded as the first position;

controlling the second light emitting unit of the screen to emit second pre-standard light;

receiving second spectral response data corresponding to the second light emitting unit through the image sensor;

Extracting the physical pixels in the image sensor corresponding to the response data exceeding the preset value in the second spectral response data, and corresponding to the response data exceeding the preset value in the second spectral response data The position of the physical pixel of is recorded as the second position; and

Determining that the physical pixels overlapping the physical pixels at the first position and the physical pixels at the second position in the image sensor are non-spectral pixels, and determining the physical pixels other than the non-spectral pixels in the physical pixels at the first position The pixels are spectral pixels.

Projecting the first detection light to the subject;

receiving the first detection light reflected back by the subject and generating first spectral information and image information of the subject based on the first detection light;

projecting second detection light to the subject;

receiving the second detected light reflected back by the subject and generating second spectral information of the subject based on the second detected light; and

Live body detection and object recognition are performed based on the first spectral information, the image information, and the second spectral information.

In the living body identification method according to the present application, the first detection light is mixed light, and the second detection light is monochromatic light.

In the living body recognition method according to the present application, live body detection and object recognition are performed based on the first spectral information, the image information and the second spectral information, including:

processing the first spectral information and the second spectral information to generate a first spectral response result and a second spectral response result;

processing the image information to generate an image of the subject;

comparing the image of the subject with a pre-stored reference image; and

In response to successful matching between the image of the subject and the reference image, it is determined whether the subject is a living body based on the first spectral response result and/or the second spectral response result.

According to another aspect of the present application, a kind of spectrometer is provided, it comprises:

The substrate has a plurality of slit units arranged periodically for modulating incident light, and the slit units form a transmission spectrum curve corresponding thereto;

A photosensitive module, the photosensitive module is located at the lower end of the screen, and includes: an image sensor for receiving modulated incident light to obtain spectral information of the incident light, the substrate is arranged on the image sensor on the optical path.

In the spectrometer according to the present application, each slit unit includes at least one slit and/or small hole.

In the spectrometer according to the present application, the substrate is a screen.

In the spectrometer according to the present application, the screen includes a glass cover and a light emitting unit located under the glass cover.

In the spectrometer according to the present application, the spectrometer further includes a light source, and the light source is the light emitting unit.

In the spectrometer according to the present application, the photosensitive module further includes an optical assembly including an aperture and at least one lens, and the optical assembly is located on the photosensitive path of the image sensor.

In the spectrometer according to the present application, the substrate is a modulation cover.

In the spectrometer according to the present application, the modulation cover plate includes a glass cover plate made of a transparent material and an opaque material covering the glass cover plate, and the modulation cover plate is not covered with the opaque material. The slit unit is formed at the light-transmitting material.

In the spectrometer according to the present application, the opaque material includes opaque conductive materials arranged in parallel, and the conductive materials arranged in parallel form a capacitive structure.

In the spectrometer according to the present application, the light-impermeable material includes a light-impermeable non-conductive material.

In the spectrometer according to the present application, the spectrometer further includes a circuit board electrically connected to the image sensor, and the circuit board is adapted to be connected to the capacitive structure.

In the spectrometer according to the present application, the modulation mask is a mask.

In the spectrometer according to the present application, the modulation cover is a protective cover of an electronic device, the protective cover has a light-transmitting area and a non-light-transmitting area, and the light-transmitting area forms the slit unit.

In the spectrometer according to the present application, the photosensitive module includes a light filtering structure and an image sensor, and the light filtering structure is located on a photosensitive path of the image sensor.

In the spectrometer according to the present application, the spectrometer further includes a filter located on the photosensitive path of the image sensor.

In the spectrometer according to the present application, any one of the slit units and its adjacent two slit units define two vectors and an area whose area is equal to the area of the two vectors, and the pattern in this area is periodic After translating an integer number of displacements of the vectors along the vector directions corresponding to the two vectors within the range of the region, the slits in the region coincide with the slits in the region after the translation, wherein the period The area is an area formed by a plurality of slit units arranged periodically.

In the spectrometer according to the present application, the angle between the two vectors is 90°.

In the spectrometer according to the present application, the periodic region has at least 25 slit units.

Description of drawings

Various other advantages and benefits of the present application will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings in the description are for the purpose of illustrating preferred embodiments only and are not to be considered as limiting the application. Apparently, the drawings described below are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative efforts. Also throughout the drawings, the same reference numerals are used to denote the same parts.

Figure 1 illustrates a schematic diagram of reflection spectrum data corresponding to real fingers and fingerprint materials, taking silica gel and human skin tests as examples.

FIG. 2 illustrates a schematic diagram of a screen of a living fingerprint identification system according to the present invention.

FIG. 3A is a schematic diagram of spectral information directly acquired by an image sensor.

FIG. 3B is a schematic diagram of spectral information acquired by an image sensor after being modulated by an OLED screen according to the present invention.

FIG. 4A is a diagram of light intensity information corresponding to a part of the image sensor area where the incident light is in the 450nm band.

FIG. 4B is a light intensity information map corresponding to the same region of the image sensor when the incident light is in the 580nm band.

Fig. 5 illustrates a schematic diagram of an OLED screen distributed with R, G, and B light emitting units.

Fig. 6A is a schematic diagram of a first example of slits and/or apertures of an OLED screen according to the present invention.

Fig. 6B is a schematic diagram of a second example of slits and/or apertures of an OLED screen according to the present invention.

7A and 7B are schematic diagrams of the imaging light path of the OLED screen according to the present invention.

Fig. 8A illustrates a schematic diagram of a first modification example of the OLED screen according to the present invention.

Fig. 8B illustrates a schematic diagram of a second modification example of the OLED screen according to the present invention.

FIG. 8C illustrates a schematic diagram of a third modification example of the OLED screen according to the present invention.

Fig. 9 illustrates a schematic diagram of a variant embodiment of the living fingerprint identification system according to the present invention.

Fig. 10 illustrates an example of the operation of the filter of the living fingerprint identification system.

Fig. 11 illustrates a schematic diagram of pixel binning of the image sensor of the living fingerprint identification system according to the present invention.

Fig. 12 illustrates a flowchart of a first example of a living body recognition method according to the present invention.

Fig. 13 is a schematic diagram of the neural network model of the present invention.

Fig. 14 illustrates a flowchart of a second example of the living body recognition method according to the present invention.

FIG. 15 illustrates a flow chart of the liveness detection and object recognition steps in the method shown in FIG. 14 .

Fig. 16 illustrates a schematic diagram of a first example of a spectral pixel array of an image sensor according to the present invention.

Fig. 17 illustrates a schematic diagram of a second example of a spectral pixel array of an image sensor according to the present invention.

Fig. 18 illustrates a schematic diagram of the living fingerprint identification process according to the present invention.

Fig. 19 illustrates a schematic diagram of a region of interest according to the present invention.

Figure 20 illustrates a schematic diagram of a spectroscopic device according to an alternative embodiment of the present application.

FIG. 21 is a schematic diagram illustrating an example of the structure of a modulation cover plate of the spectrum device as shown in FIG. 20 .

FIG. 22 illustrates a schematic diagram of another example of the structure of the modulation cover plate of the spectrum device as shown in FIG. 20 .

FIG. 23 illustrates a schematic diagram of a configuration including optical components of the spectroscopic device shown in FIG. 20 .

Fig. 24 is a schematic view of the back of an existing mobile phone.

Fig. 25 is a schematic diagram of the back of the mobile phone with a protective cover in this embodiment.

Detailed ways

Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited by the exemplary embodiments described here.

Application overview

Due to the physiological characteristics such as capillaries (blood) and sweat pores in human skin, it is difficult to be forged compared with fingerprint lines, and due to the existence of physiological characteristics, the skin has different spectral absorption/reflection degrees for different bands, which also shows that , can judge the living body according to the spectral information reflected by the skin, so as to realize the living body detection of the fingerprint. Specifically, through reflection spectrum tests on real fingers and fingerprint materials, it can be known that at wavelengths of 300nm-1100nm, there is a huge difference between the reflection spectrum of real fingers and the reflection spectrum of fingerprint materials. Fig. 1 illustrates a schematic diagram of reflection spectrum data corresponding to real fingers and fingerprint materials, taking silica gel and human skin tests as examples. As shown in Figure 1, the difference between the two is large, so it is feasible to judge the living body through the received reflection spectrum.

exemplary system

Based on the above theory, the present invention provides a living fingerprint identification system. The identification system includes a screen and a photosensitive module. The photosensitive module is located at the lower end of the screen. The projected light is reflected by the finger to be tested to generate incident light, which will generate diffraction and/or interference when passing through the slit of the screen, and then be received by the photosensitive module to obtain spectral information. The spectral information received by the sensor is processed to obtain the living information and image information of the fingerprint, so as to realize the living identification and image recognition of fingerprint lines. Wherein, the screen may be implemented as an LCD screen, an OLED screen, a microLED screen, a ULED screen, and the like. Generally speaking, the under-screen fingerprint recognition system of traditional technology needs algorithm correction to avoid screen interference, including but not limited to diffraction and/or interference phenomena, so as not to cause the image information received by the photosensitive module to fail to realize fingerprint recognition. The screen, especially the corresponding slit, the photosensitive module, and the corresponding relationship between the screen and the module will be designed and adjusted to suppress diffraction and/or interference as much as possible, and the present invention can utilize the diffraction and/or interference caused by the slit structure of the screen. The modulation effect of interference is equivalent to the spatial dispersion modulation of the incident light, and then the photosensitive module receives the modulated fingerprint image information, and then obtains the spectral information, and realizes the living body judgment and recognition by processing the spectral information. FIG. 2 illustrates a schematic diagram of a screen of a living fingerprint identification system according to the present invention.

Specifically, the present invention takes the implementation of the screen as an OLED screen as an example, that is, the screen can be used for display and as a light source to project light to the object (finger) to be measured, and because the OLED screen has a slit that can be used to In order to diffract and/or interfere the incident light generated by the reflection of the object to be tested, so as to realize the spatial dispersion modulation of the incident light; the photosensitive module includes an image sensor, and the image sensor can be implemented as an imaging chip such as a CMOS chip or a CCD chip , and preferably the physical pixels on the image sensor are preferably black and white pixels (that is, no Bayer array); the incident light is modulated by the OLED screen and then received by the image sensor to obtain image information, and then obtain the spectrum information; the living body can be judged by processing the spectral information. It should be understood that the information received by the image sensor includes both information that can be used for fingerprint imaging and spectral information for living body judgment.

Further, in order to highlight the role of the slit of the OLED screen in the present invention, FIG. 3A and FIG. 3B are provided. Wherein, FIG. 3A is a schematic diagram of spectral information directly acquired by an image sensor. As shown in the figure, the light projected by the light source is reflected to generate incident light after reaching the finger to be tested, and the incident light is directly received by the image sensor. The spectral information on the image sensor, in which the spectral information of each area of the image sensor is relatively uniform; that is, it is understood as incident light Not modulated by the OLED screen. Fig. 3B is a schematic diagram of spectral information acquired by the image sensor after being modulated by the OLED screen of the present invention. That is, the spectral information of the same incident light modulated by the OLED screen. Through comparison, it will be found that the information displayed by the two is completely different. Relatively speaking, after being modulated by the OLED screen, the image information received by the image sensor will contain more information. Specifically, after the incident light is adjusted by the spatial dispersion of the screen, the dispersion characteristics of the light, that is, the spectral characteristics (spectral information), will be included in the image information. Therefore, spectral information can be extracted from it, and then used to realize living body judgment.

In order to further illustrate the advantages brought by the slit, a test chart is provided. After incident light of different wavelengths passes through the OLED screen, different patterns will appear on the image sensor, that is, the modulation effect of the OLED screen on different wavelength bands of incident light is different. Figure 4A is the incident light in the 450nm band, corresponding to the light intensity information map of some areas of the image sensor, and Figure 4B is the incident light in the 580nm band, corresponding to the light intensity information map of the same area of the image sensor, the difference between the two can be clearly seen.

Specifically, the present invention is based on the fact that the OLED screen can achieve diffraction and/or interference. Preferably, the OLED screen of the present invention can produce interference effects as much as possible, so the design of the screen or slit needs to be considered. It should be understood that the OLED screen will follow the R, G, and B light-emitting units are regularly distributed according to demand. For example, as shown in Figure 5, conventionally, R, G, G, and B light-emitting units are arranged in an array, and the slits are formed between the light-emitting units. Between, a group of RGGB light-emitting units will form multiple slits (the white box in the figure can be understood as a group of light-emitting units), and multiple slits are defined as a group of slit units, then the slit unit It needs to be arranged in a fixed period, that is, the distance (period) between adjacent slit units is equal (only need to ensure that the screen area participating in the modulation of incident light has this characteristic), which can ensure that the interference effect is as obvious as possible, so that The image information received by the image sensor contains spectral characteristics. It should be understood that the present invention does not limit that the screens must be arranged in an RGGB array, and the screens can be arranged in other ways. At this time, the slit units can also be adjusted accordingly. A cell containing at least one slit is defined as a slit cell. Preferably, the slits of the plurality of slit units are arranged in the same way. It should be understood that the different slit units do not need to be exactly the same, but the difference should not be too large, so as to avoid no interference effect. Here, FIG. 5 illustrates a schematic diagram of an OLED screen distributed with R, G, and B light emitting units.

For a better understanding, the slit unit is further explained. The OLED screen has a light-emitting unit of the pixel layer (light-emitting layer) and a TFT structure of the circuit layer and a reflective layer (generally removed in the under-screen module solution), which will cause incident Light cannot pass through, but there are slits and/or small holes between pixels (light-emitting units) and between TFT structures, allowing incident light to pass through. These light-transmitting slits and small holes are periodic in a certain range, for example, they are periodic in the entire screen, or the test area corresponding to the photosensitive module is periodic, and of course they can be other areas. That is, at least one slit and/or small hole constitutes a slit unit, and any slit unit can define a vector a and a vector b with its two adjacent slit units, that is, vector a, vector b and The area of a is equal to the area of point a multiplied by b (the area of the parallelogram formed by vector a and vector b). The pattern in this area is within the scope of the periodic area, and after the displacement of an integer number of vectors a and an integer number of vectors b is translated along the corresponding vector direction, the slits and/or small holes will basically overlap. Any one of the slit units and its adjacent two slit units define two vectors and a region whose area is equal to the area of the two vectors, and the pattern of this region is respectively along the two described vectors within the periodic region. After the vector direction corresponding to the vector is translated by an integer number of displacements of the vector, the slits in the area coincide with the slits in the area after the translation, wherein the periodic area is composed of a plurality of periodic arrays The area formed by the slit unit. The periodic area has at least 25 slit units. Generally, the angle between vector a and vector b is 90 degrees. Specifically, as shown in Figure 6A and Figure 6B, they are two different OLED screen slits and/or small holes, the bright area is the OLED slit and/or small hole, and the rectangular area framed is the slit unit, Of course, different screens correspond to different slit unit areas, which are also limited to rectangular areas. It should be noted that the slits between the slit units in the present invention may be different from each other, but the shape, structure and size of the slit units are basically the same, that is, preferably, a plurality of the slits The cells are uniform in shape and/or structure and/or size. However, due to certain errors in processing, there may be certain differences among the slit units, which can also be understood as being consistent with the concept of the present invention and covered by the present invention. Here, FIG. 6A is a schematic diagram of a first example of a slit and/or a small hole of an OLED screen according to the present invention, and FIG. 6B is a schematic diagram of a second example of a slit and/or small hole of an OLED screen according to the present invention. .

Further, the intensity signal of the incident light at different wavelengths λ is denoted as x(λ), and the transmission spectrum curve formed by the slit unit of the OLED screen is denoted as T(λ), which can be denoted as Ti(λ) (i=1,2 ,3,...,m); then at least some of the physical pixels of the image sensor acquire spectral information bi modulated by the OLED screen; then

bi=∫x(λ)*Ti(λ)*R(λ)dλ

Among them, R(λ) is the response of the image sensor;

Specifically, the light intensity information received by all the physical pixels of the image sensor will contain image information and spectral information, and the spectral information can be processed to judge the living body, while the corresponding image information is used for imaging.

It should be noted that, in order for the image sensor to obtain spectral information as fully as possible, there should be at least two transmission spectrum curve correlations between the transmission spectrum curves Ti(λ) (i=1,2,3,...,m) degree is less than or equal to 0.4, and the degree of correlation can be defined by Pearson correlation coefficient (Pearson correlation coefficient). Further, to define the transmission spectrum curve of the present invention, it should be understood that the existence of the transmission spectrum curve is mainly due to the presence of slits in the OLED screen, and the incident light will be modulated when passing through the slits, and the transmission spectrum curve can be considered as determining the modulation of the incident light. Effect, so multiple slits constitute a slit unit will have a corresponding transmission spectrum curve, but the preferred transmission spectrum curve of the present invention is not determined by a single slit unit, it may be affected by the surrounding slit units, that is, the preferred transmission spectrum curve of the present invention The transmission spectrum curve is determined by at least two slit units. Further, the number of the transmission spectrum curves is equal to the number of effective light intensity bi obtained, and the effective light intensity bi refers to the light intensity information used for spectrum recovery or spectral response judgment, and its number n is equal to the number of transmission spectrum curves; generally In the application, the incident light will be discretely and uniformly sampled, and there are n sampling points in total, such as the 200-400nm band, and the spectral resolution is 1nm, then the sampling point is 201; at this time, the transmission spectrum matrix formed by the transmission spectrum curve is A matrix of n*m.

Further, a detailed example is provided for ease of understanding, as shown in FIG. 7, assuming that the thickness of the glass cover plate of the OLED screen is A and the refractive index is n, the photosensitive module has a beam shrinking system (lens group), which shrinks The beam ratio is N:1; the pixel size of the image plane of the image sensor of the photosensitive module is P, the structure size of the LED array is D, the distance from the array to the diaphragm is L, and the field of view angle of the pixel points of the image plane is K.

It should be noted that the thickness A of the cover plate, the angle of view of the pixel point K, the distance L from the optical component (lens group) to the slit unit, and the beam reduction ratio N of the lens group are mutually coupled, and will be affected by the beam reduction system (lens group) ) parameter influence. All the following discussions are based on the case where the paraxial approximation holds.

According to the shrinkage ratio, the divergence angle of the reflected or transmitted light corresponding to the object to be measured is K/N.

The spot diameter of the incident light at the slit unit is d=(K/N)*A/n.

The period of the slit unit covered by the light spot is d/D, and it is speculated that this value needs to reach at least 2-5.

When the pixel moves by one grid, the incident light at the slit unit moves horizontally by N*P.

The chief ray angle deflects m=N*P/(L+A/n), and the position of the light spot moves v=m*L at the LED array.

According to the interference theory, the interference fringe period should be c=2λ/D.

In order to make the spectrum difference between pixels, it is necessary to satisfy that m is not much smaller than c. Presumably m should be greater than c/20, preferably greater than c/12.

v is generally not much smaller than D, such as v>D/6.

Further, the under-screen living fingerprint recognition system also needs to pay attention to issues such as stray light and improvement of fingerprint image resolution. Therefore, the general under-screen living fingerprint recognition system also includes a filter, which filters the incident light so that Light in a wavelength band can enter or can be cut off; for example, the filter can block light with a wavelength above 600nm. The filter can be located between the screen and the photosensitive module, or can be arranged on the photosensitive module. Under certain circumstances, the filter can isolate stray light in ambient light and improve the resolution of fingerprint images.

The photosensitive module can also include an optical assembly, the optical assembly includes at least one lens, and the optical assembly is located on the photosensitive path of the image sensor; the optical assembly can further include an aperture, and the aperture It is used to limit the angle of incident light so as to prevent stray light from entering the photosensitive module, as shown in FIG. 8A . Fig. 8A illustrates a schematic diagram of a first modification example of the OLED screen according to the present invention.

Further, the OLED screen includes a glass cover and a light-emitting unit located at the lower end of the glass cover. During the identification process, the object to be tested needs to be placed on the glass cover, and the light-emitting unit projects a projected light to the object to be tested. The object, after being reflected by the object to be measured, generates incident light, and the incident light is modulated through the slit of the OLED screen, and then received by the image sensor to obtain image information modulated by spatial dispersion, and then obtain spectral information. However, it should be noted that part of the projected light A projected by the light-emitting unit will directly enter the slit to reach the image sensor; part of the projected light B will reach the glass cover and directly reflect into the slit, and then be received by the image sensor; After reaching the object (finger) to be measured, it is reflected into the slit and received by the image sensor; and part of the projected light D is absorbed by the object (finger) to be measured. As for the living body recognition, its original intention is that due to the existence of capillaries and sweat glands, fingers have different absorption of light in different wavelength bands, so that under the same light source, the reflected light passing through the finger is inconsistent, which is different from that of conventional silica gel and pseudo-finger to projected light. There is a difference in absorption, and living body judgment can be judged based on the difference. Therefore, the really useful projected light should be projected light C and projected light D, while projected light A, projected light B and ambient light are stray light to a certain extent, as shown in FIG. 8B . Fig. 8B illustrates a schematic diagram of a second modification example of the OLED screen according to the present invention.

Therefore, it is possible to cover the black light-absorbing member in a dark room environment or in the test area, so that the light-emitting unit projects the same projected light. At this time, the incident light received by the image sensor can basically be regarded as the projected light A and projected light B after passing through the slit received by the image sensor. The collected spectral information in this case is recorded as the reference spectral information, and the stray light brought by the projection light A and projection light B can be removed by subtracting the reference spectral information from the spectral information obtained by the subsequent test of the object to be measured, as shown in the figure 8C. FIG. 8C illustrates a schematic diagram of a third modification example of the OLED screen according to the present invention.

Variant embodiment

The difference from the above embodiments is that the photosensitive module includes a filter structure and an image sensor, the filter structure is located on the photosensitive path of the image sensor, and the filter structure is a broadband filter in the frequency domain or wavelength domain. structure. The pass spectra of different wavelengths of the filter structures are not exactly the same. Filtering structures can be metasurfaces, photonic crystals, nanocolumns, multilayer films, dyes, quantum dots, MEMS (microelectromechanical systems), FP etalon (FP etalon), cavity layer (resonant cavity layer), waveguide layer (waveguide layer) layer), diffraction elements and other structures or materials with filter properties. For example, in the embodiment of the present application, the light filtering structure may be the light modulation layer in Chinese patent CN201921223201.2. Further, the spectral device includes an optical system, the optical system is located on the photosensitive path of the image sensor, the light is adjusted by the optical system and then modulated by the filter structure, and then received by the image sensor to obtain a spectral response; wherein the The optical system may be an optical system such as a lens component and a uniform light component. The image sensor may be a CMOS image sensor (CIS), a CCD, an array photodetector, or the like. In addition, the spectrum device further includes a data processing unit, which may be a processing unit such as MCU, CPU, GPU, FPGA, NPU, ASIC, etc., which can export the data generated by the image sensor to the outside for processing.

It should be noted that the filter structure also has a transmission spectrum matrix A. At this time, the transmission spectrum matrix T corresponding to the entire living fingerprint identification system is always composed of the transmission spectrum matrix A of the filter structure, the transmission spectrum matrix T of the OLED screen, and the image The response of the sensor is determined. The incident light will respectively pass through the OLED screen and filter structure to reach the image sensor and be modulated. Fig. 9 illustrates a schematic diagram of a variant embodiment of the living fingerprint identification system according to the present invention.

Identification scheme

It should be noted that although in principle the OLED screen and the image sensor can form an optical system, that is, all physical pixels on the image sensor can obtain image information and spectral information, that is, after the OLED screen modulates the incident light, it is The image sensor receives and acquires the corresponding light intensity information, and the light intensity information can be used to restore the spectral curve and also can be used for imaging; preferably, the optical path of the image sensor also includes an optical component, and the optical component is implemented as a lens Group.

However, in the application of living fingerprint identification, it is necessary to extract physical pixels containing strong spectral information of living bodies or materials. Therefore, it is further necessary to determine the spectral pixels, that is, among all the physical pixels of the image sensor, according to the characteristics of the OLED screen, select the corresponding physical pixels and define them as spectral pixels. Spectral pixels can be understood as physical pixels that respond more significantly to a certain waveband. pixels. Specifically, this embodiment provides a method for determining spectral pixels (also called a calibration method), by selecting the first light-emitting unit to emit light, and placing a test piece with a low refractive index or a high reflectivity on the glass cover plate of the OLED screen. , such as black paper, black rubber (low-refractive-index material) or white paper (high-refractive-index material), the image sensor receives the first spectral response data of the first light-emitting unit modulated by the slit unit of the OLED screen, and responds to the second A spectral response data is extracted corresponding to the physical pixel with strong light intensity, and recorded as corresponding to the first position of the image sensor, for example, extracting n points; then projecting the second light-emitting unit, receiving and obtaining the second spectral response data, extracting The physical pixel with a strong spectral response is recorded as the second position corresponding to the image sensor, comparing the first position and the second position to remove the overlapped point on the image sensor from the first position, then the physical pixel at the remaining position is defined as For spectral pixels, it can be understood that the transmission spectrum matrix corresponding to the spectral pixels will have a higher transmittance for a specific band of light, that is, the modulation effect will be better. Further, other lights can be projected to further filter the spectral pixels.

For example, a fingerprint biometric system generally sets a filter, which will cut off the band above 600nm, that is, the fingerprint biometric system will perform fingerprint biometric recognition in the 400-600nm band, and further fingers are relatively more sensitive to blue light. , green light and blue light can be used for calibration, that is, the first light-emitting unit emits blue light, the second light-emitting unit emits green light, and the second light-emitting unit emits green light to remove the physical pixels, the remaining positions correspond to spectral pixels, which only have a strong transmittance for blue light, and this process can be understood as blue light and green light calibration. Fig. 10 illustrates an example of the operation of the filter of the living fingerprint identification system.

In the stage of live fingerprint recognition, it is only necessary to project mixed light, such as white light (R, G, and B light-emitting units emit light at the same time), and the mixed light is projected to the finger, and then absorbed and reflected by the finger, and the reflected light will enter the OLED. The slit elements of the screen are modulated and received by the image sensor. The image sensor can be understood to be divided into non-spectral pixels and spectral pixels, and the spectral pixels will obtain spectral information, and judge whether it is a living body based on the spectral information. It should be understood that although the OLED screen will interfere and/or diffract the incident light, the non-spectral pixels will still obtain strong texture information, which can be used for fingerprint imaging. In order to allow both imaging and living judgment, the spectral pixels are in the Image sensors account for 10-25% of the physical pixels. Then divide or subtract the value obtained by the selected spectral pixel from the average value of the surrounding 8 adjacent physical pixels, so as to extract the modulation difference of the spectral pixel (equivalent to the spectral feature), that is, convert the wide-spectrum information into narrow-spectrum information. The key contrast of material is the narrow spectral information of the spectral pixels utilized. Generally, the spectrum for distinguishing "living" materials exists in the blue light information (narrow-band information) of 400-500 nm or the green light information (narrow-band information) of 500-600 nm.

Correspondingly, the present application provides a living body recognition method, which includes: determining spectral pixels and non-spectral pixels of an image sensor; projecting mixed light to an object to be shot; acquiring image data of the object to be shot through the non-spectral pixels , acquiring spectral data of the object to be shot through the spectral pixels; and performing living body detection and object identification based on the spectral data.

In the process of determining the spectral pixels and non-spectral pixels of the image sensor, first, place a low-refractive-index material or a test piece with high reflectivity on the glass cover of the screen, and control the first light-emitting unit of the screen to emit the first pre-standard light (for example, blue light); then, receive the first spectral response data corresponding to the first light-emitting unit through the image sensor; then, respond to the response data in the image sensor that exceeds the preset value in the first spectral response data The corresponding physical pixel (that is, the first spectral response data corresponds to the physical pixel with stronger light intensity, that is, the physical pixel with stronger spectral response to the reflected light corresponding to the first light-emitting unit) is extracted, and combined with The position of the physical pixel corresponding to the response data exceeding the preset value in the first spectral response data is recorded as the first position; Next, control the second light emitting unit of the screen to emit the second pre-marked light (for example , green light); then, receive the second spectral response data corresponding to the second light-emitting unit through the image sensor; then, respond to the image sensor and the second spectral response data exceeding a preset value The physical pixel corresponding to the data (that is, the second spectral response data corresponds to the physical pixel with stronger light intensity, that is, the physical pixel with stronger spectral response to the reflected light corresponding to the second light-emitting unit) is extracted, and the The position of the physical pixel corresponding to the response data exceeding the preset value in the second spectral response data is recorded as the second position; subsequently, the physical pixel at the first position in the image sensor and the The physical pixels overlapping the physical pixels at the second position are non-spectral pixels, and the physical pixels other than the non-spectral pixels in the image sensor are determined to be spectral pixels, that is, the physical pixels at the first position in the image sensor are determined The physical pixels other than the physical pixels overlapping with the physical pixels at the second position are spectral pixels.

offset compensation scheme

Due to interference and/or diffraction in the slit of the screen in the present invention, the screen modulates the incident light, so that the image sensor can obtain spectral information and image information; generally, the screen and the image sensor are required to remain relatively stable, otherwise It will cause the parameters (or modulation effect) of the whole system to change, resulting in inaccurate test results. Further, a method for judging whether to shift is provided, including:

1. Pre-store the position information of the reference physical pixel. By placing white or black paper on the glass cover of the screen, the projected light (the light is consistent with the type of light used in use) can be mixed light, such as white light , can also be monochromatic light. At this time, the image sensor receives the light intensity information, and takes N points (the brightest spots) with the strongest light intensity, and N is greater than or equal to 2, for example, 100 points. Record the position information of the N points on the image sensor, for example, record the N points as a reference position array ai(x, y), i=1, 2, 3...N, such as N=100.

2. Judging whether it is offset, when the object to be measured is set on the screen, the image sensor obtains the corresponding light intensity information, and selects the position information of N physical pixels with the strongest light intensity on the image sensor. Record the position array bi(x, y) of the N brightest physical pixels, i=1, 2, 3...N, compare the position array with the reference position array stored in physical pixels, when more than 80% When the position information changes, it is judged that an offset occurs, and the offset self-calibration step is started.

3. Offset calibration scheme 1: compare the position array bi(x, y) with the reference position array ai(x, y). For example, the comparison method is subtraction, such as ci(x,y)=bi(x,y)-ai(x,y). Perform statistics on the array values of ci(x, y). If the translation position occurs, the average value X, Y can be a fixed value or a statistical quantity with a relatively high proportion. Apply X, Y to the selection points marked by the blue and green cursors. These newly compensated blue and green light-marked physical pixels replace the pre-stored physical pixels and are applied to subsequent living body recognition.

4. Offset calibration scheme 2: Prompt and guide the terminal user to re-calibrate the blue and green light.

Merger plan

In some schemes, physical pixels on the image sensor are binned in order to reduce the amount of calculation and improve the signal-to-noise ratio; however, for this scheme, if the physical pixels of the image sensor are binned, spectral information may not be extracted. Therefore, the present invention further provides an image sensor, the physical pixels of which receive and/or output light intensity information independently.

Preferably, the present invention also provides an image sensor, the image sensor is divided into a merged area and a non-merged area, and the physical pixels corresponding to the merged area are merged, which can be 2*2, 3*3 or n*m Pixel merging outputs light intensity information, while for non-merging areas, individual physical pixels are used to receive and/or output light intensity information. Preferably, the non-merging area is located in the middle area of the image sensor. Fig. 11 illustrates a schematic diagram of pixel binning of the image sensor of the living fingerprint identification system according to the present invention.

Further, it is also possible to determine the position of the physical pixel implemented as a spectral pixel function on the image sensor, and receive and/or output light intensity information independently for the physical pixel at this position, and merge the remaining physical pixels.

As shown in Figure 18, the whole process identification process:

First, data acquisition: acquire the reference fingerprint information of the predicted object, the reference map information and the information to be identified of the object to be detected; then, perform offset detection: based on the reference fingerprint information, the reference map information and the to-be-identified information information to determine whether there is an offset between the information to be identified and the reference map information; then, perform identification data optimization processing: in response to no offset between the information to be identified and the reference map information, the De-noising the reference fingerprint information and the information to be identified; optionally, normalizing the de-noising reference information and the de-noising information to be identified; the reference fingerprint information and the de-noising information can also be The information to be identified is decombined; then, based on the optimized benchmark information, the correlation degree of the optimized information to be identified and the preset threshold value, it is judged whether the object to be detected is a living body.

Data Acquisition and Offset Detection Process:

The identification system stores a base map information in advance, that is, the reference map information, and places a test piece with a low refractive index or a high reflectivity on the glass cover of the screen, such as black paper, black rubber (low refractive index material), and then Turn on the light source, such as the light of the OLED screen, part of the light emitted by the light source is absorbed by the test piece, and the part of the light that is not absorbed will enter the slit unit of the screen to be modulated, and then received by the image sensor to obtain a spectrum Response data, record the spectral response data as the base map information. Wherein, the light emitted by the light source needs to be consistent with the projected light in the actual recognition work, that is, the waveband of the light emitted by the light source when acquiring the base map is basically the same as the waveband of the projected light in the actual recognition. For example, in the example, white light and/or blue light are used in the work to realize fingerprint recognition, then white light and/or blue light should also be used in the process of acquiring base map information. What needs to be understood is that in the actual fingerprint identification, the base map information has been burned into the identification system.

Furthermore, if the light source needs to project different types of light twice during the recognition process, then at least two base images should be required, which are obtained under different types of light respectively. For example, two projections are used for live fingerprint identification, and white light and monochromatic light, such as blue light, are projected. During the acquisition process of the base map information, the light source should also project white light and corresponding blue light, and record different types of light. The spectral response data are recorded as white light base map information and blue light base map information.

It should be noted that the data volume of the base map information can be substantially equal to the number of physical pixels of the image sensor; a specific area can also be selected on the image sensor as the base map information, for example, the middle area of the image sensor can be selected as the base Image information, this method can be called ROI (region of interest) area, in some embodiments, the ROI area can be selected as the edge area of the image sensor, for example, as shown in Figure 19, located in the four corners of the image sensor. It is also possible to independently extract physical pixels at different positions to form base map information based on requirements.

During the use process, the user first needs to enter the reference information (reference fingerprint information) of the object to be detected, such as the texture of the fingerprint, the spectral response information corresponding to the fingerprint, etc., as the reference information. In order to be more accurate, it needs to be entered at least three times, or at least three. Benchmark infographic. For example, in the case of a group of valid entries (for example, a group of valid entries is 10 entries), the correlation coefficient R is calculated between the spectral characteristic parameters of each time and the other 9 times, and the lowest correlation coefficient R_min is taken, and the system setting The parameter k is calculated with a specific formula to obtain the judgment threshold R_t for this input comparison. In actual use, the correlation coefficients of the parameters to be tested and the 10 input data are calculated respectively, and compared with the corresponding judgment threshold R_t (1-10). When 9 or more parameters are greater than the corresponding judgment threshold, the entry is successful.

Compare the reference information with the base map information to detect whether there is an offset. If there is no offset, the reference information will be burned into the recognition system; if there is an offset, it needs to be corrected, and optionally the corrected The information is undergoing offset detection, and then the reference information is burned into the recognition system; in some embodiments, if an offset occurs, the base map information needs to be collected again.

Furthermore, during the unlocking process, the user places the object to be detected, such as a finger or palm, on the screen, the light source projects corresponding light, and the image sensor receives the light modulated by the slit unit of the screen. The light reflected by the object to be detected obtains the corresponding information to be identified, for example, the texture of the fingerprint of the object to be detected, and the spectral response information corresponding to the fingerprint.

Comparing the information to be identified with the base map information to determine whether there is an offset.

For the identification and detection of offsets based on the base map information for the reference information and the information to be identified, RMSE (Root Mean Squared Error) can be used for evaluation. For example, the reference information a _nm and the information to be identified b _nm and base map information c _nm matrices are used to calculate the root mean square error; specifically, first calculate the benchmark RMSE value of the input information (benchmark information or information to be identified) and the base map information; and then make the input information and the base map Relative offset, calculate the offset RMSE value, if the base RMSE value is less than the offset RMSE value, it is judged that no offset has occurred, if the base RMSE value is greater than or equal to a certain offset RMSE value, it is determined that an offset has occurred. That is, in the process of judging whether there is an offset between the information to be identified and the reference map information based on the reference fingerprint information, the reference map information, and the information to be identified, first, the reference fingerprint information is calculated and the root mean square error value between the reference map information to obtain the reference root mean square error value; then, calculate the root mean square error value between the information to be identified and the reference map information to obtain the offset mean root mean square error value; in response to the reference root mean square error value being less than the offset root mean square error value, it is determined that there is no offset between the information to be identified and the reference map information; in response to the If the reference root mean square error value is greater than or equal to the offset root mean square error value, it is determined that an offset occurs between the information to be identified and the reference image information. The above-mentioned offset between the input information and the base map can be to shift the input information or the base map information up, down, left, and right, for example, offset one physical pixel in all four directions to obtain the offset entry information, and then calculate the RMSE value with the base map information.

The formula for calculating the RMSE value corresponding to benchmark information is as follows:

The formula for calculating the RMSE value corresponding to the information to be identified is as follows:

Identification data processing process:

The next step is to process the input information that has not been shifted after detection, so as to carry out living body recognition, and further compare the reference information and the information to be recognized with the base image information respectively, and perform debase (remove the background noise) to remove the background noise. For example, the benchmark information and the information to be identified can be converted into vectors or matrices, and subtracted from the corresponding base image information to obtain the denoised benchmark information and the information to be identified.

Further, optionally, normalize the reference information after denoising and the information to be identified, for example, divide each data of the corresponding reference information after denoising and the information to be identified by the maximum value of the corresponding information Numerical value; each data of the baseline information after denoising and the information to be identified can also be divided by the average of the corresponding information numerical values.

Secondly, de-binning (de-combining) is performed on the data, that is, the reference information and the information to be identified are extracted and processed in units of n*m, for example, taking 3*3 as an example, the intermediate physical pixel of 3*3 physical pixels The value of the pixel is subtracted from the average value of the surrounding 8 physical pixels, and then multiple data are obtained to construct a new matrix or vector to obtain the processed benchmark information and the processed information to be identified; on the one hand, de-binning can remove the data in the data. Noise can also reduce the amount of data and improve the efficiency of subsequent recognition. For example, in the case of 3*3, 9 data corresponding to the original 9 physical pixels can be changed to 1 data, and the amount of required calculation data can be reduced by 9 times.

It should be noted that the area that needs to be de-binned in this process can be the entire photosensitive area of the image sensor, or an artificially defined ROI area, or an area obtained by a calibration method. That is, this part of the area is used in the present invention to collect spectral information to determine whether it is a living body, so it is only necessary to perform de-binning on this area.

Then, the processed reference information obtained after the above processing and the processed information to be identified are subjected to living body judgment.

Identification comparison process:

Process all the reference information to obtain the corresponding processed reference information, and perform correlation calculation between the corresponding processed reference information and the processed information to be identified, for example, expressed by Pearson correlation coefficient, if the correlation R ² is greater than or equal to the predetermined A threshold of 0.7 is set as a living body. In some cases, the correlation degree can be required to be greater than or equal to 0.4, and in individual cases, the correlation degree can be required to be greater than or equal to 0.9; that is, the threshold corresponding to the correlation degree can be set artificially, that is, the manufacturer or user can identify it according to the system and Need to adjust.

In a modified embodiment, as the number of identifications increases, the correlation value of the recognition system in the case of successful living body identification in a continuous period of history is stored, and then the change trend is calculated, and the threshold value is adjusted according to the change trend. Value setting; for example, if the correlation value calculated by each identification gradually decreases, the value of the threshold should be appropriately increased, so as to ensure that non-living objects cannot crack the identification system.

Liveness identification method

Further, the present invention provides a living body identification method. The spectral information does not necessarily need to restore the spectral curve before performing living body judgment, but can directly perform living body judgment based on the spectral response. Specifically, the slit modulation based on the OLED screen is obtained. The spectral response of the final incident light on the image sensor is used to obtain the reference spectral response of the object to be measured; to a certain extent, the spectral response can be understood as the above de-binning data. comparing the acquired spectral response of the object to be measured with a pre-stored reference spectral response; and determining whether the object to be measured is a living body based on a comparison result between the reference spectral response and the spectral response of the object to be measured. On the basis of the structure of the present invention, the patent contents of the Chinese invention CN202110275126X are all introduced into the present invention. Preferably, the spectral information can be denoised by subtracting the reference spectral information, and then the conversion of the spectral response is performed to determine the living body.

Liveness identification method

A neural network can be used for live body recognition.

Fig. 13 is a schematic diagram of the neural network model of the present invention. Specifically, Figure 13 illustrates a multi-layer perceptron model. The number of nodes in the input layer is the same as the number of pixels in the target area. When performing calculations, the obtained data of each pixel is used as the data of each node of the input layer. After that, activation and full connection operations will be performed between each layer. The final output layer is 1 node, and only the fully connected layer is connected to the previous layer. The output of this layer can be operated using the Logistic function, and it will be judged whether it is a living body according to whether the output of the Logistic function is greater than 0.5.

When actually using the above network, it includes two steps of training and detection.

In the training step, the input value is the image sensor to detect living or non-living objects, and the output is whether it is living or not (for example, 1.0 for living and 0.0 for non-living). Then, the parameters in the network are trained by means of backpropagation.

In the detection step, the input value is the object to be detected, and whether it is a living body is judged according to the output value (for example, whether it is greater than 0.5).

Identification scheme

Further, a fingerprint identification method is also provided, projecting the first detection light to the subject; receiving the first detection light reflected back by the subject and generating the subject object based on the first detection light the first spectral information and image information; project the second detection light to the subject; receive the second detection light reflected back by the subject and generate the subject based on the second detection light second spectral information of the target; and performing living body detection and object recognition based on the first spectral information, the image information and the second spectral information. Wherein, the first detection light is mixed light, for example, at least two different light-emitting units in the OLED screen light R, G, and B light-emitting units emit light and project the first detection light. Preferably, the first detection light is implemented as white light ; The second detection light is preferably implemented as monochromatic light, such as green light, blue light. That is, the first detection light and the second detection light are two different types of light. The first detection light and the second detection light enter the OLED screen after being reflected by the object to be measured, and are modulated by the OLED screen, so that the corresponding image information and the first detection light are received by the image sensor. spectral information, and second spectral information. Wherein, the first spectral information and the second spectral information are acquired by spectral pixels of the image sensor.

Wherein, performing living body detection and object recognition based on the first spectral information, the image information, and the second spectral information includes: processing the first spectral information and the second spectral information to generate a second spectral information A spectral response result and a second spectral response result; processing the image information to generate an image of the subject; comparing the image of the subject with a pre-stored reference image; responding to the If the matching between the image of the object and the reference image is successful, it is determined whether the object is a living body based on the first spectral response result and/or the second spectral response result, as shown in FIG. 15 . FIG. 15 illustrates a flow chart of the liveness detection and object recognition steps in the method shown in FIG. 14 .

Live fingerprint identification process

Put the finger to be tested on the area to be tested on the screen, the light source of the screen starts to project light, the light reaches the finger to be tested, part of it is absorbed, and part of it is reflected to form incident light, and the incident light enters the screen The slit unit is modulated, received by the image sensor to obtain image information and spectral information, and then imaging and living body judgment are performed based on the image information and spectral information. Specifically, the image information is used to restore the fingerprint image, and then the fingerprint image is compared with the pre-stored reference fingerprint image, and the spectral information can be converted into a spectral response or a spectral curve, and compared with the pre-stored reference spectral response or reference Spectral curves are compared to determine whether it is a living body.

Based on the image information and spectral information, the identification and comparison of living body and fingerprint can be carried out through the neural network.

Spectrometer Example

With the development of computational spectroscopy, it is possible to miniaturize spectrometers. Currently, computational spectroscopy requires a specific structural filter structure and a corresponding algorithm to achieve spectral recovery. Its essence can be understood as, after the image sensor measures the spectral response, it is sent to the data processing unit for recovery calculation. The process is described in detail as follows:

The intensity signal of the incident light at different wavelengths λ is denoted as x(λ), the transmission spectrum curve of the filter structure is denoted as T(λ), and there are m groups of structural units on the filter (filter structure), each The transmission spectra of the group structural units are different from each other, and overall, the filter structure can be recorded as Ti(λ) (i=1,2,3,...,m). There are corresponding physical pixels under each group of structural units to detect the light intensity bi modulated by the filtered light structure. In a specific embodiment of the present application, a physical pixel is used, that is, a physical pixel corresponds to a group of structural units, but it is not limited thereto. In other embodiments, a group of multiple physical pixels may also correspond to a group Structural units. Therefore, in the computational spectroscopy device according to the embodiment of the present application, at least two groups of structural units constitute a "spectral pixel" (it can be understood that multiple groups of structural units and corresponding image sensors constitute a spectral pixel). It should be noted that the effective transmission spectrum of the filter structure (the transmission spectrum used for spectral restoration is called the effective transmission spectrum) Ti(λ) quantity and the number of structural units may be inconsistent, and the transmission spectrum of the filter structure According to the needs of identification or restoration, artificially set, test, or calculate according to certain rules (for example, the transmission spectrum of each of the above-mentioned structural units through the test is the effective transmission spectrum), so the effective transmission spectrum of the filter structure The number of can be less than the number of structural units, and may even be more than the number of structural units; in this variant embodiment, a certain transmission spectrum curve is not necessarily determined by a group of structural units. Further, the present invention can use at least one spectral pixel to restore an image. That is to say, the spectroscopic device in this application can restore the spectral curve or perform spectral imaging according to the spectral response.

The relationship between the spectral distribution of incident light and the measured value of the image sensor can be expressed by the following equation:

bi=∫x(λ)*Ti(λ)*R(λ)dλ

Then discretize, get

bi=Σ(x(λ)*Ti(λ)*R(λ))

Where R(λ) is the response of the image sensor, recorded as:

Ai(λ)=Ti(λ)*R(λ),

Then the above formula can be expanded into matrix form:

Among them, bi(i=1,2,3,...,m) is the response of the image sensor after the light to be measured passes through the filter structure, corresponding to the light intensity measurement values of the image sensor corresponding to the m structural units, when a physical pixel When corresponding to a structural unit, it can be understood as light intensity measurement values corresponding to m "physical pixels", which is a vector with a length of m. A is the light response of the system to different wavelengths, which is determined by two factors: the transmittance of the filter structure and the quantum efficiency of the image sensor. A is a matrix, and each row vector corresponds to the response of a group of structural units to incident light of different wavelengths. Here, the incident light is discretely and uniformly sampled, and there are n sampling points in total. The number of columns of A is the same as the number of sampling points of the incident light. Here, x(λ) is the light intensity of the incident light at different wavelengths λ, that is, the spectrum of the incident light to be measured.

The present invention adopts a substrate with periodic slit units, hole units or column units as the light filtering structure, and the substrate is arranged on the optical path of the image sensor, taking the slit unit as an example, wherein the slit unit consists of at least A slit is formed. The slit unit has a corresponding transmission spectrum matrix T, which can modulate the incident light, so as to be received by the image sensor to obtain a light intensity measurement value.

Taking the implementation of the substrate as an OLED screen as an example, a photosensitive module including an image sensor is placed under the OLED screen to form a spectrometer. The incident light passes through the slit unit of the OLED screen and is modulated by the slit unit. and then received by the image sensor.

The spectrometer in this embodiment can achieve diffraction and/or interference based on the OLED screen. Preferably, the OLED screen in the present invention can produce interference effects as much as possible, so it is necessary to consider the design of the screen or slit. It should be understood that the OLED screen will R, G, and B light-emitting units are regularly distributed. For example, as shown in the figure, the conventional R, G, G, and B light-emitting units are arranged in an array, and the slits are formed between the light-emitting units. A plurality of slits will be formed between the RGGB light-emitting units, and if the plurality of slits are defined as a group of slit units, the slit units need to be arranged in a fixed period, that is, the distance between adjacent slit units ( period) are equal (it is only necessary to ensure that the screen area participating in the modulation of the incident light has this characteristic), which can ensure that the interference effect is as obvious as possible, so that the image information received by the image sensor contains spectral characteristics. It should be understood that the present invention does not limit that the screens must be arranged in an RGGB array, and the screens can be arranged in other ways. At this time, the slit units can also be adjusted accordingly. A cell containing at least one slit is defined as a slit cell. It should be understood that the different slit units do not need to be exactly the same, but the difference should not be too large, so as to avoid no interference effect.

For a better understanding, the slit unit is further explained. The OLED screen has a pixel layer (light-emitting layer) and a circuit layer (TFT structure layer), which will prevent the incident light from passing through, while the pixel (light-emitting unit) and the TFT Between the structures are slits and/or small holes that allow incident light to pass through. These light-transmitting slits and small holes are periodic in a certain range, for example, they are periodic in the entire screen, or the test area corresponding to the photosensitive module is periodic, and of course they can be other areas. At least one slit and/or small hole constitutes a slit unit, and any slit unit can define a vector a and a vector b with its two adjacent slit units, that is, vector a, vector b and an area equal to The area of point a multiplied by b (the area of the parallelogram formed by vector a and vector b). The pattern in this area is within the scope of the periodic area, and after the displacement of an integer number of vectors a and an integer number of vectors b is translated along the corresponding vector direction, the slits and/or small holes will overlap. The periodic area has at least 25 slit units. Generally, the angle between vector a and vector b is 90 degrees. Specifically, Figures A and B are two different OLED screen slits and/or small holes. The bright area is the OLED slit and/or small hole, and the rectangular area framed is the slit unit. Of course, different screens The corresponding slit unit area will be different, and is also limited to a rectangular area. It should be noted that the slits between the slit units in the present invention may be different from each other, but the shape, structure and size of the slit units are basically the same, but due to certain errors in processing, the slits There may be certain differences between the units, which can also be understood as being consistent with the concept of the present invention and covered by the present invention.

It should be noted that, in order for the image sensor to obtain spectral information as fully as possible, there should be at least two transmission spectrum curve correlations between the transmission spectrum curves Ti(λ) (i=1,2,3,...,m) The degree is less than or equal to 0.4, and the degree of correlation can be defined by the Pearson correlation coefficient. Further, to define the transmission spectrum curve of the present invention, it should be understood that the existence of the transmission spectrum curve is mainly due to the presence of slits in the OLED screen, and the incident light passing through the slits will be modulated, and the transmission spectrum curve can be considered as determining the modulation of the incident light. Effect, so multiple slits constitute a slit unit will have a corresponding transmission spectrum curve, but the preferred transmission spectrum curve of the present invention is not determined by a single slit unit, it may be affected by the surrounding slit units, that is, the preferred transmission spectrum curve of the present invention The transmission spectrum curve is determined by at least two slit units. Further, the number of the transmission spectrum curves is equal to the number of effective light intensity bi obtained, and the effective light intensity bi refers to the light intensity information used for spectrum recovery or spectral response judgment, and its number n is equal to the number of transmission spectrum curves; generally In the application, the incident light will be discretely and uniformly sampled, and there are n sampling points in total, such as the 200-400nm band, and the spectral resolution is 1nm, then the sampling point is 201; at this time, the transmission spectrum matrix formed by the transmission spectrum curve is A matrix of n*m.

Further, the OLED screen includes a glass cover and a light-emitting unit located at the lower end of the glass cover. During the identification process, the incident light to be measured needs to enter the glass cover, and the incident light is modulated by the slit unit of the OLED screen. , and then received by the image sensor to obtain spectral information modulated by spatial dispersion.

The photosensitive module may further include an optical component, and the optical component adjusts the modulated light.

It should be noted that the transmission spectrum curve corresponding to the slit unit of the OLED screen is relatively sensitive to the angle of incident light, that is, the change of the angle of the incident light will cause the transmission spectrum curve to change. Therefore, when the spectrometer is applied, it is necessary to judge the angle of the incident light, and select the corresponding transmission spectrum matrix for spectrum restoration.

The spectrometer further includes a memory and a processing unit, the memory and the processing unit are communicatively connected to the image sensor, and may also be integrated into the image sensor. The transmission spectrum matrix can be digitized and stored in the memory, or the transmission spectrum matrix can be converted according to the recovery algorithm requirements, digitized and stored in the memory.

Spectrometer Embodiment Requiring a Light Source

For example, when the spectrometer is used for object recognition, the spectrometer may optionally further include a light source, and preferably the spectrum is implemented as a light emitting unit of the OLED screen. Wherein, the light-emitting unit of the OLED screen emits light to the object to be measured, and the object to be measured will partially absorb and partially reflect the light source, and the reflected light enters the slit unit of the OLED screen to be modulated, and then is received by the image sensor , to obtain the light intensity measurement value; and then obtain the spectral information (spectral curve) through calculation, so as to judge the object. It should be noted that the object recognition system also includes a placement area for the object to be measured, and the distance between the placement area and the OLED screen is preferably less than or equal to 6cm, preferably less than or equal to 3cm; thus, the incident angle of the incident light can be better conformed to the modulation demand, making the modulation effect better.

Further, the spectrometer can also be used to measure jaundice, color temperature, etc. It can restore the spectral curve according to the incident light, and then perform jaundice identification or color temperature measurement according to the spectral curve.

It should be noted that the structure and principle of the spectrometer described in the present invention are highly consistent with the above embodiment of the living fingerprint identification system. This embodiment focuses on citing the content of the above embodiment, and explains some differences and details.

alternative embodiment

As mentioned above, the present invention utilizes the principle of interference and diffraction of slits or small holes to realize the modulation of incident light. Under the condition of ensuring that the interference and diffraction effects are as obvious as possible, the image information received by the image sensor can contain spectral characteristics, and then Spectral characteristics can be used for spectrum recovery, material identification, authenticity judgment and other applications. Therefore, in an alternative embodiment of the invention, the spectroscopic device is no longer built based on a screen. Figure 20 illustrates a schematic diagram of a spectroscopic device according to an alternative embodiment of the present application. As shown in FIG. 20 , the spectroscopic device of this embodiment includes a modulation cover and an image sensor, that is, the substrate is implemented as a modulation cover. The modulation cover is located at the upper end of the image sensor, and the modulation cover has a slit unit. Specifically, the incident light is modulated through the slit unit, that is, interference and diffraction effects are produced, and the modulated incident light Received by the image sensor to obtain spectral information. The modulation cover can be made of a transparent material, such as transparent plastic or transparent glass. Preferably, because the transmittance of glass is relatively high, a glass cover can be selected, and then a layer is applied on the surface of the modulation cover. As for the opaque material, the slit unit of the present invention is formed where no opaque material is applied, and the opaque material can be formed on the modulation cover plate by processes such as evaporation and attachment. That is, the modulation cover includes a glass cover made of a transparent material and an opaque material covering the glass cover, and the modulation cover is not covered with the opaque material to form the slit unit. As shown in FIG. 21 , the slit unit includes at least one slit and/or a small hole, that is, the slit unit is composed of at least one slit or at least one small hole, so as to have interference and diffraction effects. FIG. 21 is a schematic diagram illustrating an example of the structure of a modulation cover plate of the spectrum device as shown in FIG. 20 .

Preferably, the slit unit formed by at least one slit and/or small hole in the present invention has a certain periodicity; specifically, any slit unit can define a vector a and Vector b, that is, you can find vector a, vector b and an area whose area is equal to point a multiplied by b (the area of the parallelogram formed by vector a and vector b). The pattern in this area is within the scope of the periodic area, and after the displacement of an integer number of vectors a and an integer number of vectors b is translated along the corresponding vector direction, the slits and/or small holes will overlap. The periodic area has at least 25 slit units. Generally, the angle between vector a and vector b is 90 degrees. It should be noted that the slits between the slit units in the present invention may be different from each other, but the shapes, structures, and sizes of the slit units are basically the same, that is, a slit unit may have different structures and sizes. Or shaped slits or small holes, but due to certain errors in processing, there may be certain differences between slit units, which can also be understood as being consistent with the concept of the present invention and covered by the present invention.

Taking at least one slit forming the slit unit as an example, the modulation cover plate processing technology of the present invention can be implemented as applying a photoresist on the transparent cover plate, after curing, developing, and then etching, the engraved Apply opaque material to the etching area to form a light-shielding area, and then remove the rest of the photoresist to form a slit. Relatively speaking, the slit structure and dimensional accuracy of the slit unit corresponding to this embodiment are higher than those corresponding to the OLED screen, and at the same time, it is easier to process and obtain.

In another modified example, as shown in FIG. 22 , the modulation cover can be made of opaque material, and the slit unit of the present invention includes at least a small hole, through which interference and diffraction effects are realized. For example, the modulation mask can be implemented as a random reticle, and a high-precision modulation mask can be formed using a mature process. FIG. 22 illustrates a schematic diagram of another example of the structure of the modulation cover plate of the spectrum device as shown in FIG. 20 .

Further, as shown in FIG. 23 , the spectrum device in this embodiment further includes an optical component, and the optical component is located on the photosensitive path of the image sensor. Preferably, the optical assembly is located between the modulation cover and the image sensor. The optical component can be a lens, a filter or a combination thereof, and mainly adjusts the modulated light. Further, the spectrum device may further include a bracket assembly for fixing the optical assembly and the modulation cover. Further, the spectrum device includes a circuit board, and the image sensor is electrically conductively connected to the circuit board. The bracket assembly is preferably fixed to the circuit board. FIG. 23 illustrates a schematic diagram of a configuration including optical components of the spectroscopic device shown in FIG. 20 .

That is, in this embodiment, the above-mentioned OLED screen is replaced with a specific modulation cover to realize the collection of spectral information. Its working principle and application scenarios are similar to those of the OLED screen. Further, in this embodiment, the light source can be provided separately, and the function of the OLED screen can be realized through the combination of the light source and the modulation cover plate.

Preferably, the spectroscopic device further includes a collimation system for collimating the incident light. The collimation system may be implemented as at least one lens, or an array of microlenses. The collimation system is located on the upper end of the modulation cover, that is, the incident light enters the modulation cover after passing through the collimation system for modulation.

In individual embodiments, the opaque material includes a conductive material such as a metal material, and the capacitor can be formed by using the metal material in parallel. Note that two parallel metal materials cannot be conducted, and can be assisted by a non-conductive opaque material. slit unit. That is, in this embodiment, the opaque materials are divided into conductive materials and non-conductive materials, and capacitors are formed by arranging conductive materials in parallel, and then corresponding slits are assisted by non-conductive materials to form slit units. Connect the conductive material to the circuit board, and the slits formed can be equivalent to capacitors. That is, the circuit board is suitable for being connected to the capacitor structure. Record the reference capacitance value under normal conditions. If dust or dirt enters the slit during use, the capacitance value will change. You can choose to set the threshold value. When the difference between the capacitance value and the reference capacitance value exceeds the threshold value, the user will be reminded The brew cover surface needs to be cleaned.

alternative embodiment

Existing consumer electronics, wearable devices, etc. all have at least one camera module for taking pictures. That is, the consumer electronic device includes a device main body and a camera module, and the camera module is installed on the device main body. Further, the consumer electronic device includes a protective cover, the protective cover is arranged on the main body of the device, and forms a closed space with the main body of the device, and the camera module is located in the closed space, so that Prevent dust from adhering to the lens surface of the camera module, thereby affecting imaging.

In this embodiment, a mobile phone is taken as an example. FIG. 24 is a schematic diagram of the back of an existing mobile phone, generally with at least one camera module behind; FIG. 25 is a schematic diagram of the back of a mobile phone with a protective cover in this embodiment. The electronic device includes a device main body, an image sensor, and a protective cover, the image sensor and the protective cover are arranged on the device main body, wherein the protective cover is located on the photosensitive path of the image sensor; further, the The protective cover has a light-transmitting area and a non-light-transmitting area, wherein the light-transmitting area is composed of a plurality of slits, that is, this embodiment forms a light-transmitting area and a non-light-transmitting area on the protective cover, so that the The cover plate realizes the function of the modulation cover plate in the previous embodiment. That is, the modulation cover is a protective cover of an electronic device, the protective cover has a light-transmitting area and a non-light-transmitting area, and the light-transmitting area forms the slit unit.

Preferably, an opaque material can be applied to the surface of the protective cover, so that the area with the opaque material forms a non-transmissive area, and the area without the opaque material forms a light-transmitting slit, at least A slit constitutes a slit unit for modulating incident light (using interference and diffraction effects to achieve wide-spectrum modulation). Preferably, the opaque material is located on the inner surface of the protective cover, so as to prevent dust and particles from falling between the slits, thereby affecting the modulation effect. It should be noted that the change of the slit unit in the present invention will affect the corresponding modulation effect to a certain extent, so that the built-in recovery and identification algorithms may not be able to accurately realize spectrum recovery or substance identification, so the opaque material is located on the protective cover The inner surface of the plate can make the slot not be affected by the environment.

The consumer electronic device may also include an optical assembly, a circuit board, and a bracket, and the optical assembly, the circuit board, the bracket, and the image sensor form a camera module, and the camera module is fixed on the main body of the device and the The sealed space formed by the protective cover. The optical components may be lenses and/or filters. It may further include a focusing mechanism, such as a voice coil motor, SMA, etc., to drive the lens to move to achieve focusing.

The focus of this embodiment is to implement the protective cover of consumer electronics as a light filtering structure with a modulation effect (or equivalent to the OLED screen or modulation cover in the previous embodiments)

alternative embodiment

However, it should be noted that part of the projected light A projected by the light-emitting unit will directly enter the slit to reach the image sensor; part of the projected light B will reach the glass cover and directly reflect into the slit, and then be received by the image sensor; After reaching the object (finger) to be measured, it is reflected into the slit and received by the image sensor; and part of the projected light D is absorbed by the object (finger) to be measured. The living body recognition, its original intention is that due to the presence of capillaries, sweat glands, etc., fingers have different absorption of light in different wavelength bands, which is different from conventional silica gel and pseudo-fingers in the absorption of projected light, which can be used to judge liveness. Therefore, the really useful projected light should be projected light C and projected light D, while projected light A, projected light B and ambient light are stray light to a certain extent. Therefore, in a dark room environment, the light emitting unit can project the same projection light. At this time, the incident light received by the image sensor can basically be regarded as the projection light A and projection light B being received by the image sensor after passing through the slit. The collected spectral information in this case is recorded as the reference spectral information, and the stray light brought by the projection light A and the projection light B can be removed by subtracting the reference spectral information from the spectral information acquired by subsequent testing of the object to be measured. Therefore, the restoration accuracy of the spectral curve is higher.

Spectral Curve Restoration Method

The present invention further provides a neural network-based spectral restoration method, comprising: acquiring sampled spectral data to be processed; and inputting the sampled spectral data to be processed into a neural network with predetermined parameters to output a spectral restoration result. Specifically, all the contents of the Chinese invention CN2021104180126 are introduced

Wherein, the neural network is formed by training spectral data for training, and the training process of the neural network includes: obtaining a pair of spectral data for training, wherein the pair of spectral data for training includes spectral data before sampling and spectral data after sampling Data, the pre-sampling spectral data is formed based on the superposition of at least one Gaussian distribution and/or at least one Lorentzian distribution of spectral curves; and, using the pre-sampling spectral data of the training spectral data pair as input data and The training uses the sampled spectral data of the spectral data pair as a label, and trains the neural network used for spectral recovery until the parameters of the neural network converge.

Further, wherein, obtaining the pair of spectral data for training includes: generating the pre-sampling spectral data with a first preset length based on the superposition of at least one Gaussian distribution and/or at least one Lorentzian distribution; Adding the first noise spectral data to the pre-sampling spectral data to obtain the pre-sampling spectral data after adding noise; and sampling the pre-sampling spectral data after adding noise to obtain the post-sampling spectrum with a second preset length data.

Further provide another high-resolution spectral recovery method, including:

Step 1: Obtain the discrete cosine transformed dictionary of the transmission spectrum of the spectrum chip, the discrete cosine transform dictionary, and the measured value vector of the image sensor of the spectrum chip;

Step 2: Based on the first layer modeling of Bayesian hierarchical modeling, the sparse vector corresponding to the spectral vector is modeled as a vector of normal product distribution to obtain the vector of the first normal distribution variable and the second normal distribution variable The vector of the normal distribution variable, wherein, calculate the dot product of the vector of the first normal distribution variable and the vector of the second normal distribution variable to obtain the vector of the normal product distribution, and calculate the first positive The dot product of the first covariance matrix of the vector of the vector of the state distribution variable and the second covariance matrix of the vector of the second normal distribution variable obtains the covariance matrix of the vector of the normal product distribution;

Step 3: Based on the second layer modeling of Bayesian hierarchical modeling, the first covariance matrix of the vector of the first normal distribution variable and the second covariance matrix of the vector of the second normal distribution variable The reciprocal of the product of the variance corresponding to each position in the variance matrix is modeled as a gamma distribution obeying the first hyperparameter and the second hyperparameter;

Step 4: Based on the Bayesian method, calculate an estimated vector of the first posterior probability density of the vector of the first normally distributed variable and an estimate of the second posterior probability density of the vector of the second normally distributed variable vector;

Step 5: calculating the vector of the normal product distribution based on the dot product of the estimated vector of the first posterior probability density and the estimated vector of the second posterior probability density;

Step 6: Based on the first covariance matrix, the second covariance matrix, the estimated vector of the first posterior probability density, the estimated vector of the second posterior probability density, the first hyperparameter updating a first expectation matrix and a second expectation matrix corresponding to the first covariance matrix and the second covariance matrix with the second hyperparameter;

Step 7: Repeat steps 4 to 6 until the iteration condition is met;

Step 8: Calculate the covariance matrix of the vector of the normal product distribution based on the first expectation matrix and the second expectation matrix; and

Step 9: Obtain a spectral vector based on the vector of normal product distribution and its covariance matrix and the discrete cosine transform dictionary.

For ease of understanding, the entire content of Chinese invention CN2021109755685 is introduced into the present invention.

A method for spectral restoration is further provided, comprising:

Step 1: Obtain the transmission spectrum matrix of the spectrum chip and the measured value vector of the image sensor of the spectrum chip;

Step 2: Establish an augmented matrix from the transmission spectrum matrix based on the improved regularized description model, the augmented matrix includes the first sub-matrix on the upper left, the second sub-matrix on the upper right, the third sub-matrix on the lower left and the lower right sub-matrix The fourth sub-matrix of ;

Step 3: set the first spectral vector;

Step 4: Determine the maximum residual error row based on the transmission spectrum matrix, the measured value vector and the first spectrum vector;

Step 5: Determine a first iteration vector and a first spectral residual vector based on the first spectral vector;

Step 6: updating the first iteration vector based on the row corresponding to the maximum residual row of the first sub-matrix and the second sub-matrix of the augmented matrix;

Step 7: Determine the rows to be iterated in the third sub-matrix and the fourth sub-matrix of the augmented matrix;

Step 8: updating the first spectral vector and the first spectral residual vector based on the row to be iterated and the updated first iteration vector;

Step 9: repeat steps 6 to 8 until all rows of the third sub-matrix and the fourth sub-matrix of the augmented matrix are calculated; and

Step 10: Repeat steps 4 to 9 until the first spectral residual vector satisfies a predetermined condition.

In order to facilitate understanding, all the content involved in the Chinese invention CN 2021108481584 is introduced into the present invention.

Further, a high-resolution spectral recovery method is provided, including:

Step 1, obtaining the transmission spectrum matrix of the spectrum chip and the measured value vector of the image sensor of the spectrum chip;

Step 2, setting the predetermined selection probability of each row of the transmission spectrum matrix, the predetermined selection probability is the square of the second norm of a certain row of the transmission spectrum matrix and the square of the Frobenius norm of the transmission spectrum matrix business;

Step 3, selecting a predetermined row of the transmission spectrum matrix based on the predetermined selection probability;

Step 4, based on the inner product of the spectral vector before iteration and the predetermined row, the value of the measured value vector and the corresponding position of the predetermined row, the bi-norm of the predetermined row and the predetermined row to obtain an update vector;

Step 5, subtracting the update vector from the spectral vector before the iteration to obtain the iterated spectral vector; and,

Step 6, repeat steps 3 to 5 until the iterated spectral vector satisfies the termination condition, the termination condition is based on the iterated spectral vector and its two norm, the transmission spectrum matrix and its Frobenius norm and the vector of measurements.

The spectrometer includes an OLED screen, an image sensor, a memory and a processing unit, and the memory and the processing unit can optionally be integrated in the image sensor respectively, or can only be connected in a communicative manner; wherein, the memory is connected to the OLED The transmission spectrum matrix of the slit unit of the screen is sampled, quantized and stored in a digital format; in some embodiments, the transmission spectrum matrix can also be calculated and then stored. The processing unit configures the image sensor with instructions stored thereon so that the processing unit can restore the spectral curve: the spectral response data generated on the image sensor according to the spectral transmission spectrum matrix corresponding to the incident light. Preferably, the transmission spectrum matrix may be stored in memory.

Further, a method for providing spectral resolution is provided, (a) the incident light modulated by the slit unit of the OLED screen is received by the receiving image sensor, and the spectral response data is obtained; (b) the transmission spectrum matrix is digitized; and (c) Improving spectral resolution using at least one of the following operations: Least Square estimate process, Matrix inversion, equalization, or Pseudoinverse matrix manipulation; The slit cells of the OLED screen are implemented as broadband filters; wherein the spectral responses of the different slit cells are independent of different peaks and troughs, distributed over the entire target spectral range, and integrated with the plurality of slit cells of the OLED screen overlapping. Wherein, digitizing includes digitizing steps including using sampling and quantization.

A spectral recovery method is provided. Regularization parameters are selected based on the transmission spectrum matrix of the OLED screen and the spectral response data collected by the image sensor. Preferably, the regularization parameters can be based on parameters such as generalized maximum likelihood estimation, leave-one-out cross-validation, and generalized moment estimation. Estimation method selection; preferably, the transmission spectrum matrix of the OLED screen can be reduced in dimension, thereby reducing the amount of calculation; based on the selected regularization parameters and using a processor for non-negative least squares solution, the solution method includes but is not limited to preprocessing Conjugate gradient method, trust region reflection method, bounded variable least square method, etc., to complete spectral reconstruction

Spectral Imaging Example

It should be noted that the principle of spectral imaging is to record the intensity signal of incident light at different wavelengths λ as f(λ), the transmission spectrum curve of the filter structure as T(λ), and the filter has m groups of The filter structure, each group of transmission spectrum is different, also known as "structural unit", the whole can be recorded as Ti(λ) (i=1,2,3,...,m). There are corresponding physical pixels under each group of filter structures to detect the light intensity Ii modulated by the filter structures. In a specific embodiment of the present application, it is described by taking one physical pixel corresponding to a group of structural units as an example, but it is not limited thereto. In other embodiments, a group of multiple physical pixels may also correspond to a group of structures unit.

The relationship between the spectral distribution of the incident light and the measured values of the photodetector array can be expressed by the following equation:

Ii=Σ(f(λ) Ti(λ) R(λ))

Where R(λ) is the response of the detector, recorded as:

Si(λ)=Ti(λ) R(λ)

Then the above formula can be expanded into matrix form:

Among them, Ii (i=1,2,3,...,m) is the response of the photodetector after the light to be measured passes through the broadband filter unit, corresponding to the light intensity measurement values of m photodetector units, also known as m A "physical pixel", which is a vector of length m. S is the optical response of the system to different wavelengths, which is determined by two factors: the transmittance of the filter structure and the quantum efficiency of the photodetector response. S is a matrix, and each row vector corresponds to the response of a broadband filter unit to incident light of different wavelengths. Here, the incident light is discretely and uniformly sampled, and there are n sampling points in total. The number of columns of S is the same as the number of sampling points of the incident light. Here, f(λ) is the light intensity of the incident light at different wavelengths λ, that is, the spectrum of the incident light to be measured.

In practical applications, the response parameter S of the system is known, and the spectrum f of the input light can be obtained through the light intensity reading I of the detector, and the spectrum f of the input light can be obtained by using an algorithm. The process can use different data processing methods depending on the specific situation, including but not Limited to: least squares, pseudoinverse, equalization, least squares norm, artificial neural network, etc.

Taking one physical pixel corresponding to a group of structural units as an example, the above describes how to use m groups of physical pixels (that is, pixels on the image sensor) and their corresponding m groups of structural units (the same structure on the modulation layer is defined as a structural unit ) to restore a spectral information, also known as "spectral pixel". It should be noted that, in the embodiment of the present application, multiple physical pixels may also correspond to a group of structural units. It can be further defined that a group of structural units and at least one corresponding physical pixel constitute a unit pixel, and in principle, at least one unit pixel constitutes a spectral pixel.

On the basis of the above implementation manner, the spectral pixels are arrayed to realize a snapshot spectral imaging device.

For example, as shown in Figure 16, an image sensor with 1896*1200 pixels is used (Figure 16 shows a part of the image sensor area), and m=4 is selected at the same time, that is, 4*4 unit pixels are selected to form a spectral pixel, then it can be realized at this time 474*300 mutually independent spectral pixels, each of which can calculate the spectral result independently by the above method. After the image sensor is combined with the lens group and other components, snapshot spectral imaging of the object to be measured can be performed, so that the spectral information of each point of the object to be measured can be obtained in a single exposure. Fig. 16 illustrates a schematic diagram of a first example of a spectral pixel array of an image sensor according to the present invention.

On this basis, according to actual needs, without any adjustment to the image sensor, the selection method of spectral pixels can be rearranged to improve the spatial resolution. As shown in Figure 17, the dense arrangement of solid-line boxes and dotted-line boxes can be selected to increase the spatial resolution in the above example from 474*300 to close to 1896*1200. Fig. 17 illustrates a schematic diagram of a second example of a spectral pixel array of an image sensor according to the present invention.

Further, for the same image sensor, spatial resolution and spectral resolution can be rearranged as needed. For example, in the above example, when the spectral resolution is required to be high, 8*8 unit pixels can be used to form a spectral pixel ; When the spatial resolution is required to be high, 3*3 physical pixels can be used to form a spectral pixel.

The spectral imaging system and the spectrometer system are consistent in structure, but there are differences in their restoration algorithms. Specifically, the spectral imaging algorithm is provided on the basis of the structure of the spectrometer embodiment.

A method of spectral restoration is provided, including:

Obtain the optical energy response signal matrix and standard spectrum output by the photosensitive chip of the spectral imaging device; determine the basic element recovery function and the response signal vector of the basic element recovery function based on the optical energy response signal matrix, and the basic element recovery function uses The predetermined pixel value of the photosensitive chip and its nearby pixel values are restored to the spectral image value of the corresponding predetermined channel; the restoration tensor is obtained, and the product of the restoration tensor and the response signal vector is equal to the basic element restoration The function is based on the output of the response signal vector; and the restored spectral image is obtained based on the product of the restoration tensor and the response signal vector.

Wherein, the optical energy response signal matrix is expressed as a matrix B including two dimensions of image width w and image height h, the dimension of the standard spectrum is 1, and the spectral image received by the spectral imaging device is set to be real The distance between the product of the value tensor and the standard spectrum and the spectral image tensor to be restored is the smallest.

Further, the standard spectrum is denoted as s, and the channel standard spectrum corresponding to the kth channel of the standard spectrum is denoted as s _k , so that:

x _k →O(i,j)s _k

Among them, s _k is the spectral image value of the k-th channel of a certain spectral pixel, O(i, j) is the real value tensor of the spectral curve of a certain spectral pixel, and → indicates that the Euclidean distance between the tensors is the smallest.

Wherein, obtaining the transmission spectrum matrix of the spectrum chip and the measurement value vector of the image sensor of the spectrum chip includes: obtaining the initial transmission spectrum matrix A of the spectrum chip and the initial measurement value vector b of the image sensor of the spectrum chip; A regularized description model, by extracting coefficients from the spectral vector from the initial transmission spectrum matrix A and the initial measured value vector b to obtain the matrix A' and measured value vector b' of the overdetermined system, wherein the regularized described model is:

Among them, λ>0 is the coefficient of the regularization term, D is the tridiagonal Toeplitz matrix, ‖·‖ represents the two-norm, and the matrix A' and the measured value vector b' of the overdetermined system are respectively:

And, the matrix A' and the measured value vector b' of the overdetermined system are respectively used as the transmission spectrum matrix and the measured value vector of the spectrum chip.

For ease of understanding, the entire content of Chinese invention CN2021111546565 is introduced.

Further, a spectral image reconstruction method is provided, including: obtaining the transmission spectrum data of the spectral imaging chip and the output signal data of the spectral imaging chip; obtaining the local transmission spectrum data and local output signal data of the output signal data; inputting the local output signal data into an attention model to obtain attention local data; and combining the local transmission spectrum data, the local output signal data and the attention The local data is input into the neural network model to obtain the pixels for reconstructing the spectral image.

Wherein, acquiring the local transmission spectrum data of the transmission spectrum data and the local output signal data of the output signal data based on the pixel used for reconstructing the spectral image comprises: based on the position of the pixel used for reconstructing the spectral image, acquiring the The partial transmission spectrum data of the transmission spectrum data and the partial output signal data of the output signal data in the vicinity of the position, the side length of which is a predetermined number of pixels.

Further, inputting the local output signal data into the attention model to obtain the local attention data includes: dividing the local output signal data into a plurality of predetermined regions, each predetermined region including a plurality of pixels related to the spectral imaging chip corresponding output signal data; and performing matrix multiplication for each of the predetermined regions to obtain the attention local data.

For ease of understanding, the entire content of Chinese invention CN2021111516729 is introduced.

The basic principles of the present application have been described in conjunction with specific embodiments, but it should be pointed out that the advantages, advantages, effects, etc. mentioned in the application are only examples and not limiting, and these advantages, advantages, effects, etc. Each embodiment of the application must have. In addition, the specific details disclosed above are only for the purpose of illustration and understanding, rather than limitation, and the above details do not limit the application to be implemented by using the above specific details.

The block diagrams of devices, devices, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, devices, systems may be connected, arranged, configured in any manner. Words such as "including", "comprising", "having" and the like are open-ended words meaning "including but not limited to" and may be used interchangeably therewith. As used herein, the words "or" and "and" refer to the word "and/or" and are used interchangeably therewith, unless the context clearly dictates otherwise. As used herein, the word "such as" refers to the phrase "such as but not limited to" and can be used interchangeably therewith.

It should also be pointed out that in the devices, equipment and methods of the present application, each component or each step can be decomposed and/or reassembled. These decompositions and/or recombinations should be considered equivalents of this application.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

A living fingerprint identification system is characterized in that it comprises:

The screen has a plurality of slit units arranged periodically, used to receive the incident light generated when the light projected by the light source is reflected on the finger to be tested, and modulate the incident light, and the slit units have a corresponding transmission spectrum curve;

A photosensitive module, the photosensitive module is located at the lower end of the screen, and includes:

The image sensor is configured to receive the modulated incident light, obtain spectral information of the incident light, and process the spectral information.
The living fingerprint identification system according to claim 1, wherein each slit unit comprises at least one slit and/or small hole.
The living fingerprint identification system according to claim 2, wherein the slits of the plurality of slit units are arranged in the same way.
The living fingerprint identification system according to claim 2, wherein the shapes and/or structures and/or sizes of the plurality of slit units are consistent.
The living fingerprint identification system according to claim 2, wherein any one of the slit units and its two adjacent slit units define an area whose sum of two vectors is equal to the dot product of the two vectors, and the area is After the pattern of the periodical region is translated along the vector directions corresponding to the two vectors for an integer number of displacements of the vectors, the slits in this region coincide with the slits in the region after the translation, wherein , the periodic area is an area formed by a plurality of slit units arranged in a periodic manner.
The living fingerprint identification system according to claim 2, wherein a plurality of said slit units are distributed in the whole range of said screen.
The living fingerprint identification system according to claim 2, wherein a plurality of the slit units are distributed in a local area of the screen.
The living fingerprint identification system according to claim 7, wherein a plurality of the slit units are distributed within a testing area of the screen, and the testing area corresponds to the photosensitive module.
The living fingerprint identification system according to claim 2, wherein the screen comprises a glass cover and a light emitting unit located at the lower end of the glass cover.
The living fingerprint identification system according to claim 9, wherein the photosensitive module includes an optical assembly, the optical assembly includes an aperture and at least one lens, and the optical assembly is located on the photosensitive path of the image sensor.
The living fingerprint identification system according to claim 2, wherein the photosensitive module includes a light filtering structure, and the light filtering structure is located on the photosensitive path of the image sensor.
The living fingerprint identification system according to claim 1, wherein the image sensor comprises black and white pixels.
A living body identification method, characterized in that, comprising:

determining spectral pixels and non-spectral pixels of an image sensor;

Cast mixed light to the object being shot;

Obtaining image data of the object being shot through the non-spectral pixels, and obtaining spectral data of the object being shot through the spectral pixels; and

performing liveness detection and object recognition based on the spectral data;

Among them, the spectral pixels and non-spectral pixels of the image sensor are determined, including:

Place a low refractive index material or a test piece with high reflectivity on the glass cover of the screen;

The first light-emitting unit of the control screen emits the first pre-standard light;

receiving first spectral response data corresponding to the first light-emitting unit through an image sensor;

Extracting the physical pixels in the image sensor corresponding to the response data exceeding the preset value in the first spectral response data, and corresponding to the response data exceeding the preset value in the first spectral response data The position of the physical pixel of is recorded as the first position;

controlling the second light emitting unit of the screen to emit second pre-standard light;

receiving second spectral response data corresponding to the second light emitting unit through the image sensor;

Extracting the physical pixels in the image sensor corresponding to the response data exceeding the preset value in the second spectral response data, and corresponding to the response data exceeding the preset value in the second spectral response data The position of the physical pixel of is recorded as the second position; and

Determining that the physical pixels overlapping the physical pixels at the first position and the physical pixels at the second position in the image sensor are non-spectral pixels, and determining the physical pixels other than the non-spectral pixels in the physical pixels at the first position The pixels are spectral pixels.
A living body identification method, characterized in that, comprising:

Projecting the first detection light to the subject;

receiving the first detection light reflected back by the subject and generating first spectral information and image information of the subject based on the first detection light;

projecting second detection light to the subject;

receiving the second detected light reflected back by the subject and generating second spectral information of the subject based on the second detected light; and

Live body detection and object recognition are performed based on the first spectral information, the image information, and the second spectral information.
The living body identification method according to claim 14, wherein the first detection light is mixed light, and the second detection light is monochromatic light.
The living body recognition method according to claim 14, wherein performing living body detection and object recognition based on the first spectral information, the image information and the second spectral information includes:

processing the first spectral information and the second spectral information to generate a first spectral response result and a second spectral response result;

processing the image information to generate an image of the subject;

comparing the image of the subject with a pre-stored reference image; and

In response to successful matching between the image of the subject and the reference image, it is determined whether the subject is a living body based on the first spectral response result and/or the second spectral response result.
A spectrometer, characterized in that it comprises:

The substrate has a plurality of slit units arranged periodically for modulating incident light, and the slit units form a transmission spectrum curve corresponding thereto;

A photosensitive module, the photosensitive module is located at the lower end of the screen, and includes: an image sensor for receiving modulated incident light to obtain spectral information of the incident light, the substrate is arranged on the image sensor on the optical path.
The spectrometer according to claim 17, wherein each slit unit comprises at least one slit and/or aperture.
The spectrometer of claim 18, wherein the substrate is a screen.
The spectrometer according to claim 19, wherein the screen comprises a glass cover and a light emitting unit located under the glass cover.
The spectrometer according to claim 19, wherein the spectrometer further comprises a light source which is the light emitting unit.
The spectrometer according to claim 19, wherein the photosensitive module further includes an optical assembly including an aperture and at least one lens, and the optical assembly is located on a photosensitive path of the image sensor.
The spectrometer of claim 18, wherein the substrate is a modulation cover.
The spectrometer according to claim 23, wherein the modulation cover comprises a glass cover made of a transparent material and an opaque material covering the glass cover, and the modulation cover is not covered with the The slit unit is formed at the opaque material.
The spectrometer according to claim 24, wherein the opaque material comprises opaque conductive materials arranged in parallel, and the conductive materials arranged in parallel form a capacitor structure.
The spectrometer of claim 25, wherein said opaque material comprises an opaque, non-conductive material.
The spectrometer according to claim 26, wherein the spectrometer further comprises a circuit board electrically connected to the image sensor, the circuit board being adapted to conduct to the capacitive structure.
The spectrometer according to claim 23, wherein the modulation mask is a mask.
The spectrometer according to claim 23, wherein the modulation cover is a protective cover of electronic equipment, the protective cover has a light-transmitting area and a non-light-transmitting area, and the light-transmitting area forms the slit unit .
The spectrometer according to claim 23, wherein the photosensitive module includes a light filtering structure and an image sensor, and the light filtering structure is located on a photosensitive path of the image sensor.
The spectrometer according to claim 23, wherein said spectrometer further comprises an optical filter located on a photosensitive path of said image sensor.
The spectrometer according to claim 18, wherein any one of the slit units and its adjacent two slit units define two vectors and an area whose area is equal to the area of the two vectors, and the pattern in this area is After translating an integer number of displacements of the vectors along the vector directions corresponding to the two vectors within the periodic area, the slits in this area coincide with the slits in the area after the translation, wherein the The periodic area is an area formed by a plurality of slit units arranged in a periodic manner.
The spectrometer according to claim 32, wherein the angle between the two vectors is 90°.
The spectrometer of claim 32, wherein the periodic region has at least 25 slit elements.