CN109791325B

CN109791325B - Device with peripheral taskbar display and LCD underscreen optical sensor module for on-screen fingerprint sensing

Info

Publication number: CN109791325B
Application number: CN201880003627.9A
Authority: CN
Inventors: 何毅; 皮波
Original assignee: Shenzhen Goodix Technology Co Ltd
Current assignee: Shenzhen Goodix Technology Co Ltd
Priority date: 2017-03-07
Filing date: 2018-03-07
Publication date: 2022-02-08
Anticipated expiration: 2038-03-07
Also published as: EP3504588A4; EP3504588A1; WO2018161923A1; CN109791325A

Abstract

A device or system having a display design that includes a main display area and a peripheral display area that together form a seamless continuous display area for displaying images, content or information over both areas as a single display area and that allows the peripheral display area to be operated independently of the main display area to display only certain images, information or content on the peripheral display area even when the main display area is off. In addition to providing display functionality separate from or combined with that of the main display area, the peripheral display area based on the disclosed technology may also be used to provide certain sensing functionality by including one or more sensors under the display area of the peripheral display area.

Description

Device with peripheral taskbar display and LCD underscreen optical sensor module for on-screen fingerprint sensing

Cross reference to priority claims and related applications

This patent document claims the benefit and priority of U.S. provisional patent application No. 62/468,337 entitled "LCD underscreen optical sensor module for underscreen fingerprint sensing with peripheral taskbar display in device display" filed on 3, 7, 2017.

This patent document also claims the benefit and priority of U.S. patent application No. 15/708,088 entitled "LCD sub-screen optical sensor module for on-screen fingerprint sensing" filed 2017, 9, 18, and which is a continuation-in-part application of this U.S. patent application.

Technical Field

This patent document relates to electronic devices or systems having a display and an optical sensor module under the device display for performing one or more sensing operations, such as fingerprint measurements or other parameter measurements, based on optical sensing in an electronic device or larger system, such as a mobile device or wearable device.

Background

The display of an electronic device or system is an important part of a user interface that performs user functions and device operations, and allows information to be displayed to and operations performed by a user in various forms. Such displays may be configured as touch sensitive display screens to provide touch sensing operations as an additional interaction between a user and the electronic device and communication or interaction with others over a communication link or network.

In addition, various sensors may be implemented in an electronic device or system to provide certain desired functionality. In some designs of mobile phones, tablets and other portable devices, some sensors may be placed outside the display screen but also on the same side of the device as the display screen. Examples of some sensors in some smartphones or tablets include front-facing cameras, proximity sensors, ambient light sensors, or other types of sensors (e.g., projectors and imaging sensors for facial recognition).

Sensors for user authentication are another example of sensors for devices including portable or mobile computing devices (e.g., laptops, tablets, smartphones), gaming systems, various databases, information systems, or larger computer-controlled systems that may employ user authentication mechanisms to protect personal data and prevent unauthorized access. User authentication on an electronic device may be performed through one or more forms of biometric identifiers that may be used alone or on the basis of conventional cryptographic authentication methods. One common form of biometric identifier is a human fingerprint pattern. A fingerprint sensor may be built into an electronic device to read a fingerprint pattern of a user so that the device can only be unlocked by an authorized user of the device by authenticating the authorized user's fingerprint pattern. Another example of a sensor for an electronic device or system is a biomedical sensor, such as a heartbeat sensor in a wearable device like a wrist band device or a watch. In general, different sensors may be provided in an electronic device to achieve different sensing operations and functions.

Fingerprints may be used to authenticate a user for access to an electronic device, computer-controlled system, electronic database, or information system, which may be used as a stand-alone authentication method or in conjunction with one or more other authentication methods (e.g., password authentication methods). For example, electronic devices and gaming systems, including portable or mobile computing devices such as laptops, tablets, smartphones, and the like, may employ user authentication mechanisms to protect personal data and prevent unauthorized access. In another example, a computer or computer controlled device or system for an organization or enterprise should be protected to allow only authorized personnel access to protect information or use of the device or system of the organization or enterprise. The information stored in the portable devices and computer controlled databases, devices or systems may be essentially personal information such as personal contacts or phone books, personal photographs, personal health information or other personal information, or confidential information specific to an organization or business such as business financial information, employee data, business secrets and other proprietary information. If the security of accessing the electronic device or system is compromised, such data may be accessed by others, resulting in a loss of privacy or loss of valuable confidential information. In addition to the security of information, secure access to computers and computer-controlled devices or systems also allows for securing the use of devices or systems controlled by a computer or computer processor, such as computer-controlled automobiles and other systems such as ATMs.

Disclosure of Invention

This patent document discloses a display design for an electronic device or system that includes a main display area and a peripheral display area that together form a seamless continuous display area for displaying images, content, or information on both areas as a single display area, and that allows the peripheral display area to be operated independently of the main display area, displaying only certain images, information, or content on the peripheral display area, even when the main display area is off. In some implementations, as shown in the following examples, when the main display area and the peripheral display area are implemented based on LCD display technology, both areas may share the same backlight module, so that both areas are available when the backlight module is turned on, and in addition, a separate illumination backlight source may be provided as an additional illumination source for the peripheral display area to allow it to operate when the main display area and the backlight module for the two area sharing are turned off. In addition to providing display functionality separate from or combined with that of the main display area, the peripheral display area based on the disclosed technology may also be used to provide certain sensing functionality by including one or more sensors under the display area of the peripheral display area.

In one aspect, the technology disclosed in this patent document may be realized to construct an electronic device capable of detecting a fingerprint by optical sensing, the electronic device including: a Liquid Crystal Display (LCD) screen providing touch sensing operations and including an LCD display panel structure for displaying images, wherein the LCD display screen includes (1) a main display area having LCD display pixels and a peripheral display area having LCD display pixels, wherein the main display area and the peripheral display area together form a seamless continuous LCD display area, and (2) an LCD backlight module providing backlight that illuminates the main display area and the peripheral display area; a designated peripheral display area illumination module positioned relative to the LCD screen and configured to generate illumination light and direct the illumination light only to the peripheral display area to enable the peripheral display area to display images or information independently of the main display area and operable to display images or information when the LCD backlight module is off; a top transparent layer formed over the LCD screen as a user touch interface for touch sensing operations and as an interface for transmitting light from the LCD screen to display images or information to a user; and an optical sensor module located under the LCD display panel structure and configured to receive a detection light from an object contacting or approaching the peripheral display area and passing through the LCD screen to detect a fingerprint.

On the other hand, the technology disclosed in this patent document can be realized to construct an electronic apparatus capable of detecting a fingerprint by optical sensing, the apparatus including: a Liquid Crystal Display (LCD) display panel structure for displaying images or information and for providing touch sensing operations, the LCD display panel structure comprising an LCD layer for displaying images or information by processing illumination backlight in a main display area and a peripheral display area, the main display area and the peripheral display area together forming a seamless continuous LCD display area, a light diffuser layer for diffusing illumination backlight in the main display area and the peripheral display area, a waveguide layer for receiving illumination backlight and directing the received illumination backlight to the main display area and the peripheral display area, and a light reflector layer for redirecting illumination backlight into the LCD layers of the main display area and the peripheral display area for display operations, each of the light diffuser layer and the light reflector layer being configured to include holes or channels at selected regions in the peripheral display area of the LCD display panel structure, to allow light to be transmitted. The apparatus further comprises: an LCD backlight module coupled to the waveguide layer of the LCD display panel structure to generate backlight to the LCD layer for displaying images or information; a designated peripheral display area illumination module positioned to generate illumination light and direct illumination light only to the peripheral display area, enabling the peripheral display area to display images or information independently of the main display area and operable to display images or information when the LCD backlight module is off; a top transparent layer formed over the LCD display panel structure as a user touch interface for touch sensing operations; an optical sensor module located below the LCD display panel structure to receive probe light from the top transparent layer and through the LCD display panel structure to detect a fingerprint; and one or more detection light sources separated from the LCD backlight module and located under the top transparent layer to generate the detection light illuminating a designated fingerprint sensing area for a user to place a finger on the top transparent layer in a peripheral display area for optical fingerprint sensing by the optical sensor module.

In yet another aspect, the technology disclosed in this patent document may be realized to construct an electronic device capable of detecting a fingerprint by optical sensing, the device including a Liquid Crystal Display (LCD) screen, providing a touch sensing operation, and including an LCD display panel structure for displaying an image. The LCD display screen comprises a main display area with LCD display pixels and a peripheral display area with LCD display pixels, wherein the main display area and the peripheral display area together form a seamless continuous LCD display area. The LCD display screen may also include an LCD backlight module providing backlight for illuminating the main display area and the peripheral display area; and a designated peripheral display region illumination module that supplies illumination light only to the peripheral display region to cause the peripheral display region to display an image independently of the main display region. The apparatus further comprises: a top transparent layer formed over the device screen as a user touch interface for touch sensing operations and as an interface for transmitting light from the display structure to display an image to a user; and an optical sensor module located below the display panel structure and configured such that at least a portion of the optical sensor module is located below the peripheral display region. The optical sensor module is configured to receive detection light from an object in contact with or close to the peripheral display area and passing through the LCD screen to detect a fingerprint.

In implementing the above-described LCD display screen having (1) a main display area and (2) a peripheral display area, the control module may be coupled to the main display area, the peripheral display area, the LCD backlight module, and the designated peripheral display area illumination module, such that the designated peripheral display area illumination module is capable of illuminating the peripheral display area to display an image when the LCD backlight module is turned off. In some implementations, the control module may be configured to: the peripheral display area is controlled to display a fingerprint sensing area in the peripheral display area for a user to place a finger there for fingerprint sensing, to display a message, to display an icon allowing the user to launch an application or to perform a function by touching the icon, or to display other information.

In implementing the above apparatus, the optical sensor module may include different optical designs for directing probe light to the optical detector array for sensing. In some designs, the probe light may be projected onto the optical detector array without a lens. In some designs, one or more imaging lenses may be used to direct the probe light onto the optical detector array. In some other designs, the optical sensor module may include an optical collimator array of optical collimators for receiving the probe light and an optical detector array of optical detectors for receiving the probe light from the optical collimator array.

The following figures, descriptions, and claims provide a more detailed description of the above and other aspects and features of the disclosed technology and its implementations.

Drawings

Fig. 1A, 1B, 1C, and 1D illustrate examples of different uses of a peripheral display region of a display having a main display region and a peripheral display region to allow the peripheral display region to operate independently of the main display region to display certain images, information, or content while enabling the main display region and the peripheral display region to function as a single, seamless, continuous display region.

Fig. 2A and 2B illustrate examples of a device having an LCD display module to provide a designated peripheral display area illumination module having one or more designated illumination light sources under an LCD panel for a main display area and a peripheral display area in the same LCD panel.

Fig. 2C and 2D illustrate another example of a device having an LCD display module to provide a designated peripheral display area illumination module having one or more designated illumination light sources within an LCD panel for a main display area and a peripheral display area in the same LCD panel.

Fig. 2E and 2F show two examples of devices having LCD display modules to provide a designated peripheral display area illumination module and a designated area illumination waveguide for dispersing illumination light only to the peripheral display area for the main display area and the peripheral display area in the same LCD panel.

Fig. 3A is a block diagram of an example of a system having a fingerprint sensing module that can be implemented to include the optical fingerprint sensor disclosed in this document.

Fig. 3B and 3C illustrate one exemplary implementation of an electronic device 200, the electronic device 200 having a touch-sensing display screen assembly and an optical sensor module located below the touch-sensing display screen assembly.

Fig. 3D and 3E show examples of devices implementing the optical sensor module in fig. 3B and 3C.

Fig. 4A and 4B illustrate an example of one implementation of an optical sensor module under a display screen assembly for implementing the design in fig. 3B and 3C.

Fig. 5A, 5B and 5C illustrate signal generation for returned light from sensing regions on the top sensing surface under two different optical conditions in order to understand the operation of the underscreen optical sensor module.

Fig. 6A-6C, 7, 8A-8B, 9, and 10A-10B illustrate example designs of an underscreen optical sensor module.

Fig. 11 shows the imaging of a fingerprint sensing area on the top transparent layer by an imaging module under different laying conditions, wherein the imaging device images the fingerprint sensing area onto an optical detector array, and the imaging device may be optically transmissive or optically reflective.

Fig. 12 illustrates an example of operation of a fingerprint sensor for reducing or eliminating undesirable effects from background light in fingerprint sensing.

Fig. 13 shows a process of operating an off-screen optical sensor module for collecting a fingerprint pattern.

Fig. 14, 15 and 16 show examples of operation procedures for determining whether or not an object in contact with the LCD display screen is part of a finger of a living person by illuminating the finger with light of two different light colors.

Fig. 17A-17B, 18 and 19A-19C illustrate optical collimator designs for optical fingerprint sensing suitable for implementing the disclosed underscreen optical sensor module technology.

Fig. 20, 21A, 21B, 22A, and 22B illustrate examples of various designs for fingerprint sensing using an under-screen optical sensor module that uses an optical collimator array or a pinhole array to direct signal light carrying fingerprint information to an optical detector array.

Fig. 23 and 24 show examples of an underscreen optical sensor module having an optical collimator.

Fig. 25 shows an example of an optical collimator array that utilizes optical filtering to reduce background light reaching a photodetector array in an underscreen optical sensor module.

Fig. 26A, 26B, 27 and 28 show examples of optical collimator designs for optical sensing under an LCD display screen.

Fig. 29, 30 and 31 show improved optical imaging resolution based on pinhole machine effect in designing an optical sensor module.

Fig. 32 shows an example of an under-LCD optical sensor module using optical pinhole array for optical sensing.

Fig. 33A and 33B illustrate an example of an optical fingerprint sensor under an LCD display panel having an optical deflecting or diffracting device or layer.

Fig. 34A, 34B, and 34C show examples of LCD diffuser designs for improved optical sensing under an LCD.

Fig. 35A and 35B show examples of LCD reflector designs for improved optical sensing under an LCD.

FIG. 36 shows an example of an LCD light source design for improved optical sensing under an LCD.

Fig. 37A-37D illustrate examples of enhanced features for improved optical sensing under an LCD.

FIG. 38 shows an example of an LCD waveguide design for improved optical sensing under an LCD.

Fig. 39A to 39C show examples of LCD backlight light sources and illumination light sources for improved optical sensing under an LCD.

Fig. 40 shows two different fingerprint patterns of the same finger under different pressing forces: a lightly pressed fingerprint and a heavily pressed fingerprint.

Fig. 41A, 41B and 42 show examples of specific implementations of an optical sensor module placed under an LCD display of a device having an LCD display module to provide a main display area and a peripheral display area in the same LCD panel.

Fig. 43 shows an example of the optical transmission spectral profiles of a typical human thumb and pinky at several different optical wavelengths from about 525nm to about 940 nm.

Fig. 44 shows the influence of background light in an example of an under-screen optical sensor module.

Fig. 45 shows an example of a design algorithm for designing optical filtering in an under-screen optical sensor module to reduce the effect of background light in optical sensing.

Fig. 46 shows two examples in fig. 46A and 46B of an under-screen optical sensor module having an optical collimator array or an optical pinhole array in front of an optical detector array as part of receiving optics with a small optical numerical aperture to reduce background light entering the optical detector array.

FIG. 47 shows an example of a sensor initialization process that measures the baseline background level at the optical detector array each time a fingerprint is obtained.

Fig. 48 shows an example of a device with an optical sensor module placed under an LCD display, by using two different detection light sources at two different locations to generate detection light that is transparent to finger tissue for optical sensing.

Fig. 49 shows a specific example of implementing the detection light design in fig. 48 in the example of the apparatus in fig. 42 with an LCD display configured to provide a main display area and a peripheral display area, and to generate detection light transparent to finger tissue for optical sensing by using two different detection light sources at two different locations.

Detailed Description

This patent document discloses a display design for an electronic device or system that includes a main display area and a peripheral display area to display images, information, icons, and perform user operations. The two display areas are designed to be placed adjacent to each other such that the two together may form a seamless continuous display area displaying images, content or information as a single display area and further allowing the peripheral display area to be operated independently of the main display area to display certain images, information or content. This combination of having a main display area and a peripheral display area provides multiple applications for display operations and for user interface operations, either for a common single display area or for two separate display areas.

In addition to providing display functionality separate from or combined with that of the main display area, the peripheral display area based on the disclosed technology may also be used to provide certain sensing functionality by including one or more sensors under the display area of the peripheral display area. For example, the peripheral display area may include an optical sensing area for collecting light to be detected by an optical sensor module placed under or near the peripheral display area of the device display for optical sensing to provide optical sensing functionality, such as optical fingerprint sensing, optical sensing of other parameters, or optical sensing for determining whether a contacted object is from a living person.

For example, in a device having the underscreen optical sensing functionality disclosed in this document, the peripheral display region may include an optical sensing area for collecting returned light carrying fingerprint patterns and other information to be detected by an optical sensor module placed under or near the peripheral display region of the device display for optical sensing to provide one or more optical sensing functions including optical fingerprint sensing and optical sensing for determining whether a contacted object is from a living person, among other functions. The techniques disclosed herein based on a dual area design of adjacent main and peripheral display areas may be implemented based on a Liquid Crystal Display (LCD) panel or other display panel based on illumination light from a backlight source outside each LCD display pixel (e.g., an OLED panel based on light emitted within each organic light-emitting diode (OLED) pixel).

Fig. 1A, 1B, 1C and 1D show examples of different display modes of a smartphone with adjacent main 10 and peripheral 20 display areas, the different display modes being for different uses of the peripheral display area 20. A display having a main display area 10 and a peripheral display area 20 is designed to allow the peripheral display area 20 to operate independently of the main display area 10 to display certain images, information or content while enabling the main display area and the peripheral display area to be used together as a single seamless continuous display area. To illustrate various features of the disclosed dual area display technology, the main display area 10 and the peripheral display area 20 may be implemented by LCD panels in this example or other examples.

The displays in fig. 1A, 1B, 1C, and 1D may be LCD screens integrated with touch sensing layers to provide touch sensing operations and display images, information, and other objects. Such an LCD screen may be designed to include (1) a main display area 10 with LCD display pixels as shown in the upper larger area and (2) a peripheral display area 20 with LCD display pixels as shown in the lower smaller bar area. The two

display areas

10 and 20 are adjacent to and integrated with each other such that the main display area 10 and the peripheral display area 20 can be collectively formed and used as a single, seamless and continuous LCD display area. The peripheral display area 20 in many implementations is a display area that is smaller than the main display area 10, for example as a bottom taskbar area as shown in the specific examples of fig. 1A-1D. In the example shown, some device sensors are placed outside the two

display areas

10 and 20, but some sensors may be placed inside the two

display areas

10 and 20, for example optical sensor modules that do not require space outside the display screen.

In some implementations of such an LCD display, two illumination light modules may be provided: (1) an LCD backlight module that provides visible backlight that illuminates the LCD pixels in the main display area 10 and the peripheral display area 20, and (2) an additional designated peripheral display area illumination module that generates visible illumination only to the LCD pixels in the peripheral display area 20 for displaying images, information, and other objects independently of the main display area, including when the main display area is off. For example, a dual-zone display device may include a top transparent layer formed over the device screen as a user touch interface for touch sensing operations and as an interface for transmitting light from the display structure to display an image to a user, and, in various implementations, the top transparent layer may be a single seamless layer over two zones located on top.

In addition, a control module may be coupled to the main display area 10, the peripheral display area 20, the LCD backlight module, and the designated peripheral display area illumination module to control the operation of the two

display areas

10 and 20. For example, such a control module may control the LCD backlight module to provide backlight illumination to the two

regions

10 and 20, and control the LCD pixels in the two regions to perform various modes of the display functions shown in fig. 1A to 1D, where the peripheral display region 20 may be added to the main display region 10 as a combined single display, or display its own information independently of the main display region 10. As another example, as shown in the example of fig. 1B, the control module may enable the designated peripheral display area illumination module to illuminate the peripheral display area 20 to display an image when the LCD backlight module is off.

In many applications, the optical sensor module may be disposed in a position below the display panel structure of the display screen, and may be configured such that at least a portion of the optical sensor module is located below the peripheral display region 20, as described in detail below. Such optical sensor modules may be configured to receive probe light from an object in contact with or near the peripheral display area and passing through the LCD screen to detect a fingerprint, other biometric information associated with a finger, or other sensing function based on optical sensing. The top surface of the display screen may include a fingerprint sensing (FPS) area or region in which probe light from below the top surface is present to illuminate the finger for fingerprint sensing or for one or more other optical sensing operations. Such an FPS region can be marked as visible by displaying an image representing the boundary of the specified FPS region or displaying an image of the FPS region.

The peripheral display area 20 in the specific example is a display area smaller than the main display area 10, as a bottom taskbar area that can provide various functions, including displaying icons, images or other objects (fig. 1A and 1C), displaying information or text messages (fig. 1B), and displaying an extension of the large image displayed in the main display area, such that the two

areas

10 and 20 form a single, seamless and continuous display area (fig. 1D). This different display mode or configuration of the two

display areas

10 and 20 may be controlled by the control module.

For example, in some implementations, the control module may be used to control the peripheral display area 20 to display a fingerprint sensing (FPS) area of the peripheral display area 20 for a user to place a finger there for fingerprint sensing or other optical sensing. The FPS area of this display is one of the objects displayed in the peripheral display area 20 of FIG. 1A. The control module may be used to control the peripheral display area 20 to display an icon that allows a user to launch an application or perform a function by touching the icon. See fig. 1A and 1C. The control module may control the peripheral display area 20 to display the set of icons at different locations in the peripheral display area. As also shown in fig. 1A, different icons may be placed in the peripheral display area 20 to allow a user to select by touch or to allow a user to display additional icons by moving some or all of the currently displayed icons or objects out of the peripheral display area 20 by a sliding touch while placing additional or other icons or objects in the peripheral display area 20. This expands the functions that can be performed in the peripheral display area 20.

In some other implementations, as shown in FIG. 1B, the control module may be used to control the peripheral display area 20 to display messages or information. The displayed messages or information may be repeated identical text messages or information or scrolling different text messages or information. In the particular screen capture shown in FIG. 1B, the main display area is shown off, but in other display modes, the main display area 20 may be illuminated for display while one or more text messages or other information are displayed using the peripheral display area.

Fig. 2A shows an example of a device having an LCD display module to provide a main display area 10 and a peripheral display area 20 in the same LCD panel. This example includes a touch-sensing display system placed under a top transparent layer 431, such as a cover glass, that serves as a surface for a user interface to enable various user interface operations, including, for example: touch sensing operation by a user, display of an image to a user, and the like; and receiving the finger as an optical sensing interface for optical fingerprint sensing and other optical sensing operations, wherein probe light is directed from inside the device to the top cover glass 431 to illuminate the finger. An example of such a particular display system includes a multi-layer LCD module 433, where the multi-layer LCD module 433 includes an LCD display backlight module 434 that provides white backlight to the LCD module 433 (e.g., an edge-lit backlight configuration or an LED lamp in a backlit backlight configuration), and an optical waveguide layer 433c coupled to the LCD display backlight source 434 for receiving backlight and directing the backlight to an LCD structure layer 433a (e.g., an edge-lit backlight configuration). The LCD structure layer 433a may include, for example, a Liquid Crystal (LC) cell layer, an LCD electrode layer, a transparent conductive ITO layer, front and rear optical polarizer layers on two opposite sides of the LCD cell, a color filter layer having color filters for generating colors, and a touch sensing layer for a touch sensing operation. The LCD structure layer 433a in FIG. 2A is modified to the LCD structure layer 433f to accommodate the peripheral display area, as described below.

Multilayer LCD module 433 also includes a plurality of layers for managing backlight in an optical waveguide layer 433c from an LCD display backlight source 434, the plurality of layers including: a backlight diffuser layer 433b positioned above the light guide layer 433c and below the LCD structure layer 433f for propagating backlight through the space to illuminate LCD display pixels in the LCD structure layer 433 f; and an optical reflector mode layer 433d under the optical waveguide layer 433c for recycling the backlight to the LCD structure layer 433a to improve light utilization and display brightness.

Unlike multi-layer LCD modules in various smart phones or other devices, multi-layer LCD module 433 has two different display areas: as shown in fig. 1A to 1D, a main display area 10 and a peripheral display area 20. Under the peripheral display area 20, additional modules are provided for the peripheral display area 20: a designated peripheral display area illumination module 436b for generating illumination light as a small peripheral LCD panel area of the multi-layer LCD module 433 that illuminates the peripheral display area 20 to allow the peripheral display area 20 to operate even when the LCD display backlight light source 434 is turned off so that the main display area 10 of the multi-layer LCD module 433 is turned off due to lack of backlight. As shown in this example, designated peripheral display area illumination module 436b is positioned below multi-layer LCD module 433. With this design, illumination from a given peripheral display area illumination module 436b needs to penetrate through the optical reflector film layer 433d and the optical waveguide layer 433c and the backlight diffuser layer 433b to reach the LCD structure layer 433f where the LCD pixels are controlled to display images by filtering the received illumination. Accordingly, the optical reflector film layer 433d is configured to include a partial transmission region or window in the peripheral display region 20 to allow illumination light from the designated peripheral display region illumination module 436b to pass therethrough. As one example of achieving this goal, fig. 2A shows a micro-transparent structure 951e formed in the optical reflector film layer 433d, which may be perforated or otherwise structured to provide some optical transmission.

FIG. 2A also shows an example of an optical sensor module formed under the multi-layer LCD module 433 in the vicinity of the peripheral display region 20 to provide optical sensing on or over the top transparent layer 431. In particular, in this example, such an optical sensor module includes an optical detector array 621a of optical detectors or optical sensors (e.g., photodiodes or other optical sensing elements) for receiving and detecting optical probe light returning from the top transparent layer 431 for optical detection such as optical fingerprint sensing or other optical sensing. One or more optical probe light sources 436a are provided under the multi-layer LCD module 433 to direct probe light through the multi-layer LCD module 433 to the top transparent layer 431 to illuminate the sensing region for optical sensing in the peripheral display area 20, and the returned probe light is directed through the multi-layer LCD module 433 to the optical detector array 621 a. Accordingly, LCD layer 433f may be configured to include an area in peripheral display region 20 as a window to present a desired level of optical transmission that allows for detection light transmission, such as modified window 951c shown in LCD layer 433 f; similarly, the light diffuser layer 433b can be configured to include an area in the peripheral display region 20 that is an optical window to present a desired level of optical transmission that allows for detection of optical transmission, such as the modified window 951d shown in the light diffuser layer 433 b. One of the technical challenges of optical sensing under such an LCD is the undesired background or ambient light at the optical detector array 621a, and a suitable optical filtering design can be used in the optical path to reduce or reject such undesired background or ambient light by implementing an optical filtering film or layer on the applicable surface of the optical receiving path to the optical detector array 621 a.

Thus, the layers in the multi-layer LCD module 433 in the peripheral display region 20 in FIG. 2A are modified to accommodate for directing illumination light from the peripheral display region illumination module 436b below and designated below the LCD and to allow sufficient transmission of probe light for optical sensing below the LCD. Those areas in the light diffuser 433b and other LCD layers 433f for transmitting sufficient light may be formed in different ways including, for example, including forming a transparent aperture, using a transparent or partially transparent material, aligning the detection light path at a suitable angle, or cutting or otherwise removing scattering material (e.g., light diffuser, prism film) in that area in the peripheral display region 20.

In the example shown, a circuit board layer 623 is provided under the multi-layer LCD module 433 to support a designated peripheral display area illumination module 436b, one or more optical detection light sources 436a, and an optical detector array 621a for optical sensing in the peripheral display area 20.

The probe light source 436a may be designed to emit probe light at one or more suitable wavelengths, for example in the near infrared spectral range beginning at a longer visible red wavelength (e.g., 940nm band) that may be partially transmitted through the reflector film and liquid crystal cell polarizer at suitable angles of incidence. The detection light source 436a may be set at an optical wavelength different from that of the backlight illumination light from the light source module 434 so that the light guide function of the waveguide 433c is ineffective to the light from the illumination light source 436a so that the detection light from the detection light source 436a may more effectively pass through to the top transparent layer 431 above the LCD panel to illuminate the finger. Various optical systems may be designed for the optical sensor module under the LCD display. For example, if the receiving optics used to direct the returning probe light to the optical detector array 621a has no lenses, the probe light may be collimated. Some specific examples of optical sensor module designs are disclosed in the latter part of this patent document.

In some implementations, the designated illumination light source 436b in the peripheral display area or taskbar area 20 may be designed to provide low power illumination for displaying icons or images when the main display backlight 434 is off. The designated illumination source 436b should be in the visible spectrum, such as white light, to provide a color or monochrome display in the peripheral display area or taskbar area 20.

FIG. 2B illustrates the illumination operation of a designated illumination source 436B in the peripheral display area or taskbar area 20 wherein a micro-transparent structure 951e formed in an optical reflector film layer 433d receives taskbar illumination 957a from the designated illumination source 436B which may be controlled independently of the main display backlight source 434. The taskbar light 436b may be used to illuminate the entire area or a portion of the area of the peripheral display or taskbar area 20.

Fig. 2C shows another example of a device having an LCD display module to provide a main display area and a peripheral display area in the same LCD panel. Unlike the example in fig. 2A, one or more designated illumination light sources 436c for providing illumination light to LCD pixels in the peripheral display area 20 are positioned within the LCD module 433 and not below the LCD module 433 while still positioning the optical detector array 621a and one or more detection light sources 436a below the LCD module 433. In particular, the light source module of one or more designated illumination sources 436c may be located in the treatment area 951d of the

light diffuser

433b or 951c of the LCD layer 433f so that the resulting illuminating visible light may reach the LCD pixels in the LCD module 433 within the peripheral display region 20.

Fig. 2D illustrates the illumination operation of a designated illumination source 436c in the peripheral display area or taskbar area 20 that may be controlled independently of the main display backlight source 434. Taskbar light 436c may be used to illuminate the entire area or a portion of the area of the peripheral display or taskbar area 20.

In some implementations, a separate peripheral display area illumination waveguide may be provided as a sidebar illumination light guide to receive light from one or more taskbar

light sources

436c or 436b and disperse the light evenly in the taskbar 20. Fig. 2E shows an example of the area light guide structure 2000 as a peripheral display region illumination waveguide formed inside the LCD module 433 to cover the peripheral display region 20 of the LCD layer 433 f. The area light guide structure 2000 is separated from the LCD waveguide 433c, and the LCD waveguide 433c distributes illumination from the main backlight module 434 to the entire LCD layer 433f of the main display area 10 and the peripheral display area 20. With this design, the area light guide structure 2000 is a waveguide designed to receive illumination from the one or more taskbar light sources 436c and operate the area light guide structure 2000 to spatially disperse the received illumination from the one or more taskbar light sources 436c in a spatially more uniform manner at different LCD pixel locations in the peripheral display area 20.

The area light guide structure for illuminating the peripheral display area 20 may be placed at other locations than shown in the example of fig. 2E. And the presence of an area light guide structure can be used to provide some design flexibility in placement of one or more designated illumination sources, as long as illumination light from the output of the one or more designated illumination sources can be coupled into the area light guide structure. FIG. 2F shows another example of an area light guide structure 2000 formed inside the LCD module 433 to cover the peripheral display region 20 of the LCD layer 433F. As shown in the specific design of fig. 2F, one or more designated illumination sources 436F are located at the edge of the LCD layer 433F, not in the OCD layer 433F in the example of fig. 2E, and are optically coupled to the area optical waveguide structure 2000 located inside the LCD layer 433F. This particular arrangement of one or more designated illumination sources 436f simplifies the multi-layer construction of an LCD having independently operating peripheral display regions 20 and optical sensors under the LCD.

In the examples in fig. 2A, 2C, 2E and 2F with a main display area and a peripheral display area in the same LCD panel, the following three different illumination light sources are provided: (1) a backlight module 434 for supplying visible illumination light or white illumination light to the main display area and the peripheral display area in the same LCD panel, (2) a designated illumination light module having one or more

light sources

436b or 436c to supply designated illumination light to LCD pixels in the LCD module 433 within the peripheral display area 20, thereby allowing a display operation of the peripheral display area 20 when the backlight module 434 is turned off, and (3) a detection light module having one or more detection light sources 436a for optical sensing under the LCD. Various light illumination operations can be achieved. As another example, unlike the backlight module 434 and the designated illumination light module having one or more

light sources

436b or 436c, an illumination light detection light module having one or more detection light sources 436a for optical sensing under the LCD, like other detection light sources for optical sensing, may be designed to emit detection light for optical sensing at one or more optical wavelengths different from the LCD display illumination light wavelength, and the illumination light detection light module may be, for example, in an infrared spectrum range beyond the visible spectrum range or in the visible spectrum range, and may be used to detect different optical response characteristics from a target or finger using different detection wavelengths. The illumination sources used for optical sensing may be placed in the same general location (e.g., beside the optical sensor or below the adjacent reflector film 433 d), or at different locations where one or more illumination sources are placed below the LCD module and one or more illumination sources are placed near the edge of the LCD module below the top transparent layer 431.

The following sections describe designs and features of a display screen module and an optical sensor module below the display screen for performing fingerprint sensing and other optical sensing of a finger or object touching or near the top surface of the display screen. This under-screen optical sensing design can be used in dual-area displays having a main display area and a peripheral display area, as well as in single-area displays without a peripheral display area.

An electronic device or system may be equipped with a fingerprint authentication mechanism to improve the security of access to the device. Such electronic devices or systems may include portable or mobile computing devices such as smart phones, tablet computers, wrist-worn devices, and other wearable or portable devices, as well as larger electronic devices or systems such as personal computers in portable or desktop form, ATMs, various terminals for commercial or government use to various electronic systems, databases or information systems, and automotive, boat, train, airplane, and other motor transportation systems.

Fingerprint sensing is useful in mobile applications and other applications that use or require secure access. For example, fingerprint sensing may be used to provide secure access to mobile devices and secure financial transactions including online purchases. It is desirable to include robust and reliable fingerprint sensing suitable for mobile devices and other applications. In mobile, portable or wearable devices, it is desirable for the fingerprint sensor to minimize or eliminate the occupancy of fingerprint sensing due to the limited space on these devices, especially in view of the need for maximum display area on a given device.

The light generated by the display screen for displaying an image must pass through the top surface of the display screen in order to be seen by the user. A finger may touch the top surface to interact with light at the top surface such that light reflected or scattered at the touched surface area carries spatial image information of the finger and returns to the display panel below the top surface. In touch sensing display devices, the top surface is the touch sensing interface that engages the user, and this interaction between the light used to display the image and the user's finger or hand occurs constantly, but this information-carrying light that returns to the display panel is largely wasted and not used in most touch sensing devices. In various mobile or portable devices having touch sensing display and fingerprint sensing capabilities, fingerprint sensors tend to be separate devices from the display screen, either being disposed on the same surface of the display screen at locations other than the display screen area, such as in some models of apple iphone and samsung smartphones, or on the back of smartphones, such as smartphones of some models of hua, associative, millet or google, to avoid taking up valuable space for disposing a large display screen on the front. These fingerprint sensors are devices that are separate from the display screen and thus need to be compact to save space for display and other functions while still providing reliable and fast fingerprint sensing with a spatial image resolution above some acceptable level. However, because high spatial image resolution in acquiring fingerprint images based on various suitable fingerprint sensing technologies (e.g., capacitive touch sensing or optical imaging) requires a large sensor area with a large number of sensing pixels, the need to be compact and small and the need to provide high spatial image resolution when acquiring fingerprint patterns directly conflict with each other in many fingerprint sensors.

Sensor technology and examples of implementations of sensor technology described in this patent document provide an optical sensor module that illuminates a fingerprint dry sensing area on a touch sensing surface of a display screen and performs one or more sensing operations based on optical sensing of light from the display screen at least in part using the light as illumination probe light. A suitable display screen for implementing the disclosed optical sensor technology may be based on a variety of display technologies or configurations, including a display screen having light-emitting display pixels without the use of a backlight, where each individual pixel generates light for forming a display image on a screen such as a Liquid Crystal Display (LCD) screen, an organic light-emitting diode (OLED) display screen, or an electroluminescent display screen.

In disclosed examples for integrating optical sensing into an LCD based on the disclosed optical sensor technology, an LCD lower optical sensor may be used to detect a portion of light used to display an image in an LCD display screen, where the portion of light used for the display screen may be scattered light, reflected light, or some stray light. For example, in some implementations, image light of a backlight-based LCD screen may be reflected or scattered back into the LCD display screen as return light when encountering a target such as a user's finger or palm, or a user pointer device like a stylus. This returned light may be collected for performing one or more optical sensing operations using the disclosed optical sensor technology. Optical sensor modules based on the disclosed optical sensor technology are specifically designed to be integrated into LCD display screens due to the use of light from the LCD display screens at the time of optical sensing, wherein the integrated manner maintains the display operation and functionality of the LCD display screens without interference while providing optical sensing operation and functionality to enhance the overall functionality, device integration, and user experience of electronic devices or systems such as smartphones, tablets, or mobile/wearable devices.

Additionally, in various implementations of the disclosed optical sensing technology, one or more designated probing light sources may be provided to generate additional illuminating probing light for the optical sensing operation of the LCD sub-screen optical sensing module. In such applications, the light from the backlight of the LCD screen and the probe light from one or more designated probe light sources collectively form the illumination light for the optical sensing operation.

With respect to additional optical sensing functions in addition to fingerprint detection, optical sensing may be used to measure other parameters. For example, the disclosed optical sensor technology is capable of measuring the pattern of a human palm of a large touch area available on the entire LCD display screen (in contrast, some designated fingerprint sensors, such as those in the home button of the apple iPhone/iPad device, have a fairly small and designated off-screen fingerprint sensing area that is highly limited in the size of the sensing area, which may not be suitable for sensing large patterns). As another example, the disclosed optical sensor technology may be used not only to capture and detect patterns of fingers or palms associated with a person using optical sensing, but also to detect whether a fingerprint or palm pattern captured or detected by a "live finger" detection mechanism is from a live person's hand using optical sensing or other sensing mechanisms that may be based on, for example, different light absorption behavior of blood at different optical wavelengths, in fact, the person's fingers are typically moving or stretching due to natural movement or motion (intentional or unintentional) of the live person or pulsation as blood flows through a body connected to a heartbeat. In one implementation, the optical sensor module may detect changes in the returned light from the finger or palm due to heartbeat/blood flow changes, thereby detecting whether there is a live heartbeat in the target appearing as a finger or palm. User authentication may enhance access control based on a combination of optical sensing of a fingerprint/palm pattern and positive determination of the presence of a living person. As another example, the optical sensor module may include sensing functionality for measuring glucose levels or oxygen saturation based on optical sensing of returned light from the finger or palm. As another example, when a person touches the LCD display screen, changes in the touch force can be reflected in one or more ways, including fingerprint pattern distortion, changes in the contact area between the finger and the screen surface, fingerprint ridge broadening, or dynamic changes in blood flow. These and other variations can be measured by optical sensing based on the disclosed optical sensor technology and can be used to calculate touch force. Such touch force sensing can be used to add more functionality to the optical sensor module than fingerprint sensing.

For useful operational or control features related to touch sensing aspects of an LCD display screen, the disclosed optical sensor technology may provide a trigger function or additional function based on one or more sensing results from the optical sensor module to perform certain operations related to touch sensing control on the LCD display screen. For example, the optical properties (e.g., refractive index) of finger skin are often different from other artificial objects. The optical sensor module may hereby be designed to selectively receive and detect returned light caused by a finger in contact with the surface of the LCD display screen, while returned light caused by other objects is not detected by the optical sensor module. Such object selective optical detection can be used to provide useful user control through touch sensing, such as waking a smartphone or device only via a touch of a human finger or palm, while a touch of other objects does not cause the device to wake up, for power saving operation and prolonged battery use. Such operation may be achieved by control based on the output of the optical sensor module to control the wake-up circuit operation of the LCD display screen, wherein any touch of a human finger or palm is sensed by turning off the LCD pixels (and turning off the LCD backlight) to be in a "sleep" mode and turning on one or more illumination light sources (e.g., LEDs) of the optical sensor module under the LCD panel to be in a flash mode to intermittently flash light to the screen surface. With this design, the optical sensor module operates one or more illumination light sources to produce flickering light that wakes up sensed in a "sleep" mode, enabling the optical sensor module to detect light returned by such wake up sensed light caused by a finger touching on the LCD display, and in response to the front detection, the LCD backlight and LCD display are illuminated or "woken up". In some implementations, the wake-up sensing light may be in a spectral range that is invisible to infrared, so the user does not experience any flickering of visual light. The LCD display screen operation may be controlled to provide improved fingerprint sensing by eliminating background light for optical sensing of fingerprints. For example, in one implementation, each display scan frame generates a frame fingerprint signal. If two frames of display-related fingerprint signals are generated, where one frame of fingerprint signal is generated when the LCD display is illuminated and another frame of fingerprint signal is generated when the LCD display is extinguished, the difference between the two frames of fingerprint signals may be used to reduce the effect of ambient background light. In some implementations, by operating the fingerprint sensing frame rate to be half of the display frame rate, background light noise in fingerprint sensing can be reduced.

Optical sensor modules based on the disclosed optical sensor technology can be coupled to the back of an LCD display screen without the need to create a designated area on the surface side of the LCD display screen that would take up valuable device surface space in some electronic devices such as smartphones, tablets, or wearable devices. This aspect of the disclosed technology may be used to provide certain advantages or benefits in device design and product integration or manufacture.

In some implementations, optical sensor modules based on the disclosed optical sensor technology can be configured as non-invasive modules that can be easily integrated into a display screen without the need to change the design of the LCD display screen to provide the desired optical sensing functionality, such as fingerprint sensing. In this regard, optical sensor modules based on the disclosed optical sensor technology may be independent of the design of a particular LCD display screen design due to the following properties of the optical sensor module: optical sensing of such optical sensor modules is performed by detecting light emitted by one or more illumination sources of the optical sensor module and returning from the top surface of the display area, and the disclosed optical sensor module is coupled as an off-screen optical sensor module to the back surface of the LCD display screen for receiving the returning light from the top surface of the display area, thereby eliminating the need for a specific sensing port or sensing area separate from the display screen area. Thus, such an underscreen optical sensor module may be used in combination with an LCD display screen to provide optical fingerprint sensing and other sensor functionality on the LCD display screen without using a specially designed LCD display screen having hardware specifically designed to provide such optical sensing. This aspect of the disclosed optical sensor technology may make LCD display screens more widespread in smartphones, tablets, or other electronic devices with enhanced functionality of optical sensing from the disclosed optical sensor technology.

For example, for existing phone component designs that do not provide a separate fingerprint sensor, like certain apple (iPhone) or samsung (Galaxy) smartphones, such existing phone component designs may integrate an off-screen optical sensor module as disclosed herein without changing the touch-sensing display screen component to provide increased on-screen fingerprint sensing functionality. Because the disclosed optical sensing does not require a separate designated sensing area or port, some smartphones like some apple (iPhone)/samsung (Galaxy) phones have a front fingerprint sensor outside the display area, or some smartphones like hua shi, millet, google, or some models of association have a designated rear fingerprint sensor on the back, the integration of on-screen fingerprint sensing disclosed herein does not require substantial changes to existing phone component designs or touch-sensing display modules with touch-sensing and display layers. Based on the optical sensing technology disclosed in this document, no external sensing port and external hardware buttons are needed outside the device, which requires the addition of the disclosed optical sensor module for fingerprint sensing. The added optical sensor module and associated circuitry is under the display screen within the phone housing and can be conveniently fingerprint sensed on the same touch sensing surface of the touch screen.

As another example, due to the above-described properties of optical sensor modules for fingerprint sensing, smartphones incorporating such optical sensor modules can be upgraded with improved designs, functions and integration mechanisms without affecting or burdening the design or manufacture of LCD displays to provide desired flexibility for device manufacturing and upgrading/upgrading in the product cycle while maintaining the availability of newer versions of optical fingerprint sensing functions in smartphones, tablets or other electronic devices that use LCD displays. In particular, the touch sensing layer or LCD display layer can be updated at the next product release without any significant hardware changes to the fingerprint sensing features implemented with the disclosed off-screen optical sensor module. Also, on-screen optical sensing implemented based on such optical sensor modules for fingerprint sensing or other optical sensing functionality improvements, including the addition of additional optical sensing functionality, can be implemented by using new versions of off-screen optical sensor modules in new products without requiring significant changes to the phone component design.

The above or other features of the disclosed optical sensor technology may be implemented to provide improved fingerprint sensing and other sensing functionality to new generation electronic devices, particularly smartphones, tablets, and other electronic devices having LCD display screens, to provide various touch sensing operations and functionality, and to enhance the user experience of such devices. The features of the optical sensor module disclosed in this patent document can be applied to various display panels based on different technologies including LCD and OLED displays. The following specific examples are directed to an LCD display panel and an optical sensor module disposed under the LCD display panel.

In implementations of the disclosed technical features, additional sensing functionality or sensing module such as a biomedical sensor may be provided, for example a heartbeat sensor in a wearable device like a wrist band device or a watch. In general, different sensors may be provided in an electronic device or system to achieve different sensing operations and functions.

The disclosed technology may be implemented as devices, systems, and techniques that provide authentication that performs optical sensing of human fingerprints and access attempts to authenticate a locked computer-controlled device, such as a mobile device or computer-controlled system, equipped with a fingerprint detection module. The disclosed technology may be used to secure access to a variety of electronic devices and systems, including portable or mobile computing devices such as laptops, tablets, smart phones, and gaming devices, as well as other electronic devices or systems such as electronic databases, automobiles, bank ATMs, and the like.

Fig. 3A is a block diagram of an example of a system 180 with a fingerprint sensing module 180 comprising a fingerprint sensor 181, which may be implemented as an optical fingerprint sensor comprising optical sensing based on fingerprints disclosed in this document. The system 180 includes a fingerprint sensor control circuit 184 and a digital processor 186, which digital processor 186 may include one or more processors for processing a fingerprint pattern and determining whether the input fingerprint pattern is that of an authorized user. The fingerprint sensing system 180 uses the fingerprint sensor 181 to obtain a fingerprint and compares the obtained fingerprint to a stored fingerprint to enable or disable functions in a device or system 188 protected by the fingerprint sensing system 180. In operation, the fingerprint processor 186 controls access to the device 188 based on whether the captured user fingerprint is from an authorized user. As shown, the fingerprint sensor 181 may include a plurality of fingerprint sensing pixels, such as pixels 182A-182E that collectively represent at least a portion of a fingerprint. For example, the fingerprint sensing system 180 may be implemented at an ATM as the system 188 to determine the fingerprint of a customer requesting access to funds or other transactions. Based on a comparison of the customer's fingerprint obtained from the fingerprint sensor 181 with one or more stored fingerprints, the fingerprint sensing system 180 may cause the ATM system 188 to grant the requested access to the user account in response to a positive identification, or may deny the access in response to a negative identification. As another example, the device or system 188 may be a smartphone or portable device, and the fingerprint sensing system 180 is a module integrated into the device 188. As another example, the device or system 188 may be a door or a secure portal to a facility or home that uses the fingerprint sensor 181 to grant or deny access. As another example, the device or system 188 may be a car or other vehicle that is linked to the start of the engine using the fingerprint sensor 181 and identifies whether a person is authorized to operate the car or vehicle.

As a specific example, fig. 3B and 3C illustrate one exemplary implementation of an electronic device 200, the electronic device 200 having a touch-sensing display screen assembly and an optical sensor module located below the touch-sensing display screen assembly. In this particular example, the display technology may be implemented by an LCD display screen having a backlight for optically illuminating LCD pixels or another display screen (e.g., an OLED display screen) having light-emitting display pixels without using a backlight. The electronic device 200 may be a portable device such as a smartphone or tablet, and the electronic device 200 may be the device 188 as shown in fig. 3A.

Fig. 3B illustrates the front side of the device 200, which is similar to some features in some existing smartphones or tablets. The device screen is on the front side of the device 200, occupies all, most, or a significant portion of the front side space, and provides fingerprint sensing functionality on the device screen, such as one or more sensing areas for receiving a finger on the device screen. As an example, fig. 3B shows a fingerprint sensing area for finger touch in a device screen, which may be illuminated as a clearly identifiable region or area, where a user places a finger for fingerprint sensing. Such a fingerprint sensing region may be used to display images as the rest of the device screen. As shown, in various implementations, the device housing of device 200 may have a side that supports side control buttons that are common in various smartphones currently on the market. Also, as shown in one example of the upper left corner of the device housing in FIG. 3B, one or more optional sensors may be provided on the front side of the device 200 outside the device screen.

Fig. 3C shows an example of the structural configuration of the modules of the device 200 that are relevant for optical fingerprint sensing disclosed in this document. The device screen assembly shown in fig. 3C includes: for example, a touch-sensing screen module having a touch-sensing layer on top, and a display screen module having a display layer located below the touch-sensing screen module. An optical sensor module is coupled to and below the display screen assembly module to receive and collect return light from the top surface of the touch sensing screen module and direct and image the return light onto an optical detector array of optical sensing pixels or photodetectors that convert the optical image in the return light into pixel signals for further processing. Beneath the optical sensor module is a device electronics structure that contains certain electronic circuitry for the optical sensor module and other components in device 200. The device electronics may be disposed inside the device housing and may include a portion of the underside of the optical sensor module as shown in fig. 3C.

In implementations, the top surface of the device screen assembly may be a surface of an optically transparent layer that serves as a user touch sensing surface to provide a variety of functions, such as (1) a display output surface through which light carrying a display image passes to the eyes of a viewer, (2) a touch sensing interface that receives user touch for touch sensing operations of the touch sensing module, and (3) an optical interface for on-screen fingerprint sensing (and possibly one or more other optical sensing functions). Such optically transparent layers may be rigid or flexible layers such as glass or crystalline layers.

One example of a display screen is an LCD display having LCD layers and a Thin Film Transistor (TFT) structure or substrate. The LCD display panel is a multi-layer Liquid Crystal Display (LCD) module that includes LCD display backlight light sources (e.g., LED lamps) that emit LCD illumination light for the LCD pixels, an optical waveguide layer that guides the backlight, and LCD structure layers that may include, for example, a Liquid Crystal (LC) cell layer, a transparent conductive ITO layer, an optical polarizer layer, a filter layer, and a touch sensing layer. The LCD module also includes a backlight diffuser below the LCD structural layer and above the light guide layer for spatially propagating backlight for illuminating the LCD display pixels, and an optical reflector film layer below the light guide layer for recycling backlight to the LCD structural layer to improve light utilization and display brightness.

Referring to FIG. 3C, the optical sensor module in this example is located below the LCD display panel to collect the returned light from the top touch sensing surface and acquire a high resolution image of the fingerprint pattern when the user's finger is in contact with the sensing area on the top surface. In other implementations, the disclosed under-screen optical sensor module for fingerprint sensing can be implemented on a device without touch sensing features. In addition, suitable display panels may have various screen designs other than OLED displays.

Fig. 3D and 3E show examples of devices implementing the optical sensor module in fig. 3B and 3C. Fig. 3D shows a cross-sectional view of a portion of a device containing an underscreen optical sensor module. Fig. 3D shows a view of the front side of a device with a touch-sensing display on the left side, representing a fingerprint sensing area on the lower part of the display screen, and a perspective view of a part of the device containing an optical sensor module located below the device display screen assembly on the right side. Fig. 3D also shows an example of a layout of a flex tape with circuit elements.

In the design examples of fig. 3B, 3C, 3D, and 3E, the optical fingerprint sensor design differs from some other fingerprint sensor designs that use a fingerprint sensor structure that is separate from the display screen and that has a physical demarcation between the display screen and the fingerprint sensor on the surface of the mobile device (e.g., button-like structures in the opening of the top glass cover plate in some mobile phone designs). In the design shown herein, the optical fingerprint sensor for detecting fingerprint sensing and other optical signals is located below the top cover glass or layer (e.g., fig. 3D), such that the top surface of the cover glass serves as the top surface of the mobile device as a continuous and uniform glass surface across the vertically stacked and vertically overlapping display screen layer and optical detector sensors. This design for integrating optical fingerprint sensing and a touch sensitive display screen under a common and uniform surface provides benefits including improved device integration, enhanced device packaging, enhanced device resistance to external elements, failure, wear and tear, and enhanced user experience during ownership of the device.

Referring back to fig. 3B and 3C, the illustrated off-screen optical sensor module for on-screen fingerprint sensing may be implemented in various configurations.

In one implementation, a device based on the above design may be configured to include a device screen that provides touch sensing operations and includes an LCD display panel structure that forms a display image, the device further including a top transparent layer formed over the device screen as a user touch interface for the touch sensing operations and as an interface for transmitting light from the display structure to display the image to a user, the device further including an optical sensor module located below the display panel structure to receive light returned from the top transparent layer to detect a fingerprint.

Such devices and other devices disclosed in this document may also be configured to include various features.

For example, a device electronic control module may be included in the device to authorize a user's access to the device when a detected fingerprint matches an authorized user's fingerprint. Additionally, the optical sensor module is further configured to detect, in addition to the fingerprint, a biometric parameter other than the fingerprint by optical sensing to indicate whether a touch associated with the detected fingerprint at the top transparent layer is from a living person, and the device electronic control module is configured to authorize user access to the device if (1) the detected fingerprint matches the fingerprint of the authorized user and (2) the detected biometric parameter indicates that the detected fingerprint is from a living person. The biometric parameter may include, for example, whether the finger contains a person's blood flow or heartbeat.

For example, the device may include a device electronic control module coupled to the display panel structure to provide power to the light emitting display pixels and control the display of images by the display panel structure, and in a fingerprint sensing operation, the device electronic control module operates to turn off the light emitting display pixels in one frame and turn on the light emitting display pixels in a next frame to allow the optical detector array to capture two fingerprint images with and without illumination of the light emitting display pixels to reduce the effect of background light in fingerprint sensing.

As another example, the device electronic control module may be coupled to the display panel structure to provide power to the light-emitting LCD display panel and to turn off power to a backlight of the LCD display panel in the sleep mode, and the device electronic control module may be configured to wake up the display panel structure from the sleep mode when the optical sensor module detects the presence of the skin of the person at the designated fingerprint sensing area of the top transparent layer. More specifically, in some implementations, the device electronic control module may be configured to operate one or more illumination light sources in the optical sensor module to intermittently emit light while the LCD display panel is powered off (in a sleep mode), directing the intermittently emitted illumination light to designated fingerprint sensing areas of the top transparent layer to monitor the presence or absence of human skin in contact with the designated fingerprint sensing areas for waking the device from the sleep mode.

As another example, the device may include a device electronic control module coupled to the optical sensor module to receive information of a plurality of detected fingerprints obtained by sensing a touch of the finger, and operated to measure changes in the plurality of detected fingerprints and determine a touch force causing the measured changes. For example, the plurality of detected variations of the fingerprint may include variations of an image of the fingerprint due to a touch force, variations of a touch area due to a touch force, or variations of a pitch of fingerprint ridges.

As another example, the top transparent layer may include a designated fingerprint sensing area for a user to touch with a finger for fingerprint sensing, and the optical sensor module below the display panel structure may include a transparent block in contact with the display panel substrate to receive light emitted from the display panel structure and returning from the top transparent layer, the optical sensor module may further include an optical detector array to receive the light, and an optical imaging module to image the light received in the transparent block onto the optical detector array. The optical sensor module may be positioned relative to a designated fingerprint sensing area and configured to: the return light by total internal reflection at the top surface of the top transparent layer is selectively received when in contact with the skin of the person, and the return light from the designated fingerprint sensing area is not received when there is no contact with the skin of the person.

As another example, an optical sensor module may be configured to include an optical wedge positioned below a display panel structure to modify a total reflection condition on a bottom surface of the display panel structure engaged with the optical wedge to allow light extracted from the display panel structure through the bottom surface, an optical detector array to receive light extracted from the optical wedge from the display panel structure, and an optical imaging module positioned between the optical wedge and the optical detector array to image light from the optical wedge onto the optical detector array.

Specific examples of an off-screen optical sensor module for on-screen fingerprint sensing are provided below.

Fig. 4A and 4B show an example of one implementation of an optical sensor module underneath a display screen assembly for implementing the design in fig. 3B and 3C. The device in fig. 4A-4B includes a display assembly 423 having a top transparent layer 431 formed over a device screen assembly 432 as a user touch interface for touch sensing operations and as an interface for transmitting light from a display structure to display an image to a user. In some implementations, the top transparent layer 431 can be a cover glass or a crystalline material. The device screen assembly 423 may include an LCD display module 433 below a top transparent layer 431. The LCD display layer allows partial light transmission such that light from the top surface can partially pass through the LCD display layer to the under-LCD optical sensor module. For example, the LCD display layer includes electrodes and wiring structures that optically act as an array of apertures and light scattering targets. A device circuit module 435 may be provided under the OLED display panel to control the operation of the device and perform functions for a user to operate the device.

The optical sensor module 702 in this particular implementation example is located under the LCD display module 433. One or more illumination sources, such as an illumination source 436 under the LCD display module 433 or/and another one or more illumination sources under the top cover glass 431, are provided for providing illumination or detection light for optical sensing by the optical sensing module 702, and may be controlled to emit light to illuminate the fingerprint sensing region 615 at least partially through the LCD display module onto the top transparent layer 431 within the area of the screen of the device for a user to place a finger therein for fingerprint recognition. Illumination light from one or more illumination light sources 436 may be directed toward the fingerprint sensing region 615 on the top surface as if the illumination light were from the fingerprint illumination light zone 613. Another illumination source or sources located below the top cover glass 431 may be placed near the fingerprint sensing area 615 on the top surface to direct the generated illumination to the top cover glass 433 without passing through the LCD display module 433. As shown, a finger 445 is placed in an illuminated fingerprint sensing region 615, the fingerprint sensing region 615 acting as an active sensing region for fingerprint sensing. A portion of the reflected or scattered light in the fingerprint sensing region 615 is directed into the optical sensor module under the LCD display module 433, and a photodetector sensing array within the optical sensor module receives this light and collects fingerprint pattern information carried by the received light.

In such designs that use one or more light sources (e.g., 436) to provide illumination light for optical fingerprint sensing, in some implementations, each illumination light source 436 may be controlled to turn on intermittently at relatively slow periods to reduce power for optical sensing operations. In some implementations, the fingerprint sensing operation can be implemented in a two-step process: first, one or more light sources 436 are turned on in a flash mode without turning on the LCD display panel to sense whether a finger touches the sensing region 615 using a flickering light, and then, once a touch in the region 615 is detected, the optical sensing module is operated to perform fingerprint sensing based on optical sensing and the LCD display panel may be turned on.

In the example of fig. 4B, the underscreen optical sensor module includes a transparent block 701 coupled to the display panel, the transparent block 701 receiving returned light from the top surface of the device component, the underscreen optical sensor module further including an optical imaging block 702 performing optical imaging and imaging acquisition. Light from the illumination source 436 is reflected or scattered back from the cover top surface after reaching the cover top surface, e.g., the cover top surface at the sensing area 615 where the user's finger touches. When the top surface of the cover plate in the sensing area 615 is in close contact with the fingerprint ridge, the light reflection under the fingerprint ridge is different from the light reflection at another location under the fingerprint valley where there is no skin or tissue of the finger due to the presence of the skin or tissue of the finger in contact at that location. This difference in light reflection conditions at the location of the ridges and valleys in the area touched by the finger on the top surface of the cover sheet forms an image representing the image or spatial distribution of the ridges and valleys of the touched portion of the finger. The reflected light is directed back toward the LCD display module 433 and, after passing through the aperture of the LCD display module 433, reaches the interface of the low index optically transparent block 701 of the optical sensor module. The refractive index of the low index optically transparent block 701 is configured to be less than the refractive index of the LCD display panel so that the returned light can be extracted from the LCD display panel into the optically transparent block 701. Once the returned light is received within the optically transparent block 701, this received light enters an optical imaging unit that is part of the imaging sensing block 702 and is imaged onto a photodetector sensing array or optical sensing arrangement within block 702. The difference in light reflection between the ridges and valleys of the fingerprint causes a contrast in the fingerprint image. As shown in fig. 4B, a control circuit 704 (e.g., a microcontroller or MCU) is coupled to the imaging sensing block 702 and other circuitry such as a device main processor 705 on the main circuit board.

In this particular example, the optical path design is: light rays enter the cover plate top surface within the total reflection angle on the top surface between the substrate and air interface and will be most efficiently collected by the imaging optics and imaging sensor array in block 702. In this design, the image of the fingerprint ridge/valley area exhibits the greatest contrast. Such imaging systems may have undesirable optical distortions that can adversely affect fingerprint sensing. Thus, upon processing the output signals of the optical detector array in block 702, the acquired image may also be corrected by distortion correction during imaging reconstruction based on the optical distortion at the optical detector array along the optical path of the returned light. By scanning the test image pattern of one row of pixels at a time over the entire sensing area of the X-direction lines and the Y-direction lines, distortion correction coefficients may be generated from the pattern acquired at each photodetector pixel. This correction process may also use images from tuning a single pixel at a time and scanning the entire image area of the photodetector array. This correction factor only needs to be generated once after assembly of the sensor.

Ambient light from the environment (e.g., sunlight or room light) can enter the image sensor through the hole in the LCD display assembly 433 through the top surface of the LCD panel. This background light can produce a background baseline in the valuable image from the finger, and this background baseline is undesirable. Different methods may be used to reduce this baseline intensity. One example is to tune the illumination source 436 on and off at a frequency f, and the image sensor acquires the received image at the same frequency by phase synchronizing the source drive pulses with the image sensor frame. In this operation, only one of the image phases contains light from the light source. By subtracting the odd and even frames it is possible to obtain an image which is mostly composed of the light emitted by the modulated illumination source. Based on the design, each display scan frame generates a frame of the fingerprint signal. If two consecutive signal frames are removed by tuning the illumination light on in one frame and the Zhao-Ming light off in another frame, the ambient background light effects can be minimized or largely eliminated. In an implementation, the fingerprint sensing frame rate may be half of the display frame rate.

A portion of the light from the illumination source 436 may also pass through the cover top surface and into the finger tissue. This portion of the optical power is scattered around, and a portion of this scattered light can eventually be collected by the imaging sensor array in the optical sensor module. The light intensity of this scattered light depends on the skin tone of the finger and the blood concentration in the finger tissue, and this information carried by this scattered light on the finger is useful for fingerprint sensing and can be detected as part of the fingerprint sensing operation. For example, by integrating the intensity of the areas of the user's finger image, it may be observed that the increase/decrease of blood concentration depends on the phase of the user's heartbeat. Such features may be used to determine the user's heart rate, determine whether the user's finger is a live finger, or a spoofing device with a spoofed fingerprint pattern.

One or more of the illumination sources 436 in fig. 4B may be designed to emit light of different colors or wavelengths, and the optical sensor module may collect the returned light of different colors or wavelengths from the human finger. By recording the intensity of the returned light of the respective measured different colors or wavelengths, information associated with the user's skin tone can be determined. For example, when the user registers a finger for fingerprint authentication operation, the optical fingerprint sensor also measures the intensity of scattered light from the colors a and B of the finger as the intensities Ia and Ib. The ratio Ia/Ib may be recorded for comparison with later measurements of the measured fingerprint when the user's finger is placed on the sensing area. The method may help to reject spoofed devices that may not match the user's skin tone.

One or more illumination sources 436 may be controlled by the same electronics 704 (e.g., MCU) used to control the image sensor array in block 702. The one or more illumination light sources 436 may be pulsed for short periods of time at low duty cycles to intermittently emit light and provide pulsed light for image sensing. The image sensor array may be operated to monitor the light pattern at the same pulse duty cycle. If there is a human finger touching sensing area 615 on the screen, the image acquired at the imaging sensing array in block 702 may be used to detect a touch event. The control electronics or MCU 704 connected to the image sensor array in block 702 can be operable to determine if the touch is a human finger touch. If a human finger touch event is determined, MCU 704 may be operable to wake up the smartphone system, turn on illumination source 436 for performing optical fingerprint sensing, and acquire a complete fingerprint image using the normal mode. The image sensor array in block 702 may send the acquired fingerprint image to the smartphone host processor 705, and the smartphone host processor 705 may be operable to match the acquired fingerprint image to a registered fingerprint database. If the matching exists, the smart phone unlocks the mobile phone and starts normal operation. If the acquired image is not matched, the smart phone feeds back the authentication failure to the user. The user may try again, or enter a password.

In the example of fig. 4A and 4B, the sub-screen optical sensor module optically images a fingerprint pattern of a touching finger in contact with the top surface of the display screen on the photodetector sensing array using an optically transparent block 701 and an imaging sensing block 702 having the photodetector sensing array. The optical imaging or detection axis 625 is shown in FIG. 4B from the sensing region 615 to the photodetector array in block 702. The optically transparent block 701 and the front end of the imaging sensing block 702 before the photodetector sensing array form a volume imaging module to enable suitable imaging for optical fingerprint sensing. Due to optical distortion in the imaging process, as explained above, distortion correction can be used to achieve a desired imaging operation.

In the optical sensing disclosed herein by the underscreen optical sensor module in fig. 4 and 4B and other designs, the optical signal from the sensing region 615 on the top transparent layer 431 to the underscreen optical sensor module includes different light components. Fig. 5A, 5B, and 5C illustrate signal generation for light returning from sensing region 615 under two different optical conditions in order to understand the operation of the underscreen optical sensor module.

Fig. 5A shows how the illumination light from the illumination light source 436 propagates through the OLED display module 433 and, after passing through the top transparent layer 431, generates different return light signals to the underscreen optical sensor module, including light signals carrying fingerprint pattern information. For simplicity, the two

illumination rays

80 and 82 at different locations are directed to the top transparent layer 431 without undergoing total reflection at the interface with the top transparent layer 431. Specifically, illumination rays 80 and 82 are perpendicular or near perpendicular to top layer 431. Finger 60 is in contact with sensing region 615 on top transparent layer 431. As shown, the illumination beam 80, after passing through the top transparent layer 431, reaches the finger ridge in contact with the top transparent layer 431 to generate a beam 183 in the finger tissue and another beam 181 back to the LCD display module 433. The illumination beam 82, after passing through the top transparent layer 431, reaches the finger valley above the top transparent layer to generate a beam 185 returning from the interface of the top transparent layer 431 towards the LCD display module 433, a second beam 189 entering the finger tissue, and a third beam 187 reflected by the finger valley.

In the example of fig. 5A, it is assumed that the equivalent index of refraction of the finger skin at 550nm is about 1.44, and the cover glass index of refraction of the top transparent layer 431 is about 1.51. The finger ridge-cover glass interface reflects a portion of the light of beam 80 as reflected light 181 to a bottom layer 524 below the LCD display module 433. In some LCD panels, the reflectivity is low, for example, about 0.1%. Most of the light in beam 80 becomes beam 183 that is transmitted into finger tissue 60, and finger tissue 60 causes scattering of light 183, producing scattered light 191 that returns to LCD display module 433 and bottom layer 524. The scattering of transmitted light beam 189 from LCD pixel 73 in the finger tissue also contributes to the returning scattered light 191.

The cover glass surface reflects about 3.5% (light 185) of the power of the light beam 82 at the finger skin valley location 63 to the bottom layer 524, and the finger valley surface reflects about 3.3% (light 187) of the incident light power to the bottom layer 524. The total reflectance was about 6.8%. Most of the light 189 is transmitted into the finger tissue 60. A portion of the optical power in the transmitted light 189 in the finger tissue is scattered by the finger tissue, contributing to the scattered light 191 toward and into the bottom layer 524.

Thus, the reflection of light from the various interfaces or surfaces at the valleys and ridges of the finger touching the finger is different, the reflectance difference carries the fingerprint map information, and can be measured to extract the fingerprint pattern of the portion in contact with the top transparent layer 431 and illuminated by the OLED light.

Fig. 5B and 5C show the optical paths of two additional illuminating light rays at the top surface under different conditions and at different positions relative to the valleys or ridges of the finger, including the optical path at the interface with the top transparent layer 431 under total reflection conditions. The illustrated illumination light generates different return light signals, including light signals that carry fingerprint pattern information to the off-screen optical sensor module. It is assumed that the cover glass 431 and the LCD display module 433 are bonded together without any air gap therebetween, so that O illumination light having a large incident angle to the cover glass 431 will be totally reflected at the cover glass-air interface. 5A, 5B, and 5C show examples of three different sets of diverging beams: (1) the center beam 82, which has a small incident angle to the cover glass 431 and is not totally reflected (fig. 5A), (2) the high contrast beams 201, 202, 211, and 212, which are totally reflected at the cover glass 431 when the cover glass surface is not touched and can be coupled into the finger tissue when the finger touches the cover glass 431 (fig. 5B and 5C), and (3) the escape beam, which has a large incident angle, is totally reflected at the cover glass 431 even at the position where the finger tissue touches.

For the center beam 82, the cover glass surface reflects about 0.1% to about 3.5% of the light toward beam 185 and this portion is transmitted into the bottom layer 524, and the finger skin reflects about 0.1% to about 3.3% of the light toward beam 187 and this portion is also transmitted into the bottom layer 524. The difference in reflection depends on whether the light beam 82 meets the finger skin ridge 61 or valley 63. The remaining light beam 189 is coupled into finger tissue 60.

For high contrast beams 201 and 202, if the cover glass surface is not touched, the cover glass surface reflects nearly 100% of the light to

beams

205 and 206, respectively. When the finger skin ridge touches the surface of the reprographic glass and at the location of

beams

201 and 202, most of the optical power is coupled into the finger tissue 60 by

beams

203 and 204.

For high contrast light beams 211 and 212, the cover glass surface reflects nearly 100% of the light to

light beams

213 and 214, respectively, if the cover glass surface is not touched. When a finger touches the cover glass surface and the finger skin valley happens to be at the location of

light beams

211 and 212, no optical power is coupled into the finger tissue 60.

As shown in fig. 5A, the light beam coupled into the finger tissue 60 may experience random scattering of the finger tissue to form low contrast light 191, and a portion of the low contrast light 191 may pass through the LCD display module to the optical sensor module.

Thus, in areas illuminated by the high contrast light beam, the finger skin ridges and valleys cause different optical reflections, and the reflection difference pattern carries fingerprint pattern information. High contrast fingerprint signals can be achieved by comparing such differences.

Based on the designs in fig. 3B and 3C, the disclosed underscreen optical sensing techniques may optically acquire fingerprints in various configurations.

For example, the specific implementation in fig. 4B may be implemented in various configurations based on optical imaging through the use of a volume imaging module in the optical sensing module. Fig. 6A-6C, 7, 8A-8B, 9, 10A-10B, 11, and 12 illustrate examples of various implementations, additional features, and operations of an off-screen optical sensor module design for optical fingerprint sensing.

Fig. 6A, 6B and 6C show an example of an off-screen optical sensor module based on optical imaging by a lens for picking up a fingerprint from a finger 445 pressing a display cover glass 423. Fig. 6C is an enlarged view of the optical sensor module portion shown in fig. 6B. The under-screen photosensor module shown in FIG. 6B, which includes an optically transparent spacer 617 in contact with the bottom surface of the LCD display module 433 to receive returning light from the sensing region 615 on the top surface of the top transparent layer 431, is located under the LCD display module 433, and includes an imaging lens 621 located between the spacer 617 and the photodetector array 623, the imaging lens 621 imaging the received returning light from the sensing region 615 onto the photodetector array 623. Similar to the imaging system in the example of fig. 4B, the imaging system for the optical sensor module in fig. 6B may experience image distortion, and suitable optical correction calibrations may be used to reduce such distortion, e.g., the distortion correction methods described for the system in fig. 4B.

Similar to the assumptions in fig. 5A, 5B, and 5C, it is assumed that the equivalent refractive index of the finger skin at 550nm is about 1.44, and for the cover glass 423, the refractive index of the bare cover glass is about 1.51. When the OLED display module 433 is bonded on the cover glass 431 without any air gap, total internal reflection occurs at a large angle greater than the critical incident angle of the interface. The total reflection incident angle is about 41.8 ° if the cover glass top surface is not touched, and about 73.7 ° if the finger skin touches the cover glass top surface. The corresponding difference in total reflection angle is about 31.9 °.

In this design, microlens 621 and photodiode array 623 define a viewing angle θ that captures an image of a touching finger in sensing region 615. To detect the desired portion of the cover glass surface in sensing region 615, the viewing angle can be appropriately aligned by controlling a physical parameter or configuration. For example, the viewing angle may be aligned to detect total internal reflection of the LCD display assembly. Specifically, active sensing region 615 on the cover glass surface is sensed aligned with viewing angle θ. The effectively sensed cover glass surface 615 may be considered a mirror such that the photodetector array effectively detects an image of the fingerprint illumination photo year module 613 in the LCD display that is projected onto the photodetector array by the sensed cover glass surface 615. The photodiode/photodetector array 623 can receive an image of the region 613 reflected by the sensed cover glass surface 615. When a finger touches sensing region 615, a portion of the light may couple into the ridge of the fingerprint, which causes the photodetector array to receive light from the ridge location to appear as a darker fingerprint image. Since the geometry of the optical detection path is known, distortions of the fingerprint image caused in the optical path in the optical sensor module can be corrected.

As a specific example, considering that the distance H from the central axis of the detection module to the top surface of the cover glass in fig. 6B is 2mm, this design can directly cover a 5mm active sensing region 615, which active sensing region 615 has a width Wc on the cover glass. Adjusting the thickness of spacer 617 adjusts detector position parameter H and optimizes effective sensing zone width Wc. Since H includes the thickness of the cover glass 431 and the display module 433, the application design should take these layers into account. The pad 617, microlens 621 and photodiode array 623 may be integrated under the color coating 619 on the bottom surface of the top transparent layer 431.

Fig. 7 illustrates an example of another design consideration for the optical imaging design of the optical sensor module shown in fig. 6-6C, by using special spacer 618 instead of spacer 617 in fig. 6B-6C to increase the size of sensing region 615. The spacer 618 is designed to have a width Ws and a thickness Hs with a low Refractive Index (RI) ns, and the spacer 618 is located under the LCD display module 433, e.g., attached (e.g., glued) to the bottom surface of the LCD display module 433. The end face of the spacer 618 is an angled or slanted face that engages the microlens 621. This relative position of the pads and lenses is different from the lens being located below the pad 617 in fig. 6B-6C. The microlens 621 and the photodiode array 623 are assembled into an optical detection module having a detection angle size θ. The axis 625 bending is detected due to optical refraction at the interface between the spacer 618 and the display module 433 and at the interface between the cover glass 431 and the air. The local angles of incidence φ 1 and φ 2 are determined by the refractive indices of the component materials RI, ns, nc, and na.

If nc is greater than ns, then φ 1 is greater than φ 2. Thereby, the sensing width Wc is magnified by refraction. For example, assuming that the equivalent refractive index RI of the finger skin is about 1.44 at 550nm and the refractive index RI of the cover glass is about 1.51, the total reflection incident angle is estimated to be about 41.8 ° if the top surface of the cover glass is not touched, and the total reflection angle is about 73.7 ° if the finger skin touches the top surface of the cover glass. The corresponding difference in total reflection angle is about 31.9 °. If the spacer 618 is made of the same material as the cover glass, the distance from the center of the detection module to the top surface of the cover glass is 2mm, and if the detection angle θ is 31.9 °, the effective sensing area width Wc is about 5 mm. The local incident angle of the corresponding central axis is 57.75 ° with Φ 1 ═ Φ 2. If the material of the special spacer 618 has a refractive index ns of about 1.4 and Hs of 1.2mm, the detection module is tilted at phi 1-70 deg.. The effective sensing area width is increased to greater than 6.5 mm. Under these parameters, the detection angular width in the cover glass is reduced to 19 °. Thus, the imaging system of the optical sensor module may be designed with the desire to enlarge the size of the sensing region 615 on the top transparent layer 431.

When the refractive index RI of a particular spacer 618 is designed to be sufficiently low (e.g., MgF2, CaF2, or even air is used to form the spacer), the width Wc of the active sensing region 615 is no longer limited by the thickness of the cover glass 431 and the display module 433. This property provides the desired design flexibility. In principle, the effective sensing area may even be increased to cover all display screens if the detection module has sufficient resolution.

Because the disclosed optical sensor technology can be used to provide a large sensing area to capture patterns, the disclosed underscreen optical sensor module can be used to capture and detect not only patterns of fingers, but also patterns of larger sizes, such as the palm of a person's hand associated with a person, for user authentication.

Fig. 8A to 8B show an example of another design consideration for the optical imaging design of the optical sensor module shown in fig. 7, in which the opposing detection angle θ' of the photodetector array in the display screen surface and the distance L between the lens 621 and the spacer 618 are set. Fig. 8A shows a cross-sectional view in a direction perpendicular to the surface of the display screen, and fig. 8B shows a view of the device as seen from the bottom or top of the display screen. A fill material 618c may be used to fill the space between the lens 621 and the photodetector array 623. For example, the filler material 618c may be the same material as the particular shim 618 or another different material. In some designs, the filler material 618c may be an air space.

Fig. 9 shows another example of an underscreen optical sensor module in the design based on fig. 7, wherein one or more illumination light sources 61 are provided to illuminate a top surface sensing region 615 for optical fingerprint sensing. The illumination source 614 may be an extended type or a collimated type of light source such that all points within the active sensing region 615 are illuminated. The illumination source 614 may be a single element light source or an array of light sources.

Fig. 10A-10B illustrate an example of an underscreen optical sensor module using an optical coupler 628 shaped as a thin wedge to improve optical detection at an optical detector array 623. Fig. 10A shows a cross-section of a device structure with an off-screen optical sensor module for fingerprint sensing, and fig. 10B shows a top view of the device screen. Optical wedge 628 (having refractive index n)_s) Under the display panel structure to modify the total reflection conditions at the bottom surface of the display panel structure engaging the optical wedge 628 to allow light to be extracted from the display panel structure through the bottom surface. An optical detector array 623 receives light extracted from the display panel structure from the optical wedge 628, and an optical imaging module 621 is located between the optical wedge 628 and the optical detector array 623 to image the light from the optical wedge 628 onto the optical detector array 623. In the example shown, wedge 628 comprises an angled wedge surface facing the optical imaging module and the optical sensing array 623. Also, as shown, there is free space between wedge 628 and optical imaging module 621.

If the light is totally reflected at the sensing surface of the cover glass 431, the reflectivity is 100%, with the highest efficiency. However, if the light is parallel to the cover glass surface, the light is also totally reflected at the LCD bottom surface light diffuser layer 433 b. The wedge coupler 628 is used to modify the local surface angle so that light can be coupled out for detection at the optical detector array 623. The micro-holes in the LCD display module 433 provide a desired light propagation path that allows light to pass through the LCD display module 433 for underscreen optical sensing. The actual light transmission efficiency may gradually decrease if the light transmission angle becomes too large or when the TFT layer becomes too thick. When the angle is close to the angle of total reflection, i.e., about 41.8 °, and the cover glass has a refractive index of 1.5, the fingerprint image looks good. Accordingly, the wedge angle of the wedge coupler 628 may be adjusted to a logarithmic degree, so that the detection efficiency is improved or optimized. If a higher refractive index of the cover glass is selected, the total reflection angle becomes smaller. For example, if the cover glass is made of sapphire having a refractive index of about 1.76, the total reflection angle is about 34.62 °. The efficiency of detecting light transmission in the display is also improved. Therefore, this design uses a thin wedge to set the detection angle higher than the total reflection angle, and/or uses a cover glass material with a high refractive index to improve the detection efficiency.

In the underscreen optical sensor module in fig. 6A-10B, the sensing area 615 on the top transparent surface is not vertical or perpendicular to the detection axis 625 of the optical sensor module, such that the image plane of the sensing area is also not vertical or perpendicular to the detection axis 625. Accordingly, the plane of the photodetector array 523 may be tilted with respect to the detection axis 625 to achieve high quality imaging at the light detection array 623.

Fig. 11 shows three example configurations of such tilting. Fig. 11(1) shows that the sensing region 615a is inclined and not perpendicular to the detection axis 625. In the particular case shown in (2), the sensing region 615b is aligned on the detection axis 625, and its image plane would also lie on the detection axis 625. In practice, the lens 621 may be partially cut away to simplify packaging. In various implementations, the microlenses 621 may also be transmissive or reflective lenses. For example, a specific route is shown in (3). Sensing region 615c is imaged by imaging mirror 621 a. The photodiode array 623b is aligned to detect a signal.

In the above-described design using lens 621, the effective aperture of lens 621 can be designed to be larger than the aperture of the hole in the LCD display layer, which allows light to pass through the LCD display module for optical fingerprint sensing. Such a design may reduce the resulting undesirable effects of wiring structures and other scattering objects in the LCD display module.

Fig. 12 illustrates an example of operation of a fingerprint sensor to reduce or eliminate undesirable effects from background light in fingerprint sensing. The optical sensor array may be used to acquire various frames, and the acquired frames may be used to perform a difference and average operation between multiple frames to reduce the effects of background light. For example, in frame a, the illumination source for optical fingerprint sensing is turned on to illuminate the area touched by the finger, and in frame B, the illumination is changed or turned off. The subtraction of the signal of frame B from the signal of frame a may be used in image processing to reduce unwanted background light effects.

It is also possible to reduce unwanted background light at the time of fingerprint sensing by providing suitable optical filtering in the light path. One or more optical filters may be used to reject ambient light wavelengths, e.g., near infrared IR light and portions of red light, etc. In some implementations, such optical filter coatings can be fabricated on surfaces of optical components, including the display bottom surface, the prism surface, or the sensor surface, among others. For example, a human finger absorbs most of the energy at wavelengths below-580 nm, and the effect of ambient light on the optical detection of fingerprint sensing can be greatly reduced if one or more optical filters or optical filter coatings can be designed to reject light at wavelengths from 580nm to the infrared.

Fig. 13 shows an example of an operation procedure for correcting image distortion in the optical sensor module. At step 1301, one or more illumination sources are controlled and operated to emit light in a particular area, and the light emission of such pixels is modulated by the frequency F. At step 1302, the frame rate of the imaging sensor under the display panel is operated to be the same as the frequency F to acquire an image. In optical fingerprint sensing operations, a finger is placed on top of a display panel cover substrate, and the presence of the finger modulates the intensity of light reflection from the top surface of the display panel cover substrate. An imaging sensor under the display captures a fingerprint modulated reflected light pattern. At step 1303, demodulation of the signal from the image sensor is synchronized at frequency F, and background filtering is performed. The resulting image reduces the effects of background light and includes the image resulting from light emitted by the pixels. At step 1304, the acquired image is processed and calibrated to correct for image system distortion. At step 1305, the corrected image is used as a human fingerprint image for user authentication.

The same optical sensor used for capturing the user's fingerprint can also be used to capture scattered light from the illuminated finger, as shown by light 191 scattered into the bottom layer in FIG. 5A. The detector signals from the region of interest of the light 191 scattered into the underlayer in fig. 5A can be integrated to produce an intensity signal. The intensity variation of the intensity signal is evaluated to determine the heart rate of the user.

The fingerprint sensor described above can be attacked by malicious individual hackers who can obtain the fingerprint of an authorized user and copy the stolen fingerprint pattern on a carrier similar to a human finger. Such unauthorized fingerprint patterns may be used on a fingerprint sensor to unlock a target device. Thus, a fingerprint pattern, although a unique biometric identifier, may not be a completely reliable or secure identification by itself. The off-screen optical sensor module may also function as an optical anti-spoofing sensor for sensing whether an input target having a fingerprint pattern is a finger from a live person and for determining whether the fingerprint input is a spoofing attack. There is no need to use a separate optical sensor to provide such an optical anti-spoofing sensing function. Optical anti-spoofing can provide a high speed response without compromising the overall response speed of the fingerprint sensing operation.

Fig. 14 shows exemplary extinction coefficients of materials monitored in blood, the optical absorption in blood differing between the visible spectral range of red light, e.g., 660nm, and the infrared range of infrared IR light, e.g., 940 nm. By illuminating the finger with probe light at a second, different wavelength, such as a first visible wavelength (color a) and an infrared IR wavelength (color B), differences in optical absorption of the input object can be collected to determine whether the object touched is a finger from a living person. One or more illumination light sources for providing optical sensing illumination may be used to emit light of different colors and as probe or illumination light of at least two different optical wavelengths for live finger detection using different optical absorption behavior of blood. When a person's heart beats, the pulse pressure pumps blood to flow in the artery, so the extinction ratio of the material monitored in the blood changes with the pulse. The received signal carries a pulse signal. These properties of blood can be used to detect whether the material being monitored is a live or false fingerprint.

Fig. 15 shows a comparison between optical signal behavior in reflected light from an inanimate material (such as a fake finger) and a live finger. Optical fingerprint sensors can also be used as heartbeat sensors to monitor living creatures. When two or more wavelengths of probe light are detected, the difference in extinction ratios can be used to quickly determine whether the material being monitored is a living creature, such as a live fingerprint. In the example shown in fig. 15, probe lights of different wavelengths are used, one of which is a visible wavelength and the other of which is an infrared RI wavelength, as shown in fig. 14.

When an inanimate material touches the top cover glass over the fingerprint sensor module, the received signal reveals an intensity level associated with the surface pattern of the inanimate material, and the received signal does not contain signal components associated with a live person's finger. However, when a live person's finger touches the top cover glass, the received signal reveals the signal characteristics associated with the live person, and includes significantly different intensity levels due to the different extinction ratios at the different wavelengths. This method does not require a long time to determine whether the touch material is part of a living person. In fig. 15, the pulse-like signal reflects the case of multiple touches, not blood pulses. Multiple touches similar to inanimate material do not show differences caused by live fingers.

Such optical sensing of different optical absorption behavior of blood at different optical wavelengths may be performed in short periods for live finger detection and may be faster than optical detection of a heartbeat of a person using the same optical sensor.

In the LCD display, the LCD backlight illumination light is white light, thereby containing light in the visible spectral range and the infrared IR spectral range in which the above-described living finger detection is performed at the optical sensor module. The LCD color filters in the LCD display module may be used to allow the optical sensor module to obtain the measurements in fig. 14 and 15. In addition, the designated light source 436 for generating optically sensed illumination light may be operated to emit probe light at selected visible and infrared IR wavelengths at different times, and the returned probe light of two different wavelengths is collected by the optical detector array 621 to determine whether the touch object is a live finger based on the above-described operations shown in fig. 14 and 15. It should be noted that although the reflected detection light of the selected visible and IR wavelengths at different times may reflect different light absorption characteristics of blood, the fingerprint image is always collected by the detection light of the selected visible wavelength and the detection light of the infrared IR wavelength at different times. Thus, fingerprint sensing can be performed at visible and infrared IR wavelengths.

FIG. 16 shows an example of an operational procedure for determining whether a target in contact with an LCD display screen is part of a live human finger by operating one or more illumination sources to illuminate the finger with two different colors of light.

As another example, the disclosed optical sensor technology can be used to detect whether a captured or detected pattern of a fingerprint or palm is from a live human hand using a "live finger" detection mechanism by mechanisms other than the different optical absorption of blood at different optical wavelengths described above. For example, a live person's finger is typically moving or stretched due to the person's natural movement or movement (intentional or unintentional) or a heartbeat-related pulse as blood flows through the body. In one implementation, the optical sensor module may detect changes in the returned light from the finger or palm due to heartbeat/blood flow changes, thereby detecting whether there is a live heartbeat in the target appearing as a finger or palm. User authentication may enhance access control based on a combination of optical sensing of a fingerprint/palm pattern and positive determination of the presence of a living person. As another example, when a person touches the LCD display screen, changes in the touch force may be reflected in one or more ways, including fingerprint pattern distortion, changes in the contact area between the finger and the screen surface, fingerprint ridge broadening, or dynamic changes in blood flow. These and other variations can be measured by optical sensing based on the disclosed optical sensor technology and can be used to calculate touch force. Such touch force sensing can be used to add more functionality to the optical sensor module than fingerprint sensing.

In the above example, where a fingerprint pattern is captured across the optical detector via the imaging module, as shown in fig. 4B and 6B, optical distortion typically reduces image sensing fidelity. This image distortion can be corrected in various ways. For example, a known pattern may be used to generate an optical pattern at the optical detector array, and image coordinates in the known pattern, which are used to calibrate an imaging sensing signal output by the optical detector array for fingerprint sensing, relate to an optical image generated at the optical detector array with distortion. The fingerprint sensing module calibrates the output coordinates with reference to an image of the standard pattern.

Various implementations of the disclosed optical sensor module can be made in accordance with the disclosure in this patent document.

For example, the display panel may be configured to: each of which can be controlled individually; the display panel includes an at least partially transparent substrate and a substantially transparent cover substrate. The optical sensor module is located under the display panel to sense an image formed on top of a surface of the display panel. The optical sensor module may be used to sense an image formed by light emitted from the display panel pixels. The optical sensor module may include a transparent block having a refractive index lower than that of the display panel substrate, an imaging sensor block having an imaging sensor array, and an optical imaging lens. In some implementations, the low index block has a refractive index in a range of 1.35 to 1.46 or 1 to 1.35.

As another example, a method for fingerprint sensing may be provided, wherein light emitted from a display panel is reflected by a cover substrate, and a finger located on top of the cover substrate interacts with the light to modulate a light reflection pattern by a fingerprint. An imaging sensing module under the display panel is used for sensing the reflected light pattern image and reconstructing a fingerprint image. In one implementation, the emitted light from the display panel is modulated in the time domain, and the imaging sensor is synchronized with the modulation of the emitting pixels, and the demodulation process based on this setting rejects most of the background light (not the light from the target pixels).

Various design considerations for the disclosed under-screen optical sensor module for optical fingerprint sensing are given in appendix 3 entitled "multifunction fingerprint sensor and package" (pages 41 text and 26 drawings) as part of U.S. provisional application No. 62/289,328 and U.S. provisional application No. 62/330,833, and in international patent application No. PCT/US2016/038445 filed 2016 (for priority to U.S. provisional patent application No. 62/181,718 filed 2015 6-18 and published as WO2016/205832 at 2016 12-22) and in international patent application No. PCT/CN2016/104354 filed 2016 (for priority to U.S. provisional patent application No. 62/249,832 filed 2015 11-2-2015), and published as WO2017/076292 on 5/11 of 2017). The entire disclosure of the above-mentioned patent application is incorporated by reference into a portion of the disclosure of this patent document.

In various implementations of the under-screen optical sensor module technology for fingerprint sensing disclosed herein, optical imaging of the illuminated touch portion of the finger to an optical detector array in the under-screen optical sensor module may be achieved without the use of an imaging module such as a lens that images light returning from the touch portion of the finger under optical illumination. One technical challenge of optical fingerprint sensing without an imaging module is how to control the propagation of returned light, which may spatially disturb the light at the optical detector array that is returned from different locations on the touch portion of the finger, such that when such returned light reaches the optical detector array, spatial information of the different locations may be lost. This challenge can be addressed by using an optical collimator or pinhole array instead of an optical imaging module in the underscreen optical sensor module to detect fingerprints with optical sensing. An apparatus for implementing such optical fingerprint transmission may include: a device screen providing touch sensing operation and comprising a display panel structure having light emitting display pixels, each pixel operable to emit light to form a portion of a display image; a top transparent layer formed over the device screen as a user touch interface for touch sensing operations and an interface for transmitting light from the display structure to display an image to a user; and an optical sensor module located beneath the display panel structure to receive light emitted by at least a portion of the light emitting display pixels of the display structure and returned from the top transparent layer to detect a fingerprint, the optical sensor module comprising an array of optical detectors to receive the returned light, the optical sensor module further comprising an array of optical collimators or pinholes in a path of the returned light to the array of optical detectors. The array of optical collimators is used to collect the returned light from the display panel structure and separate the light from different locations in the top transparent layer while directing the collected returned light to the array of optical detectors.

Imaging using collimators relies on the use of different collimators at different locations to spatially separate light from different areas of the fingerprint to different optical detectors in an optical detector array. The thickness or length of each collimator along the collimator may be designed to control the narrow field of optical view of each collimator, e.g., only light from a small area on the illuminated finger is collected by each collimator and projected onto several adjacent optical detectors in the optical detector array. For example, each collimator may be designed to be large, such as several hundred microns, along the thickness or length of the collimator, such that the field of optical view of each collimator may allow the collimator to transmit imaging light to a small area on the optical detector array, such as one optical detector in the optical detector array or several adjacent optical detectors (e.g., in some cases, a ten and a few microns area on each side of the optical detector array).

The following section explains how an optical collimator array or pinhole array is used for underscreen optical fingerprint sensing by way of example, which uses optical collimators in hybrid sensing pixels at the time of optical fingerprint sensing, each hybrid sensing pixel having a capacitive sensor for acquiring fingerprint information and an optical sensor for acquiring fingerprint information.

Fig. 17A and 17B show two examples of hybrid sensing pixel designs that combine capacitive sensing and optical sensing in the same sensing pixel.

Fig. 17A shows an example of a fingerprint sensor device 2100, which fingerprint sensor device 2100 incorporates a capacitive sensor on the basis of an optical sensor for each sensor pixel of a sensor pixel array when acquiring fingerprint information. By combining a capacitive sensor and an optical sensor, the fingerprint image obtained using the optical sensor can be used to better resolve the 3D fingerprint structure obtained using the capacitive sensor. For illustrative purposes, the structure shown in fig. 17A represents one sensor pixel in an array of sensor pixels, and each sensor pixel includes an optical sensor 2102 and a capacitive sensor 2114 arranged adjacent to each other within the same pixel.

Optical sensor 2102 includes photodetector 2108 and collimator 2106 positioned above photodetector 2108 to narrow or focus light 2124 reflected from finger 2102 toward photodetector 2108. One or more light sources (not shown), such as LEDs, may be placed around collimator 2106 to emit light that is reflected by the finger as reflected light 2124 and directed to or focused toward a corresponding photodetector 2108 to collect a portion of the fingerprint image of finger 2102. Collimator 2106 may be implemented using a fiber optic bundle or one or more metal layers with holes or openings. Such use of multiple optical collimators above the optical detector array may be used as a lens-less optical design to acquire a fingerprint pattern with a desired spatial resolution for reliable optical fingerprint sensing. Fig. 17A shows a collimator 2106 implemented using one or more metal layers 2110 with holes or openings 2112. Collimator 2106 in the layer between top structure or layer 2104 and photodetector 2108 in fig. 17A comprises a plurality of individual optical collimators formed by optical fibers or holes or openings in one or more layers (e.g., silicon or metal), each of such individual optical collimators receiving light rays 2124 along the longitudinal direction of each optical collimator or within a small angular range, as shown, light rays 2124 may be collected by the top opening of each opening or hole and the tubular structure, such that light rays incident at large angles from the longitudinal direction of each optical collimator are rejected by each collimator to the optical photodiode on the other end of the optical collimator.

In the capacitive sensing portion of each sensing pixel, the capacitive sensor 2114 includes a capacitive sensor plate 2116, the capacitive sensor plate 2116 being electromagnetically coupled to a portion of a finger in proximity or contact with the sensing pixel for capacitive sensing. More specifically, when finger 2102 is in contact with or in close proximity to optional cover 2104 or a cover on a mobile device implementing finger sensor device 2100, capacitive sensor plate 2116 and finger 2102 interact as two plates of one or more capacitive elements 2122. The number of capacitive sensor plates 2116 may vary based on the design of the capacitive sensor 2114. Capacitive sensor plate 2116 may be implemented using one or more metal layers. The capacitive sensor plates 2116 are communicatively coupled to capacitive sensor circuitry 2120 such that the capacitive sensor circuitry 2120 may process signals from the capacitive sensor plates 2116 to derive signals representative of the 3D fingerprint structure. Routing or shielding material may be disposed between capacitive sensor plates 2116 and the capacitive sensor circuitry to electrically shield metal plates 2116. The capacitive sensor circuitry 2120 may be communicatively coupled to the capacitive sensor plate 2116 and the photodetector 2108 to process signals from the capacitive sensor plate 2116 and signals from the photodetector 2108. In fig. 17A, the capacitive sensor and the optical sensor within each hybrid sensing pixel are adjacent to each other and displaced from each other without spatially overlapping.

In an implementation approach, optical sensing features such as the optical collimator design in the hybrid sensor design in fig. 17A may be used in an under-screen optical sensor module. Thus, the optical sensing with optical collimator features in fig. 17A may be implemented in a mobile device or electronic device capable of detecting fingerprints by optical sensing, the mobile device or electronic device comprising: a display screen structure; a top transparent layer formed over the display screen structure as a user touch interface and as an interface for transmitting light from the display screen structure to display an image to a user; and an optical sensor module located below the display screen structure to receive light returning from the top transparent layer to detect a fingerprint. The optical sensor module includes an optical detector array of photodetectors that receive the returned light, and further includes an array of optical collimators to collect the returned light from the top transparent layer through the display screen structure and to separate light from different locations in the top transparent layer while directing the collected returned light through the optical collimators to the photodetectors in the optical detector array.

Fig. 17B shows another example of a fingerprint sensor device 2130 that structurally integrates an optical sensor and a capacitive sensor in each hybrid sensor pixel in a spatially overlapping configuration in a sensor pixel array to reduce the space occupied by each hybrid sensing pixel. The fingerprint sensor device 2130 includes a semiconductor substrate 2131 of silicon or the like. A plurality of sensing elements or sensing pixels 2139 are disposed on the substrate 2131. Each sensing element or sensing pixel 2139 includes an active electronic circuit area 2132, which active electronic circuit area 2132 includes CMOS switches, amplifiers, resistors, and capacitors to process the sensor signals. Each sensing pixel or sensing element 2139 includes a photodetector 2133 disposed or embedded in an active electronic circuit region 2132. A capacitive sensor plate or top electrode 2134 of a capacitive sensor for capacitive sensing is disposed on the photodetector 2133 and includes a hole or opening 2138 in the sensor plate 2134 that also functions as a collimator to direct light onto the photodetector 2133. Vias 2135 filled with a conductive material are provided to electrically connect the top electrode 2134 to the active circuit elements 2132. By adjusting the opening or hole and the distance of the top electrode 2134 from the photodetector 2133, the light collection angle 2137 of the photodetector (e.g., photodiode) 2133 can be adjusted. The fingerprint sensor device 2130 is covered by a protective cover 2136, which protective cover 2136 comprises a hard material, such as sapphire, glass, etc. The light collection angle 2137 of the photodetector 2133 can be designed to preserve the spatial resolution of the image collected by the photodiode array. A light source 2140 such as an LED is located under the cover plate on the side of the fingerprint sensor device 2130 to emit light which is reflected by the finger and directed to the photodetector 2133 to capture the fingerprint image. When a finger touches or is very close to the protective cover plate, the combination of the finger and the sensing top electrode 2134 forms a capacitive coupling (e.g., capacitor 2142) between the human body and the sensing top electrode 2134. The fingerprint sensor device 2130, which includes an optical sensor and a capacitive sensor, can obtain both light reflection images and capacitively coupled images of a fingerprint. Sensing top electrode 2134 serves a dual purpose: 1) for capacitive sensing, and 2) as a collimator (by making one or more holes in the sensing top electrode 2134) to direct the light returning from the finger to the photodetector 2133 or to narrow or focus towards the photodetector 2133. The reuse of the sensing top electrode 2134 eliminates the need for additional metal layers or fiber bundles, thereby reducing the size of each pixel and thus the overall size of the fingerprint sensor device 2130.

In fig. 17B, the optical sensing design uses the holes or openings 2138 formed between the top layer 2136 and the bottom array 2133 of photodetectors as optical collimators to select only light rays within certain angles 2137 to maintain the spatial resolution of the image collected by the photodetectors 2133 in the photodetector array as shown. Similar to the optical fiber or other tubular optical collimator of fig. 17A, the holes or openings 2138 formed between the top layer 2136 and the bottom array of photodetectors 2133 constitute an optical collimator that collects the returned light from the top transparent layer via the display screen structure and separates the light from different locations in the top transparent layer while directing the collected returned light through the optical collimator to the photodetectors 2133.

Fig. 18 is a top view of an exemplary hybrid fingerprint sensor device 2200 that incorporates an optical sensor and a capacitive sensor into each hybrid sensing pixel. The fingerprint sensor device 2200 is implemented as a CMOS silicon chip 2221, which CMOS silicon chip 2221 includes a hybrid (incorporating optical and capacitive sensors) array of sensing elements or pixels 2222. Alternatively, the layout in fig. 18 can also be used for all optical sensing designs disclosed in this document, where the opening or hole 2223 represents the optical collimator in fig. 17A and 17B. For example, the size or dimensions of the sensing elements may be in the range of 25 μm to 250 μm. Hybrid sensor device 2220 may include an array of support circuits including amplifiers, analog-to-digital converters ADC and buffer memory in side area 2224. Further, hybrid sensor device 2200 may include a region 2225 for wire bonding or bump bonding. The top layer 2226 of the hybrid sensor element 2222 may include metal electrodes for capacitive sensing. One or more openings or holes 2223 may be made in each top metal electrode 23 to structurally act as a collimator to direct light in a vertical direction onto the photodetector under the top electrode. Thus, the top layer 2226 structure can serve dual purposes of optical sensing and capacitive sensing. A sensor device processor may be provided to process the pixel output signals from the hybrid sensing pixels to extract fingerprint information.

One example of a sensor signal detection circuit may be shared between the optical sensor and the capacitive sensor to detect sensor signals from the photodetector and the capacitive sensor plate, except that the same structure for capacitive sensing and for focusing light in the vertical direction is shared as a collimator.

Fig. 19A shows an exemplary hybrid fingerprint sensing element or pixel 2300 with capacitive sensing and optical sensing functionality for a fingerprint. The exemplary sensor pixel 2300 includes a sensor signal detection circuit 2316 to selectively switch between detecting or acquiring a sensor signal from a sensing top electrode (e.g., top metal layer) 2308 based on capacitive sensing and a sensing signal from a photodetector (e.g., photodiode) 2314 based on optical sensing to acquire a reflected optical image from the photodetector 2314 and a capacitively coupled image from the capacitive sensor electrode 2308 of the finger. In some implementations, the two images from the two sensing mechanisms in each hybrid sensing pixel can be processed serially by the sensor signal detection circuitry. In the example shown, switches 2310 and 2312 have: a first terminal electrically coupled to the sensing top electrode 2308 and the photodetector 2314, respectively, and a second terminal coupled to a common input of the sensor signal detection circuit 2316 to provide a corresponding optical detector signal from the photodetector 2314 and a corresponding capacitive sensing signal from the sensing top electrode 2308 to the sensor signal detection circuit 2316. When the switch 2310 is open (CAP _ EN ═ 0) and the switch 2312 is closed (Optical _ EN ═ 1), the sensor signal detection circuit 2316 acquires an Optical detector signal representative of the Optical image of the scanned fingerprint received at a particular hybrid sensing pixel. When CAP _ EN of the switch 2310 is 1 and optical _ EN is 0, the sensor signal detection circuit 2316 may acquire a capacitance sensing signal representing a capacitance image of a scanned fingerprint. After both the optical and capacitance images are acquired, the two images may be processed in downstream circuitry, either individually or in combination, to identify fingerprint features.

With the two imaging modalities of hybrid sensing pixels described above, the performance of fingerprint recognition can be enhanced by utilizing the two types of images in different ways. Such enhanced fingerprint recognition may be implemented by a sensor device processor, such as sensor device processor 2321, for processing pixel output signals from the hybrid sensing pixels to extract fingerprint information. For example, the capacitance image may provide a 3D image of the depth of the ridges and valleys of the fingerprint features. In addition to the 3D capacitive image, the optical image may provide high resolution 2D information about the fingerprint features. Since both image information are about the same ridge of the fingerprint, an optical 2D image with higher spatial resolution can be used to restore the capacitive sensing image resolution. In some implementations, the capacitive sensing method may more sensitively and accurately identify the valleys of the fingerprint than the optical sensing method, and the spatial resolution of the image acquired using the capacitive sensing method may be degraded based on the thickness of the cover plate. This aspect of capacitive sensing can be compensated for by optical sensing. In operation, the sensor response may be fixed, and the point spread function of the capacitive sensor may be fixed for all sensor positions. Higher resolution optical sensing can be used as a resolution recovery method and can be applied on the capacitive sensing image to enhance the 3D image. Partial high resolution images from optical sensing can be used to assist the restoration method. Thus, the 3D capacitive image can be enhanced by interpolation or restoration based on the high resolution 2D image to provide more information about the valleys and ridges.

The enhanced 3D image may provide improved fingerprint identification and matching. In another example, the optical image and the capacitance image may be stored together to provide two comparisons each time a fingerprint identification or match is made. The use of two types of images for comparison enhances the accuracy and security of the fingerprint sensing system.

The sensor signal detection circuit 2316 can be implemented in a variety of ways using a variety of different circuit designs. In one example, the integral sensing circuit 2318 may be implemented to store charge caused by a touch of a ridge or valley or a cover of a fingerprint sensor device that is very close to the cover of the mobile device. The inclusion of the integration circuit 2318 enhances the signal-to-noise ratio (SNR). The integrating sensing circuit includes an operational amplifier 2322 to amplify a sensor signal, such as a capacitance-related signal or an optical-related signal (e.g., a voltage signal), detected by the sensing top electrode 2308 or the photodetector 2314 of the exemplary sensor pixel 2300. The sensing top electrode 2308, which comprises an electrically conductive material of one of a variety of metals, is electrically connected to the negative or inverting input 2328 of the amplifier 2322 through a switch 2310. The sensing top electrode 2108 and the local surface 2302 of the finger act as opposite plates of the capacitor Cf 2302. The capacitance of the capacitor Cf 2302 varies based on the distance'd', i.e., the distance between the two plates of the capacitor Cf 2302, between the local surface of the finger and the sensing top electrode 2308. The capacitance of the capacitor Cf 2302 is inversely proportional to the distance'd' between the two plates of the capacitor Cf 2302. The capacitance of the capacitor Cf 2302 when the sensing top electrode 2308 is opposite the ridge of a finger is larger than the capacitance of the capacitor Cf 2302 when the sensing top electrode 2308 is opposite the valley of a finger.

Furthermore, in the exemplary sensor pixel 2300, various parasitic or other capacitors may be formed between different conductive elements. For example, a parasitic capacitor CP 2304 may be formed between sensing top electrode 2308 and device ground 2305. The device ground is closely coupled to ground. Another capacitor Cr 2324 may be formed between the output conductor of the amplifier 2322 and the negative or inverting input 2328 of the amplifier 2322 and serves as the feedback capacitance of the amplifier 2322. Also, a switch 2326 may be coupled between the output of the amplifier 2322 and the negative or inverting input 2328 of the amplifier 2322 to reset the integration circuit 2318.

The positive terminal of the amplifier 2322 is electrically connected to the excitation signal Vref. In each sensor pixel, the excitation signal Verf can be provided directly to the positive terminal of a dedicated amplifier. Exemplary sensor pixel 2100 becomes an active sensor pixel by providing an excitation signal Verf directly to the positive terminal of amplifier 2322. Furthermore, providing the excitation signal Verf directly to the positive terminal of the amplifier 2322 eliminates the need to include an excitation electrode common to all sensor pixels, which reduces the conductive (e.g., metal) layer from the semiconductor structure of the sensor chip. In some implementations, the optional excitation electrode 2306 can be implemented to enhance SNR based on the design of the sensor pixel. Furthermore, by providing the excitation signal Verf directly to the amplifier 2322, the excitation signal Vref is not applied directly to the finger to avoid potential irritation or injury to the finger. Furthermore, when the excitation electrode that directly applies the excitation signal to the finger is not used, all components of the fingerprint sensor device may be integrated into a single packaging device, and the entire fingerprint sensor device may be disposed under the protective cover glass. Since the entire fingerprint sensor device is arranged under the protective cover glass, the fingerprint sensor device is protected from fingers and other external elements that may damage the fingerprint sensor.

In fig. 19A, the output signals (both optical and capacitive) of sensor signal detection circuit 2316 in sensor pixel 2300 (e.g., Vpo of amplifier 2322) are electrically coupled to switch 2320 to selectively output the output signal Vpo from sensor pixel 2300 to signal processing circuitry including a filter. Switch 2320 may be implemented using a transistor or other switching mechanism, and may be electrically coupled to a controller to control switching of switch 2320. By controlling

switches

2320, 2310 and 2312, sensor pixels in an array of sensor pixels can be selectively switched between acquiring optical signals and acquiring capacitive signals. In one implementation, an optical or capacitance signal may be acquired for each row, or column of sensor pixels in the array, and then switched to acquire other types of signals for that row, or column. The switching between optical signal acquisition and capacitance signal acquisition may be done line by line. In another implementation, one type of signal (capacitive or optical) may be acquired for all sensor pixels or elements in the array, and then switched to acquire the other type of signal for all sensor pixels or elements. Thus, switching between acquisition of different signal types may occur across the array. Other variations of switching between the acquisition of the two types of sensor signals may be implemented.

Fig. 19B shows a circuit diagram of another exemplary hybrid fingerprint sensing element or pixel 2340. The hybrid fingerprint sensing element or pixel 2340 is substantially the same as the hybrid fingerprint sensing element or pixel 2300 for components having the same reference number. For a description of common parts having the same reference numerals, refer to the description of fig. 19A.

The hybrid fingerprint sensing element or pixel 2340 utilizes the sensing top electrode 2308 to include an aperture or opening 2342 thereon that functions as a collimator to focus or narrow the reflected light 2344 toward the photodetector 2314 (e.g., photodiode). The photodetector 2314 may be located or disposed below a collimator implemented using the sensing top electrode 2308 to collect reflected light 2344 focused by the collimator 2308.

In some implementations, separate instances of sensor signal detection circuitry for the optical sensor and the capacitive sensor may be included to detect sensor signals from the photodetector and the capacitive sensor plate in parallel.

Fig. 19C shows a circuit diagram of an exemplary hybrid fingerprint sensing element or pixel 2350 for parallel detection of sensor signals from a photodetector and a capacitive sensor plate. Hybrid fingerprint sensing element or pixel 2350 is substantially identical to hybrid fingerprint sensing element or pixel 2340 for components having the same reference number. For a description of common parts having the same reference numerals, refer to the description of fig. 19A.

To perform parallel detection of sensor signals from capacitive plates and photodetectors, hybrid fingerprint sensing element or pixel 2350 includes separate sensor

signal detection circuits

2316 and 2317 communicatively coupled with sensing top electrode 2308 and photodetector 2324, respectively. The sensor signal detection circuit 2317 may be implemented substantially similar to the sensor signal detection circuit 2316. In some implementations, switches 2310 and 2312 may be provided having first terminals electrically coupled to the sensing top electrode 2308 and the photodetector 2314, respectively, and second terminals coupled to the sensor

signal detection circuits

2316 and 2317, respectively, to provide the sensor

signal detection circuits

2316 and 2317 with an optical detector signal from the photodetector 2314 and a capacitive sensing signal from the sensing top electrode 2308, respectively. When the

switches

2310 and 2312 are closed and opened together, the sensor

signal detection circuits

2316 and 2317 may detect sensor signals from the capacitive plate 2308 and the photodetector 2314 in parallel. When the

switches

2310 and 2312 are closed and opened out of phase with each other, the sensor

signal detection circuits

2316 and 2317 may detect sensor signals from the capacitive plate 2308 and the photodetector 2314 in series. Further, the sensor device processor 2321 may be directly or indirectly communicatively coupled to the sensor

signal detection circuits

2316 and 2317 through the

switches

2320A and 2320B to process the detected sensor signals from the capacitive plate 2308 and the photodetector 2314 in parallel or in series.

In another aspect of the disclosed technique, due to the heart beat and pumping action, the optical sensor described with respect to fig. 17A, 17B, 18, 19A and 19B can be used to measure the heart beat of a person by measuring the reflected light intensity in the finger over time caused by blood flow changes. This information is contained in the received light reflected, scattered or diffused by the finger and is carried by the optical detector signal. Thus, the optical sensor may be used for a variety of functions, including acquiring an optical image of a fingerprint and measuring a person's heartbeat. In an implementation, the sensor device processor is to process one or more optical detector signals to extract heartbeat information. Such a sensor device processor may be the same as the sensor device processor that processes the pixel output signals from the optical sensing pixels or the hybrid sensing pixels to extract fingerprint information.

The following sections describe examples of various designs for fingerprint sensing using an under-screen optical sensor module that utilizes an optical collimator array or pinhole array to direct signal light carrying fingerprint information to an optical detector array. Such an optical collimator or pinhole is placed between the LCD display screen and the optical detector array in the underscreen optical sensor module to couple the desired return light from the display panel while filtering out background light when the optical detector array is optically detecting. Such an implementation of an optical collimator or pinhole may simplify the optical design of the optical detector array for optical detection, e.g., the complex optical imaging designs of fig. 6B, 7, 10A, and 11 are not used in other designs disclosed in this patent document. Furthermore, the implementation of such optical collimators or pinholes may simplify the optical alignment of the entire optical layout of the optical detector array and improve the reliability and performance of the optical detection by the optical detector array. Moreover, such optical collimators or pinholes may significantly simplify the manufacturing process and reduce the overall cost of the underscreen optical sensor module.

Fig. 20 shows an example of a smartphone having a Liquid Crystal Display (LCD) display and an under-screen optical sensor module that includes an array of optical collimators for correcting and directing light to an array of optical detectors for optical fingerprint sensing. The LCD-based touch sensing display system 423 implements an optical sensing module with an array of photo-detectors 621 under the LCD display system 423.

The touch sensing display system 423 is placed under a top cover glass 431, the top cover glass 431 serving as a user interface surface for various user engagement operations including touch sensing operations by a user and displaying images to the user, etc., and as an optical sensing interface for receiving a finger for fingerprint sensing and other optical sensing, wherein probe light is directed from outside the device to the top cover glass 431 to illuminate the finger. Although the microstructures may eliminate part of the detected light energy, most of the LCD structure layer 433a becomes partially transparent when the LCD cells in the sensing window are open. The light diffuser 433b, light guide 433c, reflector film 433d and LCD module frame are used to support the fingerprint sensor and provide a transparent or partially transparent sensing light path so that a portion of the reflected light from the top surface of the cover glass 431 can reach the photodetector array 621 within the LCD sub-screen optical sensor module for fingerprint sensing and other optical sensing operations. As shown, this particular example of an optical sensor module under the LCD screen includes various fingerprint sensor components, such as an optical collimator array 617 for collimating and directing reflected probe light to a photo-detector array 621, and an optical sensor circuit module 623 that receives and conditions detector output signals from the photo-detector array 621. The optical collimator array 617 may include optical collimators, and the optical collimator array 617 may be a waveguide-based image emitter, a fiber array, a micro-lens array, or a pinhole array. The optical collimator is used to limit the Numerical Aperture (NA) of the sampled image and to form the corresponding image elements. Each optical collimator unit acquires a part of an image of the touched portion of the target finger on the top glass cover 431. The beams transmitted by all collimators together form a complete image of the target at the photodetector array 621. The photodiode array 621 may be a CMOS sensor of CMOS sensing pixels, a CDD sensor array, or a suitable optical detector array sensitive to light.

The illustrated example includes an electronics module 435 for LCD display and touch sensing operations, one or more other sensors 425 such as optical sensors for monitoring ambient light levels, and

optional side buttons

427 and 429 for controlling certain smartphone operations.

In the example of fig. 20, the light sources in the illustrated example include a display backlight light source 434 and an additional designated detection light source 436. The light beams 442a from the additional designated detection light source 436 and the light beams 442b from the display light source 434 may be used as sensor detection light for illuminating a finger in contact with the top glass cover 431 to generate the desired reflected detection light carrying fingerprint patterns and other information to the optical sensor module.

Although the microstructures can eliminate part of the detected light energy, when the LCD cell in the sensing window is opened, most of the LCD structure layer 433a (including liquid crystal cells, electrodes, transparent ITO, polarizers, color filters, touch sensing layers, etc.) becomes partially transparent. The light diffuser 433b, light guide 433c, reflector film 433d and LCD module frame are used to support the fingerprint sensor and provide a transparent or partially transparent sensing light path.

Fig. 21 further illustrates the operation of the LCD underscreen optical sensor module in the above example of fig. 20 below. On the top cover glass 431, the fingerprint sensing area or window is the area on the top surface of the top cover glass 431, which top cover glass 431 is located directly above or near the underlying optical sensor module. Since the optical sensor module is below the LCD structure, the sensing window is part of the continuous top surface of the top cover glass 431 and is also part of the display area for the LCD display. Thus, there may be no visible physical boundary on the top surface to indicate the sensing window. Such sensing windows may be indicated to the user by other means to assist the user in placing a finger within the sensing window for fingerprint sensing and other optical sensing operations. For example, an additional designated detection light source 436 may be used to illuminate the sensing window such that the area of the sensing window is significantly different from the surrounding area on the top cover glass and is easily visible to the user. This may be done when the LCD panel is off or when the LCD panel is on.

As shown in fig. 21, the user presses a finger on the sensing window, and then the probe beam 82P illuminates the finger. The finger and cover glass 431 reflects the probe light as reflected signal beam 82R. Various scattering interfaces 433S in LCD module 433 diffuse reflected signal beam 82R to form a diffused beam 82D. Each collimator element in collimator array 617 selects a light component 82S and directs the selected light component 82S into a corresponding photosensitive detector of photodetector array 621. A light sensitive detector, such as a photodiode or CMOS sensing detector, generates a corresponding sensor signal containing information about the fingerprint pattern. A portion of the light source may enter the fingerprint sensor module without first passing through the finger sensing area on the top surface of the LCD panel. This part of the light is background noise and can be eliminated by calibration. Each collimator unit of collimator array 617 selects only light that is transmitted into a portion of the corresponding photodetectors in photodetector array 621 with relatively low light loss along the direction it allows. Thus, each collimator unit in the collimator array 617 and its corresponding photodetector in the photodetector array 621 operate together to define an effective detection optical numerical aperture NA. This NA directly defines the spatial resolution of the image produced by the optical sensor module.

Based on the disclosed under-LCD optical sensor design, a human finger in direct contact with or near the LCD display screen can cause the returned light to carry information of a portion of the finger illuminated by the light output by the LCD display screen while returning into the LCD display screen. Such information may include the spatial pattern and location of ridges and valleys of the illuminated portion of the finger, etc. Thus, the integration of the optical sensor module may collect at least a portion of this returned light to detect the spatial pattern and location of the ridges and valleys of the illuminated portion of the finger through optical imaging and optical detection operations. The spatial pattern and location of the ridges and valleys of the illuminated portion of the detected finger may then be processed to construct a fingerprint pattern and perform fingerprint recognition, for example, as part of a user authentication and device access process, in comparison with a stored authorized user fingerprint pattern to determine if the detected fingerprint is a matching fingerprint. Such optically-sensing based fingerprint detection using the disclosed optical sensor technology uses an LCD display screen as the optical sensing platform and can be used to replace existing capacitive fingerprint sensors or other fingerprint sensors, which are essentially stand-alone sensors as "add-on" components, without using light from the display screen or using a display screen for fingerprint sensing of cell phones, tablets, and other electronic devices.

It is noted that optical sensor modules based on the disclosed optical sensor technology can be coupled to the back of an LCD display screen without requiring a designated area on the display surface side of the LCD display screen that, in some electronic devices such as smartphones, tablets, or wearable devices, can take up valuable device surface space. Such an optical sensor module may be placed under the display screen of the LCD, vertically overlapping the display screen area, and hidden behind the display screen area from the user's perspective. Furthermore, since the optical sensing of such an optical sensor module is performed by detecting light from the display screen of the LCD and returning from the top surface of the display area, the disclosed optical sensor module does not require a special sensing port or sensing area separate from the display screen area. Thus, in other designs, including the apple iPhone/iPad device or the samsung Galaxy smartphone model, etc., where the fingerprint sensor is located at a specific fingerprint sensor area or port (such as a home button) on the same surface of the display screen, but in a designated non-display area outside the display screen area, unlike the fingerprint sensor in this other design, an optical sensor module based on the disclosed optical sensor technology can be implemented in the following manner: by routing light returning from the finger into the optical sensor using a unique optical sensing design, and by providing a suitable optical imaging mechanism to enable high resolution optical imaging sensing, fingerprint sensing is allowed at locations on the display screen of the LCD. In this regard, implementations of the disclosed optical sensor technology provide a unique on-screen fingerprint sensing configuration and provide touch sensing operation by using the same top touch sensing surface as the display image, without a separate fingerprint sensing area or interface outside the display screen area

In addition to fingerprint detection by optical sensing, optical sensing can also be used to measure other parameters. For example, the disclosed optical sensor technology is capable of measuring patterns of human palms with large touch areas available on the display screen of an entire LCD (in contrast, some designated fingerprint sensors, such as those in the home button of an apple iPhone/iPad device, have rather small and designated off-screen fingerprint sensing areas that are highly limited in the size of the sensing area, which may not be suitable for sensing large patterns). As another example, the disclosed optical sensor technology may be used not only to capture and detect patterns of fingers or palms associated with a person using optical sensing, but also to detect whether a pattern of fingerprints or palms captured or detected by a "live finger" detection mechanism is from a live person's hand using optical sensing or other sensing mechanisms, which detection mechanism may be based on the fact that a live person's fingers are typically moving or stretched due to natural movement or motion (intentional or unintentional) of the person or pulsation as blood flows through a body connected to a heartbeat. In one implementation, the optical sensor module may detect changes in the returned light from the finger or palm due to heartbeat/blood flow changes, thereby detecting whether there is a live heartbeat in the target appearing as a finger or palm. User authentication may enhance access control based on a combination of optical sensing of a fingerprint/palm pattern and positive determination of the presence of a living person. As another example, the optical sensor module may include sensing functionality for measuring glucose levels or oxygen saturation based on optical sensing of returned light from the finger or palm. As another example, when a person touches the display screen of an LCD, changes in the touch force can be reflected in one or more ways, including fingerprint pattern distortion, changes in the contact area between a finger and the screen surface, fingerprint ridge broadening, or dynamic changes in blood flow. These and other variations can be measured by optical sensing based on the disclosed optical sensor technology and can be used to calculate touch force. Such touch force sensing can be used to add more functionality to the optical sensor module than fingerprint sensing.

For useful operational or control features related to touch sensing aspects of the display screen of the LCD, the disclosed optical sensor technology may provide a trigger function or additional function based on one or more results from the optical sensor module to perform certain operations related to touch sensing control on the display screen of the LCD. For example, the optical properties (e.g., refractive index) of finger skin are often different from other artificial objects. The optical sensor module may hereby be designed to selectively receive and detect returned light caused by a finger in contact with the surface of the display screen of the LCD, while returned light caused by other objects is not detected by the optical sensor module. Such object selective optical detection can be used to provide useful user control through touch sensing, such as waking a smartphone or device only via a touch of a human finger or palm, while a touch of other objects does not cause the device to wake up, for power saving operation and prolonged battery use. Such operation may be achieved by control based on the output of the optical sensor module to control the wake-up circuit operation of the LCD display, wherein, for example, an additional light source designed for optical sensing may be included and turned on in a flash mode to flash intermittently to the screen surface to sense any touch of a human finger or palm, while the LCD display may be placed in a sleep mode to save energy. In some implementations, the wake-up sensing light may be in a spectral range that is invisible to infrared, so the user does not experience any flickering of visual light.

An optical sensor module based on the disclosed optical sensor technology can be coupled to the back of the display screen of an LCD without the need to create a designated area on the surface side of the display screen of the LCD that would take up valuable device surface space in some electronic devices such as smartphones, tablets, or wearable devices. This aspect of the disclosed technology may be used to provide certain advantages or benefits in device design and product integration or manufacture.

In some implementations, optical sensor modules based on the disclosed optical sensor technology can be configured as non-invasive modules that can be easily integrated into the display screen of an LCD without the need to change the design of the display screen of the LCD to provide the desired optical sensing functionality, such as fingerprint sensing. In this regard, optical sensor modules based on the disclosed optical sensor technology may be independent of the design of the display screen design of a particular LCD due to the following properties of the optical sensor module: the optical sensing of such optical sensor modules is performed by detecting light from the display screen of the LCD that returns from the top surface of the display area, and the disclosed optical sensor module is coupled to the back surface of the display screen of the LCD for receiving the returned light from the top surface of the display area, thereby eliminating the need for a specific sensing port or sensing area separate from the display screen area. Thus, such an underscreen optical sensor module may be used in combination with a display screen of an LCD to provide optical fingerprint sensing and other sensor functions on the display screen of the LCD without using a specially designed display screen of the LCD having hardware specifically designed to provide such optical sensing. This aspect of the disclosed optical sensor technology may make the display screen of an LCD more extensive in smartphones, tablets, or other electronic devices with enhanced functionality of optical sensing from the disclosed optical sensor technology.

For example, for existing phone component designs that do not provide a separate fingerprint sensor, like some apple iPhone or samsung Galaxy models, such existing phone component designs may integrate an off-screen optical sensor module as described herein without changing the touch-sensing display screen component to provide increased on-screen fingerprint sensing functionality. Because the disclosed optical sensing does not require a separate designated sensing area or port, like some apple iPhone/samsung Galaxy cell phones with a front fingerprint sensor outside the display screen area, or some smart phones like hua, millet, google, or some models of association with a designated rear fingerprint sensor on the back, the integration of on-screen fingerprint sensing disclosed herein does not require substantial changes to existing phone component designs or touch sensing display modules with touch sensing layers and display layers. Thus, no external sensing port and external hardware is required outside the device, which requires the addition of the disclosed optical sensor module for fingerprint sensing. The added optical sensor module and associated circuitry is under the display screen within the phone housing and can be conveniently fingerprint sensed on the same touch sensing surface of the touch screen.

As another example, due to the above-described properties of optical sensor modules for fingerprint sensing, smartphones incorporating such optical sensor modules can be upgraded with improved designs, functions and integration mechanisms without affecting or burdening the design or manufacture of the display screen of the LCD to provide desired flexibility for device manufacturing and upgrading/upgrading over the product cycle while maintaining the availability of newer versions of optical sensing functions in smartphones, tablets or other electronic devices that use the display screen of the LCD. In particular, the touch sensing layer or the display screen of the LCD can be updated at the next product release without any significant hardware changes to the fingerprint sensing features using the disclosed off-screen optical sensor module. Also, on-screen optical sensing implemented based on such optical sensor modules for fingerprint sensing or other optical sensing function improvements, including the addition of additional optical sensing functions, can be implemented by using new versions of off-screen optical sensor modules in new products without requiring significant changes to the phone component design.

The above or other features of the disclosed optical sensor technology may be implemented to provide improved fingerprint sensing and other sensing functionality to new generation electronic devices, particularly smartphones, tablets, and other electronic devices having LCD display screens, to provide various touch sensing operations and functionality, and to enhance the user experience of such devices.

The optical sensor technology disclosed herein uses light returning from the top surface of the device display assembly for displaying images in a display screen for fingerprint sensing and other sensing operations. The returned light carries information of the object (e.g., finger) in contact with the top surface, and the collection and detection of the returned light constitutes part of design considerations in implementing a particular optical sensor module located below the display screen. Because the top surface of the touch screen assembly serves as the fingerprint sensing area, an optical image of the touched area should be captured by an optical imaging sensor array within the optical sensor module that has high image fidelity to the original fingerprint for robust fingerprint sensing. The optical sensor module may be designed to achieve such desired optical imaging by appropriately configuring the optical elements for collecting and detecting the returned light.

Fig. 22A-22B illustrate an exemplary implementation of the optical collimator design of fig. 20 and 21. The optical collimator array 2001 in this example comprises an optical collimator array 903 and an optically absorbing material filled between the optical collimators 903 to absorb light to reduce cross-talk between different optical collimators. Each collimator 903 in the collimator array 2001 may be a channel extending or elongated in a direction perpendicular to the display panel, and each collimator 903 allows light to be transmitted along its axis with low loss. The collimator array 2001 is designed to reduce optical crosstalk between different optical collimators and maintain a desired spatial resolution in optical sensing. In some implementations, one optical collimator may correspond to only one photodetector in the photodetector array 2002. In other implementations, one optical collimator may correspond to two or more photodetectors in the photodetector array 2002. As shown in fig. 22, in some designs, the axis of each collimator unit may be perpendicular to the display screen surface and may be tilted with respect to the display surface. In operation, only light propagating along the collimator axis carries image information. For example, appropriate incident light 82P is reflected to form light 82R. Light 82R is then diffracted by the TFT apertures and expands to light 82D. The light portion 82S is transmitted into the photodiode array 2002. The light portions 82E away from the axis are absorbed by the fill material. The reflection on the cover glass surface 431 carries fingerprint information. Light 901 that is angled with respect to the collimator unit axis may thus be blocked. A portion of the reflected light, e.g., 901E, is transmitted into a corresponding optical collimator to reach the photodetector array 2002.

Optical collimator arrays can be fabricated by various techniques including etching holes through a planar substrate, forming an optical waveguide array, forming a microlens array that is matched to an optical filter, using a coreless fiber bundle, or printing collimators on a transparent sheet, etc. Desirable features of such a collimator array include: (1) sufficient transmission contrast between the light components propagating along the axis and the components propagating away from the axis, such that the collimator ensures a desired spatial resolution in the optical sensing of the fingerprint pattern at the photodetector array; (2) the allowed transmission numerical aperture is small enough to achieve the desired high spatial resolution of optical sensing.

Various optical collimator array designs may be used. Each optical collimator in the array of optical collimators is configured to spatially filter by transmitting light in a direction along or near an axis of the optical collimator while blocking light in other directions, and has a small optical transmission numerical aperture to achieve high spatial resolution through the array of optical collimators. The small optical transmission numerical aperture also reduces the amount of background light entering the optical detector array. The collimator element aperture and pitch (i.e. the distance between two adjacent collimator elements) may be designed to achieve a desired spatial resolution of the optical fingerprint sensing.

Figure 23 shows an example of a collimator design that is part of a CMOS structure by using aligned holes in two different metal layers in the CMOS structure. In this particular example, each optical collimator in the array is an elongated channel in a direction perpendicular to the display panel.

FIG. 24 illustrates an example of an optical fingerprint sensor module in an LCD display configuration incorporating an optical detector array and an integrated collimator array for each optical sensor pixel in gathering fingerprint information. As shown, the optical detector array includes a photodetector array and a collimator array disposed over the photodetector array to include optically transparent vias as optical collimators and optically opaque metal structures between the vias. Illumination light is directed to illuminate the touch portion of the finger, and light reflected from the finger is directed through an array of optical collimators to an array of photodetectors that capture a portion of the fingerprint image of the finger. The collimator array may be implemented using one or more metal layers with holes or openings integrated by CMOS processes.

Such optical collimators in the underscreen optical sensor module may be configured to provide direct point-to-point imaging. For example, the size of the optical collimator array and the size of the individual collimator may be designed to closely match the size of the photodetector array and the size of the individual photodetector, respectively, to enable one-to-one imaging between the optical collimator and the photodetector. The entire image carried by the light received by the optical sensor module can be acquired simultaneously by the photodetector array at a single photodetector without stitching.

The spatial filtering operation of the optical collimator array may advantageously reduce the amount of background light entering the photodetector array in the optical sensor module. Furthermore, due to the illumination of the light emitted from the OLED pixels, one or more optical filters may be provided in the optical sensor module to filter out background light and reduce the amount of background light at the photodetector array to improve the optical sensing of the returned light from the fingerprint sensing area. For example, the one or more optical filters may be configured as, for example, band pass filters to allow for transmission of the illumination light generated for optical sensing while blocking other light components such as infrared, IR, light in sunlight. Such optical filtering may effectively reduce background light caused by sunlight when the device is used outdoors. The one or more optical filters may be implemented, for example, as an optical filter coating formed on one or more interfaces along an optical path to a photodetector array in an optical sensor module, or may be implemented as one or more discrete optical filters.

FIG. 25 shows an example of an optical collimator array that uses optical filtering to reduce background light reaching a photodetector array in an underscreen optical sensor module. This example uses an optical waveguide array as an optical collimator and one or more optical filter films are coupled to the optical waveguide array to reduce unwanted background light, such as infrared IR light from sunlight, reaching a photodetector array coupled to the optical waveguide array while transmitting desired light in a predetermined spectral band of probe light for illuminating a finger. The optical waveguide may include a waveguide core with or without an external waveguide cladding. The optical waveguide may also be formed by a coreless fiber bundle having different optical fibers, wherein each unit collimator is a fiber sheet having no fiber core structure. When coreless fibers are bundled, the filler material between the fibers may include a light absorbing material to increase the absorption of stray light not guided by the coreless fibers. The final collimator may be assembled with a multi-layered sub-collimator array.

The following sections provide examples of various optical collimator designs and their manufacture.

Fig. 26A and 26B show an example of manufacturing a collimator by etching. In fig. 26A, a layer of suitable material for forming the optical collimators in the collimator array is formed on or supported by an optically transparent support substrate. An etch mask is formed over the layer and has a pattern for etching the underlying layer to form the optical collimator. A suitable etching process is performed to form the optical collimator. The support substrate may be combined with the collimator array, and may be formed of various optically transparent materials including silicon oxide and the like.

Fig. 26B shows an example of an optical collimator array assembled by stacking a multi-layer sub-collimator array via an interlayer connector material (which may be an adhesive, glass, or suitable optically transparent material). In some implementations, different layers of the sub-collimator array may be stacked on top of each other without an interlayer connector material. Such stacking allows the fabrication of optical collimators of a desired length or depth along the collimator axis to achieve a desired optical numerical aperture. The aperture of the collimator geometrically limits the viewing angle. The transmission numerical aperture is determined by the thickness and aperture of the collimator. The holes may be filled with an optically transparent material in some applications, and may be empty in some designs.

In implementations, the support substrate may be coated with one or more optical filter films to reduce or eliminate background light, such as infrared IR light from sunlight, while transmitting desired light in a predetermined spectral band of probe light for illuminating the finger.

Fig. 27 shows an array of optical spatial filters coupled to an array of microlenses, wherein each microlens is positioned relative to a respective through-hole of the optical spatial filter such that each unit collimator includes a microlens and a micro-spatial filter, e.g., a microwell. Each micro-lens is constructed and positioned to focus received light onto a corresponding micro-spatial filter without imaging the received light. The micropores limit the effective acceptance numerical aperture. The spatial filter may be printed on an optically transparent substrate or etched on a silicon wafer. The microlens array may be etched by a MEMS process or a chemical process. The microlenses may also be made of a gradient index material, such as a piece of gradient index glass fiber cut to a quarter pitch length. The focal length of the microlens and the diameter of the spatial filter aperture can be used to control the transmission numerical aperture of each cell. Like other designs, the collimator plate may be coated with a filter film to reduce or eliminate unused bands of light in the sensor, such as infrared IR light from sunlight.

FIG. 28 shows an example of an integrated CMOS photodetector array sensor with built-in light collimation. The collimator is constructed by carding different metal layers 704 and

oxide layers

702, 703, with the metal layers interleaved to provide an array of spaced alignment holes 705. These apertures may be aligned with the photosensitive elements 701 in the optical detector array. An optical fingerprint imager is implemented with such an integrated CMOS photodetector array sensor with built-in light collimation under the LCD display module 701 and cover glass. The fingerprint of a user's finger touching the sensor window area of the cover glass may be imaged by detecting light reflected from the fingerprint valleys and ridges. Light from the ridge areas of the fingerprint is reduced because the light is absorbed by the fingerprint tissue at the ridge areas, compared to the valley areas of the fingerprint, which is stronger. This difference in light level between the ridges and valleys of the fingerprint creates a fingerprint pattern at the optical detector array.

In the above-described collimator-based optical sensor module designs, each collimator may be designed to be large enough to transmit imaging light to a small area on the optical detector array, or may be designed to be small enough to transmit imaging light to a large area on the optical detector array, along the thickness or length of the collimator. When each collimator in the collimator array is reduced to a certain point, e.g. several tens of microns, along the thickness or length of the collimator, the optical field of view of each collimator may be relatively large to cover a portion of the adjacent optical detector on the optical detector array, e.g. a 1mm x 1mm area. In some device designs, optical fingerprint sensing may be achieved by using an array of pinholes, each pinhole in the array of pinholes having a sufficiently large optical field of view to cover a portion of adjacent optical detectors in the array of optical detectors to achieve high image resolution at the array of optical detectors when sensing a fingerprint. The pinhole array may have a thinner size and a smaller number of pinholes than a design using a collimator to achieve the desired high imaging resolution without an imaging lens. Also, unlike imaging via an optical collimator, imaging with a pinhole array uses each pinhole as a pinhole camera to acquire an image, the image reconstruction process based on the operation of the pinhole camera is different from that by using an optical collimator array, namely: each pinhole creates a sub-image area and the sub-image areas of different pinholes in the pinhole array are stitched together to form the whole image. The image resolution of an optical sensor module having a pinhole array is related to the size of the sensitive elements of the detector array, so that the sensing resolution can be adjusted or optimized by adjusting the detector size.

The pinhole array can be relatively simply manufactured at a relatively low cost based on various semiconductor patterning techniques or processes or other manufacturing methods. The pinhole array may also provide a spatial filtering operation to advantageously reduce the amount of background light entering the photodetector array in the optical sensor module. Similar to optical sensor modules designed with optical collimators, due to the presence of illumination for optically sensing the generated illumination light, one or more optical filters may be provided in the optical sensor module with a pinhole array to filter out background light and reduce the amount of background light at the photodetector array to improve optical sensing of the returned light from the fingerprint sensing area. For example, the one or more optical filters may be configured as, for example, band pass filters to allow transmission of the illumination light for optical sensing while blocking other light components such as infrared, IR, light in sunlight. Such optical filtering may effectively reduce background light caused by sunlight when the device is used outdoors. The one or more optical filters may be implemented, for example, as an optical filter coating formed on one or more interfaces along an optical path to a photodetector array in an optical sensor module, or may be implemented as one or more discrete optical filters.

In an optical collimator-based optical sensor module, the optical imaging resolution at the optical detector array may be improved by configuring the optical collimator in a manner that provides a pinhole camera effect. An example of such a design is shown in fig. 29.

In fig. 29, the collimator units 618 of such an optical collimator array direct light from the respective detection area units to a photodetector array 621. The aperture of the collimator unit forms a small field of view (FOV) 618 b. If the detectors in the photodetector array 621 do not capture details in each unit FOV, then the imaging resolution is determined by the FOV of each collimator unit. To improve the detection resolution, the FOV of each collimator unit needs to be reduced. However, when a gap 618a is provided between each photodetector in the photodetector array 621 and the corresponding collimator 618, the small aperture of the collimator unit functions as a pinhole. This pinhole camera effect provides higher imaging resolution in each unit of image of the FOV. When there are multiple detector elements in a unit FOV, as shown in insert 621a, image detail in the unit FOV can be identified. This means that the detection resolution is improved. In implementations, such gaps may be provided in various ways, including, for example, the addition of an optical filter film 618a between the collimator 618 and the optical detector array 621.

By means of the pinhole camera effect, the filling factor of the collimator plate can be optimized. For example, to detect an area of 10mm × 10mm size, if each unit FOV covers an area of 1mm × 1mm, a 10 × 10 collimator array may be used. If in each unit FOV the detector can obtain a 20X 20 resolution image with an overall detection resolution of 200X 200 or 50 microns or 500 psi. This method can be applied to all types of collimation methods.

FIG. 30 illustrates another example of using the pinhole camera effect to improve optical imaging resolution. In this example, the optical sensor module includes a plurality of layers: gasket 917, pinhole array 617, which may be an array of optical collimators of sufficiently small thickness, protective material 919, photodetector array 621, and circuit board 623. The target optical distance is determined by the total material thickness from the sensing surface to the pinhole plane, including the optical thickness of the display module 433, the thickness of the gasket 917, the thickness of any filter coating, any air gap thickness, and any adhesive material thickness. The image optical distance is determined by the total material thickness from the pinhole plane to the photodetector array, including the protective material thickness, any filter coating thickness, any air gap thickness, and any adhesive material thickness. The image magnification is determined by the image optical distance compared to the target optical distance. The detection mode can be optimized by setting an appropriate magnification. Alternatively, the magnification may be set to less than 1, such as 0.7 or 0.5, etc. In some device designs, the shim and pinhole array layers may be combined into a single assembly. In other designs, the pinhole array and the protective layer may be combined into a single component, such that the center coordinates of each pinhole are predefined.

Fig. 31 shows an example of optical imaging based on the pinhole camera effect. On the target side, the entire detection zone 921 on the LCD display panel is divided into a plurality of sub-detection zones 923. A pinhole array 920 is provided for imaging the detection zone 921. Each pinhole unit in pinhole array 920 is responsible for a small field of view (FOV) 925. Each small FOV 925 covers one sub-detection zone 923. As shown in fig. 31, each small FOV of one pinhole may overlap with the small FOVs of its neighboring pinholes. On the image side, each sub-detection region 923 in the optical detector array acquires an image 933. As also shown in fig. 31, each small FOV 925 of the pinhole has a corresponding image region 935. The magnification of the system may be optimized such that the images of each sub-detection area may be distinguished separately. In other words, images of small FOVs do not overlap each other. In the detection mode, the coordinates of the center of each pinhole are predefined, the coordinates of the image points of each LCD display pixel may be pre-calibrated, and this pre-calibration may be used during sensor operation to generate a calibration table for the calibration. In this design, the image of the pinhole camera is inverted and signal processing can recover the entire image based on the calibration table.

In the above examples for optical collimators, the direction for directing light from a finger on top of the display screen to the optical collimators in the optical detector array for fingerprint sensing may be perpendicular to the top touch surface of the LCD display screen to collect probe light for fingerprint sensing returning from the finger, wherein a majority of the returning probe light is in the direction of the light perpendicular to the top touch surface. In practice, by sensing the returning probe light substantially perpendicular to the top touch surface when the touching finger is dry, the image contrast in the image detected in the optical detector array is lower than the image contrast of the same image obtained from the returning probe light angled relative to the perpendicular direction of the top touch surface. This is in part because optical sensing of the angled return light spatially filters out light that is strongly returned from the top touch surface, which is mostly perpendicular to the top touch surface. In view of this aspect of optical sensing of probe light returning from the top touch surface, the optical collimators may be oriented such that the axis of each collimator unit may be tilted with respect to the top touch surface, as shown in the example of fig. 22B.

However, in manufacturing, manufacturing a tilted collimator is more complicated and costly. One way to achieve higher contrast in optical sensing using a vertical optical collimator as shown in fig. 20 and 21B, while also selectively detecting the angled return light from the top touch surface, is to: an optical deflecting or diffractive device or layer is provided between the vertical optical collimator and the returned light from the top touch surface before the returned light enters the vertical optical collimator. In some implementations, such an optical deflecting or diffracting device or layer can be between the OLED display panel and the vertical optical collimator to select only returned probe light at a certain tilt angle to enter the vertical optical collimator for optical detection by the optical detector array on the other end of the vertical optical collimator, while preventing or reducing the amount of returned probe light perpendicular to and from the top touch surface from entering the optical collimator. The optical deflecting or diffracting device or layer can be implemented in various forms, including, for example, a prism array, an optical layer with a diffraction pattern, or other device located between the optical collimator and the display panel to select the angled probe light returning from the display panel to enter the optical collimator while reducing the amount of returning probe light perpendicular to the display panel and entering the optical collimator.

Fig. 32 shows an example of an optical sensor module for optical sensing using an optical pinhole array. As shown, a pinhole array 920a is formed between the LCD display module 433 and the optical photodetector array 621 to image the sensing area pressed by the finger 60 onto the optical photodetector array 621.

The thickness T of the pinhole layer 920a indicates the field of view (FOV) angle. Sensing area FOVs and imaging area FOVi are defined by the distance from the sensing surface to pinhole array 920a and the distance from the imaging plane to pinhole array 920 a. The image magnification is given by Di/Ds, where Di is the thickness of the optically transparent layer 919a between the pinhole array 920a and the optical detector array 621, and Ds is the thickness of the combined stack of the gasket 917, the LCD display module 433, and the top cover glass layer 431. Device parameters such as pinhole layer thicknesses T, Ds and Di may be optimized for a desired combination of FOV and image magnification. For example, if desired, the optical sensor modules may be configured to ideal parameters such that adjacent FOVs of respective adjacent pinholes in array 920a overlap appropriately. Similarly, adjacent FOVi can also be adjusted to overlap or completely separate as discrete FOVi. In an optical sensor module designed to overlap adjacent FOVs with each other, some of the points on the sensing surface may have multiple image points. This feature may be used to enhance detection.

In the example of fig. 32, an optical filter film for reducing background light may be formed or coated on the gasket 917, the pinhole layer 920a, the protection 919a, or the display surface. As shown, when background light 937 is projected onto finger tissue 60, short wavelength components are mostly absorbed and a portion of the long wavelength (e.g., red or infrared) light is transmitted to detector 621. Optical filter films can be used to reject those long wavelength components to improve detection of the returned optical signal carrying finger information.

Fig. 33A and 33B illustrate examples of optical fingerprint sensors under LCD display panels having optical deflection or diffraction devices or layers.

As shown in fig. 33A, each collimator 2001 in the collimator array may be an extended channel along an axis vertical or perpendicular to the display surface. The view angle adapter optical layer 2210 is used to adjust the view angle of the returned probe light from the display panel, and the view angle adapter optical layer 2210 is located between the optical collimator 2001 and the LCD display panel to select the angled probe light returned from the display panel to enter the optical collimator 2001 while reducing the amount of returned probe light perpendicular to the display panel and entering the optical collimator 2001.

Fig. 33B shows more detail of the view angle adapter optical layer 3210 and the primary detection optical path. For example, the viewing angle adaptor optical layer 3210 may be implemented as a diffraction pattern layer such as a prism structure 3210 a. Only the returned

probe lights

82a and 82b from the finger with appropriate incident angles from the display panel can be bent to pass through the collimator 2001. In contrast, the returned probe light perpendicular to the display panel is guided by the viewing angle adapter optical layer 2210 to an original direction away from perpendicular to the display panel, and thus becomes off-axis incident light to the optical collimator 2001. This reduces the amount of returned probe light that is perpendicular to the display panel and that can enter the optical collimator 2001.

When the viewing angle is properly adjusted, the received light from

different locations

63a and 63b of the fingerprint valleys carries the fingerprint information. For example, under the same illumination, light 82a may be stronger than light 82b due to the different viewing angle and fingerprint profile of the fingertip skin. This design allows the optical sensor module to obtain a certain degree of fingerprint shading. This arrangement improves detection when the fingers are dry.

In designing an optical sensor module under an LCD display module, various technical features or characteristics of the LCD display module should be considered and incorporated into the overall optical sensor module design to improve optical sensing operations. The following sections describe several design examples.

One common component in various LCD display modules is a light diffuser, which may be a sheet that diffuses incident light into different directions to achieve large viewing angles and spatial uniformity of the display. However, the presence of the LCD diffuser layer may degrade the optical detection of the optical sensor module under the LCD.

Fig. 34A and 34B show an LCD light diffuser layer 433B positioned between an LCD waveguide layer 433c and other LCD layers 433 a. In some LCD assemblies, the cover glass layer 431 may be separated from the underlying diffuser sheet 433b by a distance (e.g., a few millimeters in some LCD devices), and the optical collimator array 617 is separated from the diffuser sheet 433b by an optical waveguide plate 433c (which may be a secondary micron thick). With this configuration, the strong diffusion in the diffuser sheet 433b can significantly reduce the signal contrast in the signal light passing through the LCD display module 433 to the optical detector array 621. Although light diffusion at the LCD diffuser layer 433b is desirable for display operation, it reduces fingerprint detection performance.

This undesirable effect of the LCD diffuser layer 433b can be mitigated by using different techniques. Two examples are shown in fig. 34A and 34B.

Fig. 34A shows an example in which holes 951a may be made in the corresponding area in the portion of the diffuser sheet 433b above the optical sensor module or the entire diffuser sheet 433b in the LCD display module to improve the transmission of the returning light from the top cover glass 431 to the optical detector array 621. The size, shape and distribution of the apertures may be selected based on the particular design needs. For example, the size of the aperture may be larger than the probe light wavelength to avoid strong diffraction. For example, the collimator element apertures may be about 40 microns in diameter, and the diffuser sheet apertures may be 5 microns, 10 microns, 30 microns, 40 microns, or 100 microns in size, among others. In this design, the inclusion of the holes 951a in the LCD diffuser layer 433b is to establish an optical path for each collimator unit. The aperture of each collimator unit may have one or more holes in the diffuser sheet to provide the desired light path from the top cover glass 431 to the optical detector array 621. If the apertures of the collimator units are discrete at a larger pitch (e.g. around 1 mm), the holes in the diffuser sheet can be drilled at the same pitch. The inhomogeneities in the detection can be calibrated.

Fig. 34B illustrates another example, wherein the diffuser sheet can be configured to include low-diffusion optically transparent dots 951B, wherein light diffusion is weaker in the area above the optical sensor modules to improve transmission of light to the optical sensor modules. The size, shape and distribution of the transparent dots can be selected based on specific design needs. For example, the aperture size may be larger than the probe light wavelength to avoid strong diffraction, and the distribution of the spots may be such that each collimator unit has one or more transparent light paths to allow efficient reception of the returning light from the top cover glass 431 and through the LCD display layer. If the apertures of the collimator elements are discrete at a large pitch (e.g. around 1 mm), the transparent dots in the diffuser sheet can be made at the same pitch. If the diffuser sheet is made of a rough surface material for diffracting or diffusing light, a selected material may be selectively applied to the rough surface to provide some transparent material to reduce the original optical diffusion of the rough surface. Examples of suitable materials include epoxies, waxes, or oils, and these materials can effectively alter diffusion.

For a given LCD diffuser layer, the long wavelength light source may be selected to produce probe or illumination light such that the diffuse scattering for such light is weak so that more light can pass through the diffuser layer to the optical sensor module.

As another example, referring to fig. 35A and 35B, various LCD display modules include an optically reflective layer or film 433d in the LCD below an LCD waveguide layer 433c to reflect unused light back to the LCD layer to improve display brightness. However, the presence of such an optically reflective film 433d may block a majority of light from reaching the optical sensor module below the LCD, and thus may adversely affect optical fingerprint sensing. Such an optically reflective layer may be modified in a manner that maintains the desired optical reflection under the LCD waveguide layer in most locations, while allowing the desired optical transmission at the location of the optical sensor module under the LCD. In some implementations, the collimator module 617 of the optical sensor under the LCD can be fixed to contact the reflector film 433 d.

Fig. 34C illustrates another example for providing a transparent light path for directing light from one or more illumination sources 436 to improve fingerprint sensing of the detection module without being significantly diffused by the diffusion layer. For example, holes 969 may be selectively formed in the light diffuser film 433b to improve light transmission to the optical fingerprint sensor under the LCD. To avoid affecting display performance, the light path holes may be angled to maintain a degree of light diffusion function in the region of the holes 969. In addition, such holes 969 may be designed to be small, e.g., 0.3mm or less, to further enhance diffusion of the backlight while still providing improved optical imaging at the optical fingerprint sensor under the LCD. In some implementations, the optical path holes may be empty, filled with air, or filled with a transparent material.

In some designs, the light path holes 969 may not be limited to a certain area, but may be distributed throughout the light diffuser film 433b, e.g., the holes 969 may be uniformly distributed throughout the film 433 b. This design eliminates undesirable spatially non-uniform illumination of selected holes 969 in certain areas but not others. In some designs, the light path apertures 969 may be distributed in a spatially gradient pattern such that any changes in the LCD illumination caused by the apertures 969 are gradual and less noticeable.

Fig. 35A shows one example for modifying an optical reflector layer, where the optical reflector layer is modified by light transmissive apertures in the area including or forming the optical sensor module locations in the optical reflector film to allow optical reflection for LCD display in most locations of the optical reflector film, while providing a transparent light path for the optical collimator array 617 for receiving light reflected from a finger above the LCD. The size, shape and distribution of the apertures may be configured to meet optical sensing needs. For example, the size of the aperture may be larger than the probe light wavelength to avoid strong diffraction. For example, the collimator element apertures may be about 40 microns in diameter, and the diffuser sheet apertures may be 5 microns, 10 microns, 30 microns, 40 microns, or 100 microns in size, among others. Each collimator cell aperture may have one or more apertures in the optical reflector layer to provide a desired optical path for optical sensing. The inhomogeneities in the detection can be calibrated. If the apertures of the collimator units are discrete at a larger pitch (e.g. around 1 mm), the holes in the diffuser sheet can be drilled at the same pitch.

Fig. 35B illustrates another example for modifying an optical reflector layer in an LCD, where the optical reflectivity of the optical reflector film may be modified to allow a degree of optical transmission for optical sensing by the underlying optical sensor. Various commercial LCD reflector films use flexible plastic materials as substrates, and the light transmission of such plastic materials may be sufficient to transmit sufficient light to the optical sensor module for fingerprint sensing.

In the above-described designs for the LCD diffuser layer and the LCD reflector layer, apertures may be formed in the area where one or more illumination sources are located to allow illumination light to pass sufficiently through the LCD display module layers to the top cover glass for illuminating a finger for optical sensing operations.

In the above design, the optical sensor module is located below the LCD display module and thus below the LCD waveguide layer, which is designed to direct backlight from the backlight light source to the LCD display area. As shown in FIG. 36, backlight from display light sources 434 (e.g., LEDs) is guided by the waveguide 433c and diffused by the LCD diffuser layer to exit the waveguide 433c, providing the backlight required for the LCD. The light may uniformly escape from one side surface of the waveguide 433c and then be diffused by the diffuser sheet 433 b. In some LCDs, approximately half of the diffuse light 957 may propagate towards the collimator 617 and become intense background light in the optical sensing detection.

One or more additional light sources 436 may be provided in connection with the optical sensor module for illuminating the finger and providing light carrying fingerprint pattern information to the optical sensor module below the LCD. Due to the location of the illumination source (e.g., beside the optical sensor or below the reflector film 433d adjacent to the optical sensor), the light guiding function of the waveguide 433c does not work with the light from the illumination source 436, so that the light from 436 can more efficiently reach the top surface of the LCD panel to illuminate the finger.

In addition, the illumination light source 436 may be designed to emit illumination at one or more optical wavelengths that are different from the LCD display illumination light wavelengths from the LCD display backlight light source 434. The illumination source 436 may be used for fingerprint sensing and other sensing functions.

The above-described design for selecting illumination light of one or more optical wavelengths different from the optical wavelength of the backlight of the LCD display may be used to reduce power consumption. Fingerprint detection using a display backlight light source requires turning on the display backlight light source for optical fingerprint sensing. This design consumes more power than the above-described design, where the illuminating light used for optical sensing is partially different from the backlight in the optical wavelength, which allows optical sensing operation without turning on the LCD backlight. The above-described design for selecting illumination light of one or more optical wavelengths different from the optical wavelength of the backlight of the LCD display allows flexibility in the selection of illumination light sources to achieve additional advantages. For example, infrared light may be used as the illumination source 436, making the LCD diffuser layer more transparent to IR illumination light to achieve the desired higher transmission of IR illumination light. As another example, the illumination source may be selected to provide multiple wavelengths for other functions, such as anti-spoofing liveness sensing, heartbeat sensing, and the like.

In designing an optical sensor module under an LCD, the viewing angle can be adjusted using the position and spatial distribution of the illumination sources 436 to optimize the sensing quality.

When the optical sensor module is placed under the LCD module, additional optical design may be used to enhance the backlight transmission from the waveguide layer to the LCD layer while maintaining sufficient transfer of the illumination light for optical sensing to the optical sensor module.

Fig. 37A to 37D show examples of enhancement structures including two or more layers of backlight enhancement films (e.g., 433px and 433py) as part of the LCD layer structure shown at 433 a. The backlight enhancement films 433px and 433py are formed over the light diffuser layer 433 b.

In the example of fig. 37A, each of the enhancement films 433px and 433py includes a polarizing prism structure. The prism groove directions of the two enhancement films 433px and 433py are substantially perpendicular to each other to collectively form a pair of enhancement films to improve the transmission of illumination light to the LCD panel. However, if not properly configured, such functionality of the enhancement film may adversely affect the optical imaging of the optical fingerprint sensor module 621U under the LCD.

As shown in the examples of fig. 37B and 37C, the additional light source illumination direction and detector viewing direction may be specifically configured not along the prism groove direction of the enhancement films 433px and 433py to reduce the adverse imaging effect of the enhancement films on optical fingerprint sensing. This design is intended to obtain a clear image without perforations in the enhancement film. The viewing angle φ 1 and the illumination angle φ 2 should be adjusted according to the design of the enhancement film.

The example in fig. 37D shows a specific design of the collimator unit 617U and the photodetector array unit 621U. The collimator unit 617U is used to provide an imaging function implemented by a micro-lens, a pinhole, or a combination of both. A larger sensing area at the photodetector array unit 621U may be achieved by optimizing a single detection cell design or by using multiple detection cells.

Fig. 38 shows an example of an optical waveguide layer in an LCD module that includes a partially transparent portion in the detection light path to allow improved light transmission of the illumination light for optical sensing through the waveguide layer.

Fig. 39A to 39C show examples of illumination light sources designed for optical sensing in an optical sensor module under an LCD display module. In an LCD display module, an optical reflector layer enhances LCD display brightness by recycling unused backlight to the LCD layer. In this regard, defects in optical reflectivity along the optical reflector film, such as mechanical defects in the reflector film, can result in visible changes in the brightness of the LCD display, and are therefore undesirable. Fig. 39A-39C illustrate design features for reducing the adverse effects of defects in a reflector layer or film.

As shown in fig. 39A, micro-holes 973 may be provided in the reflector layer 433d at the location of the illumination light sources 436 for visible light components in the illumination light. Such visible light components are used to provide illumination in a limited area of the display to display the necessary text or identification information without turning on the display backlight.

Another solution, as shown in fig. 39B, is to select the illumination source wavelength so that it no longer falls within the operating band of the reflector film, which is typically in the visible band. The illumination source 436 for optical sensing may be outside the reflective spectral range of the reflector film, such as short wavelength range below 400nm (e.g., 380nm) or long wavelength range beyond the visible red range (e.g., 780nm, 900nm, 940nm, etc.), so that illumination light may pass through the reflector film or layer without forming holes in the reflector film.

Fig. 39C shows yet another design in which the reflector film is designed to include a narrow-band transmissive window 975 for transmitting illumination light for optical detection. Such a narrow transparent or transmission window in the reflector film may be between 525nm and 535nm, for example.

Portable devices or other devices or systems based on the optically sensed mobile phones and the like disclosed in this document may be configured to provide additional operational features.

For example, the LCD display panel may be controlled to provide a local flash mode to illuminate the fingerprint sensing area by operating selected LCD display pixels below the sensing area. This may be provided in an optical sensor module under the LCD display panel, for example, based on the optical imaging design of fig. 4A and 4B, or on the optical imaging of fig. 21A and 21B by an optical collimator array. In the case of acquiring a fingerprint image, the LCD display pixels and illumination light sources in the sensing window area may be turned on temporarily to produce high intensity illumination for optical sensing of the fingerprint, and at the same time the photo detection sensor array 621 is turned on to acquire a fingerprint image synchronized with the turning on of the illumination light. The time for turning on the illumination light may be relatively short, but the emission intensity may be set high. For this reason, such a mode for optical fingerprint sensing is a flash mode, which enables the photodetector sensor array 621 to detect a greater amount of light to improve image sensing performance.

The above disclosed optical sensor for sensing an optical fingerprint may be used to acquire a high quality image of a fingerprint, so that it is possible to distinguish small changes in the acquired fingerprint acquired at different times. It is worth noting that when a person presses the device with a finger, the contact with the top touch surface on the display screen may change due to the change in pressing force. When a finger touches a sensing area on the cover glass, the change in touch force may cause the optical detector array to undergo several detectable changes: (1) fingerprint deformation, (2) change in contact area, (3) fingerprint ridge widening, and (4) hemodynamic changes at the compressed area. These changes can be optically collected and can be used to calculate a corresponding change in touch force. Touch force sensing adds more functionality to fingerprint sensing.

Referring to fig. 40, the contact profile area increases with increasing pressing force, while the ridge impression expands with increasing pressing force. Conversely, the contact profile area decreases with decreasing pressing force, while the ridge impression contracts or contracts with decreasing pressing force. Fig. 40 shows two different fingerprint patterns of the same finger under different pressing forces: lightly pressed fingerprint 2301 and heavily pressed fingerprint 3303. The returning probe light from the selected integrated area 3305 of the fingerprint on the touch surface can be collected by a portion of the optical sensors on the optical detector array that corresponds to the selected integrated area 3305 on the touch surface. The analysis of the detection signals from those optical sensors to extract useful information is explained further below.

When a finger touches the sensor surface, the finger tissue absorbs the optical power, and therefore the received power integrated on the photodiode array is reduced. Especially in the total internal reflection mode, which does not sense low refractive index materials (water, sweat, etc.), the sensor can be used to detect whether a finger touches the sensor or other target accidental touches the sensor by analyzing the received power variation trend. Based on this sensing process, the sensor can determine whether the touch is a true fingerprint touch, and thus can detect whether to wake up the mobile device based on whether the touch is a true finger press. Because the detection is based on integrated power detection, the light source for optical fingerprint sensing is in a power saving mode.

In a detailed fingerprint map, as the pressing force increases, the fingerprint ridge expands, and more light is absorbed by the expanded fingerprint ridge at the touch interface. Thus, within a relatively small observation region 3305, the integrated received optical power change reflects the change in pressing force. Based on this, the pressing force can be detected.

Thus, by analyzing the received probe optical power variations integrated within a cell, the time domain evolution of the fingerprint ridge pattern deformation can be monitored. Then, the information on the temporal evolution of the deformation of the fingerprint ridge pattern may be used to determine the temporal evolution of the pressing force on the finger. In an application, the time-domain evolution of the pressing force of the human finger may be used to determine the dynamics of the user interaction by the touch of the finger, including determining whether the human is pressing or moving the pressing finger away from the touch surface. These user interaction dynamics may be used to trigger certain operations of the mobile device or operations of certain applications on the mobile device. For example, the temporal evolution of the pressing force of the human finger may be used to determine whether the human touch is an intended touch or an unintended touch to operate the mobile device, and based on such determination, the mobile device control system may determine whether to wake up the mobile device in a sleep mode.

Furthermore, a live human finger in contact with the touch surface may exhibit different characteristics of extinction ratios obtained at two different probe light wavelengths at different pressing forces, as illustrated in fig. 14 and 15. Returning to fig. 40, a lightly pressed fingerprint 3301 may not significantly restrict blood flow into the pressed portion of the finger, thereby producing extinction ratios obtained at two different probe light wavelengths that are indicative of living human tissue. When a person presses a finger hard to produce a heavily pressed fingerprint 3303, the blood flow to the portion of the pressed finger may be severely reduced, and therefore the corresponding extinction ratios obtained at the two different detection light wavelengths will be different from those of the lightly pressed fingerprint 3301. Therefore, the extinction ratios obtained at two different probe light wavelengths vary under different compression forces and different blood flow conditions. This variation is different from the extinction ratios obtained at two different probe wavelengths resulting from pressing with different forces of a false fingerprint pattern of an artificial material.

Thus, the extinction ratios obtained at two different probe light wavelengths can also be used to determine whether a touch is from a user's finger or other object. Such a determination may also be used to determine whether to wake up the mobile device in sleep mode.

As another example, the disclosed optical sensor technology may be used to monitor natural motion of a person's fingers due to natural movement or motion of a living person (intentionally or unintentionally) or a heartbeat-related pulse as blood flows through a person's body. The wake-up operation or user authentication may enhance access control based on a combination of optical sensing of the fingerprint pattern and positive determination of the presence of a living person. As another example, the optical sensor module may include sensing functionality for measuring glucose levels or oxygen saturation based on optical sensing of returned light from the finger or palm. As another example, when a person touches the display screen, changes in the touch force can be reflected in one or more ways, including fingerprint pattern distortion, changes in the contact area between the finger and the screen surface, widening of fingerprint ridges, or dynamic changes in blood flow. These or other variations can be measured by optical sensing based on the disclosed optical sensor technology and can be used to calculate touch force. Such touch force sensing can be used to add more functionality to the optical sensor module than fingerprint sensing.

The features associated with the above-described under-screen optical sensing in connection with fig. 3A-40 may be implemented as a device with a dual-zone display having a main display zone and a peripheral display zone, as well as a device with a single-zone display without a peripheral display zone. Fig. 41A, 41B and 42 show examples of specific implementations of an optical sensor module placed under an LCD display of a device having an LCD display module to provide a main display area and a peripheral display area in the same LCD panel.

Fig. 41A shows an example of placing an LCD lower optical sensor module under an LCD module in a peripheral display area. For optical fingerprint sensing, one or more light sources 436a are placed below the LCD module 433 within the peripheral display area 20 to emit a probe beam 967a through the LCD module 433 and the top transparent layer 431 to the top sensing surface in the peripheral display area 20 for optical sensing or other characteristics of the fingerprint. The probe beam 967a may be a collimated beam. In operation, probe light beam 967a at or near the top sensing surface interacts with finger 447, and reflected and/or scattered probe light caused by finger 447 is directed back into the top transparent layer 431 as probe signal light beam 965a, which carries fingerprint information and other information about the finger, e.g., fingerprint patterns or maps, blood flow, heart beat, etc. In this example, there is no optical lens in the design of the receiving optics used to collect and route probe signal beam 965a from the top transparent layer 431 to the optical detector array 621 b. Because the probe signal beam 965a is reflected from the cover glass surface of the top transparent layer 431, a fingerprint absorption map or pattern is added to the probe signal beam 965a and can be detected by the photodetector array 621 b. The photodetector array 621b is oriented with its detection surface parallel to the cover glass surface of the top transparent layer 431 and other layers. This arrangement forms a lens-less imaging projection configuration in which the fingerprint map is projected two-dimensionally onto the detector array 621b at some magnification. For probe light rays entering detector array 621b at different angles of incidence, the optical distortion is typically low and the detected image contrast is typically high.

FIG. 41B shows another example of a lens-less imaging projection configuration by placing an optical sensor module under an LCD under the LCD module within a peripheral display area. Unlike the design in FIG. 41A, the design in FIG. 41B orients the photodetector array 621c at an angle relative to the cover glass surface and other layers of the top transparent layer 431 layer. This tilted orientation of photodetector array 621c causes the image carried by returning probe signal beam 965a to be compressed in a direction along the surface of the photodetector array, which may reduce the size of the photodetector array pixels along that direction, and correspondingly reduce the overall cost of photodetector array 621c as compared to photodetector array 621b having a larger detector pixel or size.

The above-described lens-less imaging projection configuration for the optical sensor module under the LCD in fig. 41A and 41B can also be used for the optical sensor module placed under the single-area LCD panel. FIG. 4B shows an example of a photodetector array 702 under a single-region LCD panel using such an oblique orientation.

Referring to the optical sensor module designs in fig. 6B-11, the receiving optics of the optical sensor module under the LCD may also include one or more imaging lenses for optical sensing. Fig. 42 shows an example of an LCD lower optical sensor module disposed under an LCD module within a peripheral display area in a microphotograph configuration including a lens as part of receiving optics. In this example, the photodetector array 621d is positioned with its detector array surface perpendicular to or at an angle relative to the cover glass surface of the top transparent layer 431 and the other layers. Mirror 617p is placed in the optical path of returned probe signal beam 965b caused by probe beam 967b from probe light source 436b below the LCD module to redirect returned probe signal beam 965b to be parallel to or at an angle to the direction parallel to the cover glass surface and other layers of top transparent layer 431. Lens 617q is placed in the optical path of the reflected light beam from mirror 617p, and the lens 617q projects the reflected light beam from mirror 617p along the cover glass surface of top transparent layer 431 and other layers onto photodetector array 621 d. In such microphotocamera designs, the lens 617q produces an optical image of the reduced fingerprint pattern with higher light intensity, and such gain of the image lens 617q allows for a reduction in the overall level of detected optical power, or such gain improves signal-to-noise performance at optical sensing. In addition, because lens 617q produces a reduced optical image of the fingerprint pattern at photodetector array 621d, the micro-camera can be implemented by using a small size photodetector array as photodetector array 621 d. This reduces the cost of the photodetector array 621 d. Also, in this design, since there is a mirror 617 for changing the direction of the received probe beam to be along the cover glass surface of the top transparent layer 431 and other layers, and since the interval required for the imaging operation of the imaging lens 617q in the beam propagation direction is along the cover glass surface of the top transparent layer 431 and other layers rather than perpendicular to the cover glass surface of the top transparent layer 431 and other layers, the required interval can be adjusted. Such an optical arrangement with imaging lens 617q may be used to keep the thickness or size of the device small in the direction perpendicular to the cover glass surface of top transparent layer 431 and other layers, without compromising the proper imaging conditions of imaging lens 617 q. Similarly, the optical design for the LCD lower optical sensor module shown in fig. 6B-11 can maintain a relatively small thickness in the direction perpendicular to the display screen.

Electronic devices with displays are often used in bright environments with varying degrees of ambient or background light caused by natural light (e.g., natural sunlight from the sun) or illumination lights (e.g., in well-lit office facilities or stadiums). Such ambient or background light can adversely affect the detection of the optical sensor module under the LCD and should be suppressed. As described above, the background reduction technique can be provided in the off-screen optical sensor module by performing certain control and signal processing, as shown in the two examples shown in fig. 12 and 13. In addition, one or more additional optical design features may be added to the optical sensor module design disclosed above to filter or add additional illumination sources to reduce the background light based on the background light. Different backlight reduction techniques based on operational control/signal processing, optical filtering, and adding additional illumination sources may be incorporated in implementations in various ways.

In the apparatus having the LCD display, one or more detection light sources are provided to generate detection light for the optical sensor module under the LCD, and such detection light sources are separated from the backlight light source for the LCD display to display an image, such as the backlight module 434 or the designated peripheral display

area illumination module

436b or 436c that illuminates the peripheral display area 20. Thus, the optical detection power for optical sensing can be controlled independently of the LCD illumination to provide sufficient illumination for optical sensing to improve the optical detection signal-to-noise ratio and counteract ambient light effects. For example, one or more detection light sources may blink for a short time at high output power during fingerprint sensing to obtain optimal detection. For example, one or more of the detection light sources may be modulated to provide improved optical sensing performance without affecting the display function. Additionally, one or more probing light sources may provide flexibility in determining whether a detected finger is a live finger to deter malicious attempts to bypass fingerprint detection with false fingerprints. For example, multiple probing light sources may be arranged to include light sources with sufficiently different optical wavelengths at which the finger exhibits different optical characteristics for live finger detection, e.g., using the green and near IR LEDs for live finger detection explained with reference to FIGS. 14 and 15, where the finger tissue strongly absorbs the green light so that the finger image exhibits the desired large brightness gradient and the near IR light is all illuminated through the finger so that the finger image brightness appears more uniform.

Optical filtering techniques for reducing background light can be implemented with the various optical sensor module designs disclosed in this document. While the general goal of inserting optical filters in the optical path of the optical sensor module is to reject ambient light wavelengths, e.g., near infrared IR light and portions of red light, as well as other undesirable wavelengths, the specific implementation of such optical filters may vary based on the specific needs of each application. Such optical filters may be formed by forming an optical filter coating on selected surfaces of the optical components in the optical path to the optical detector array, including, for example, the display bottom surface, surfaces of other optical components such as optical prisms, upper sensor surfaces of the optical detector array, and the like. For example, a human finger absorbs most of the wavelength of energy at a particular wavelength (e.g., about 580nm), and if the optical filter is designed to reject light from wavelengths near about 580nm into the infrared, the undesirable effects of ambient light can be greatly reduced.

Fig. 43 shows an example of the optical transmission spectral profiles of a typical human thumb and pinky at several different optical wavelengths from about 525nm to about 940 nm. For short wavelengths, e.g., wavelengths less than 610nm, less than 0.5% of the ambient light may pass through the finger. Light of longer wavelength such as red light and near IR light has higher transmittance. Due to scattering of finger tissue, the transmission direction range of the ambient light passing through the finger is enlarged, so that the ambient light can be mixed with the signal light to be detected by the under-screen optical sensor module. When operating in sunlight, since the optical power of sunlight is high, unwanted ambient light from sunlight can be carefully treated to reduce or minimize adverse effects on the performance of the optical fingerprint sensor.

Fig. 44 shows the influence of background light in the off-screen optical sensor module 4400. Undesired ambient light that may adversely affect optical fingerprint sensing may travel different paths to reach the optical fingerprint sensor 4400: a certain amount of ambient light (e.g., light 4410) enters the optical fingerprint sensor 4400 (type 1) through the finger via the top transparent layer 431 (e.g., light 4410a) and the LCD module 433 (e.g., light 4410b), and a certain amount of ambient light (e.g., light 4420) does not pass through the finger but enters the optical fingerprint sensor 4400 (type 2) from one or more sides around the finger, e.g., light 4420a in the top transparent layer 431 and light 4420b in the LCD module 433.

In the illustrated under-screen optical sensor module 4400 for fingerprint sensing, a sensor package cover or housing 4402 may be formed on the exterior of the under-screen optical sensor module 4400 and may include an optically opaque or absorptive material as a background barrier, at least to block some incident background light incident at large angles that does not pass through a finger but enters the optical fingerprint sensor 4400 from one or more sides around the finger.

For type 1 ambient light 4410 propagating through the finger, the finger absorbs some of the incident light 4410, such that a portion of the type 1 light 4410a passes through the finger to the cover glass 431 and then through the cover glass 431 to the LCD layers in the LCD module 433. A portion of this type 1 light 4410b passes through the LCD layers into the optical fingerprint sensor package cover 4400/4402 and typically includes incident ambient light at a large angle of incidence.

The optical fingerprint sensor package 4402 may be designed to receive a large portion of the incident light from above the optical window formed by the sensor package 4402 by providing an optical window on the top to spatially block some of the incident ambient light at large angles from entering the optical fingerprint sensor 4400, such that only the top side of the optical sensor module 4400 that is bonded (e.g., adhered) to the bottom of the LCD module 433 is open to receive light, and the sensor bottom and side walls are not optically transparent, such that ambient light that may enter the optical fingerprint sensor is reduced. Thus, for ambient light that is not initially transmitted through a finger but enters the optical sensor module (e.g. type 2 ambient light), the package cover of the optical sensor module may be designed to provide absorption or blocking of such light with light blocking sidewalls or a suitably designed optical receiving aperture, such that the light is absorbed or blocked when it reaches the receiving optics material or encapsulation material.

The undesired ambient light may include different wavelength components, and thus such different ambient light components may be treated differently to reduce their impact on optical fingerprint sensing when implementing the disclosed techniques. For example, the undesirable ambient light may include (1) light components that are transmitted through the finger at red wavelengths (e.g., greater than 580nm) and longer wavelengths (e.g., infrared light) and (2) light components that are not transmitted through the finger at wavelengths shorter than the red wavelengths (e.g., less than 580 nm). Due to the wavelength dependent absorption of the finger shown in fig. 43, the transmitted ambient light through the finger typically includes some near IR light and some of the red light components. Accordingly, optical filtering may be included in the optical fingerprint sensor package to filter out unwanted ambient light that would otherwise enter the optical detector array.

An example design is to use one or more IR blocking filter coatings, such as an IR-cut filter coating, to reduce IR light or near IR light in the transmitted light from the finger. However, various IR-cut filters for imaging devices typically only limit wavelengths longer than 710 nm. Such filtering properties may not be sufficient to reduce IR background light upon optical fingerprint sensing when the device is exposed to direct or indirect sunlight. In some applications, a suitable IR filter coating should extend the short end cut wavelength to shorter wavelengths below 710nm, such as 610 nm.

A single IR cut filter with an operating band extending to shorter wavelengths may not provide the desired IR blocking performance due to the spectral response of the various IR blocking or cut coatings. In some filter designs for an under-screen optical sensor module, two or more optical filters may be used in combination to achieve the desired IR blocking performance in the sensor optical path. Part of the reason for using such two or more filters is the presence of strong background light from the natural sunlight of the sun, which is an important technical problem. In the disclosed example of an optical sensor under an LCD module, an optical filtering mechanism may be built into the under-screen optical sensor stack to block or reduce strong background light from natural sunlight from the sun entering the optical detector array 600 a. Thus, one or more optical filter layers may be integrated into the underscreen optical sensor stack above the optical detector array to block unwanted background sunlight from the sun while allowing illumination light for optical fingerprint sensing to pass through to the optical detector array.

For example, in some implementations, one or more optical filters for reducing unwanted IR ambient/background light may be placed between the LCD module and the optical detector array to optically transmit the returned probe light (e.g., probe light having a wavelength between 400nm and 650nm or other probe wavelengths) while blocking light having optical wavelengths greater than 650nm, including strong IR light in sunlight. Indeed, some commercial optical filters have transmission bands that may not be needed for the particular application of the under-screen optical sensor disclosed in this document. For example, some commercial multilayer bandpass filters can block light at wavelengths above 600nm, but have transmission peaks in the spectral range above 600nm, such as the optical transmission band between 630nm and 900 nm. Strong background light in sunlight within such optically transmissive bands may pass through to the array of optical detectors and adversely affect optical detection for optical fingerprint sensing. By combining two or more different optical filters having different spectral ranges together, an undesired optical transmission band in such optical filters may be eliminated or reduced, such that combining two or more filters together eliminates or reduces an undesired optical transmission band at wavelengths between 630nm and 900nm, wherein the undesired optical transmission band in one filter may be in the optical blocking spectral range of the other optical filter. Specifically, for example, two filters may be combined by using one filter to filter light having a wavelength from 610nm to 1100nm while transmitting visible light having a wavelength below 610nm, and another filter to filter light in a spectral range having a wavelength shifted from 700nm to 1100nm while transmitting visible light having a wavelength below 700 nm. Such a combination of two or more optical filters may be used to produce a desired rejection of background light for optical wavelengths greater than the higher transmission wavelength. Such optical filters may be formed at different positions of the layer between the LCD module and the optical sensor module.

In some implementations, when two or more optical filters disclosed above are used, an optically absorptive material can be filled between the two filters to properly absorb the rejected light band so that light reflected back and forth between the two optical filters can be absorbed. For example, a blue glass sheet having a high absorption of wavelengths from 610nm to 1100nm may be used as a substrate for the filters, both filters are coated on the upper and lower surfaces of the blue glass, and such a component may be used as a spacer or protective material over the optical sensor module.

In addition to utilizing appropriate optical filtering in the under-screen optical sensor module for cutting off background light in the red and IR ranges, the background light may also include light in the shorter wavelength spectral range (including UV wavelengths). In some implementations, ambient light in the UV band should be reduced or eliminated because the band of light produces noise upon optical sensing. Such UV elimination may be achieved by using one or more layers of UV cut optical coatings or UV absorbing materials in the detection light path above the optical sensor module. Finger tissue, silicon, black ink, and the like tend to strongly absorb UV light. Material absorption of UV light may be used to reduce the effect of UV light on optical fingerprint sensing.

Fig. 45 shows an example of a design algorithm for designing optical filtering in an under-screen optical sensor module to reduce background light in the light described above. Thus, in addition to designing a suitable optical filter into the optical path of the optical sensor module, additional design features for reducing background light may also be added to the design of the receiving optics of the optical detector array in the optical sensor module. These optical filtering and further reduction of background light, considered via operational control and signal processing, may be combined in operating such an optical sensor module to achieve a desired optical sensing performance.

In an underscreen optical sensor module having an optical collimator array or an optical pinhole array in front of an optical detector array, the optical collimator array or the optical pinhole array is part of the receiving optics and may be designed with a small optical numerical aperture to reduce background light entering the optical detector array.

Fig. 46 shows two examples of fig. 46A and 46B.

Referring to fig. 46A, collimator pinhole 4651 may be designed to be optically transparent in the probe band of light, and collimator wall material 4653 may be selected to absorb light reaching the wall 4655. If the collimator material is silicon, a blackened light-absorbing coating can be formed on each wall.

Referring to fig. 46B, a pinhole array of pinholes 4659, which is part of the receiving optics, may be configured with an effective numerical aperture to block ambient light with large angles of incidence. A light blocking layer having an array of aperture-limiting holes 4661 may be formed below the array of pinholes 4659 so that light 4667 outside the effective numerical aperture may be blocked by opaque portions of the light blocking layer having the aperture-limiting holes 4461. The

materials

4663 and 4665 forming the imaging camera pinhole 4659 and the aperture-limiting holes 4661 may be opaque materials or optically absorptive materials, such as black ink or optically reflective materials like metal films.

In some implementations, one or more optical filters may be used as a substrate for supporting a pin-hole machine type optical device, such that multiple functional parts may be combined or integrated into one piece of hardware. Integration or combination of such different backlight reduction mechanisms may reduce device cost and may also reduce device thickness.

The underscreen optical sensor module may also operate with a sensor initialization process to reduce the effects of unwanted background light. Such a sensor initialization process is intended to operate in nature, as with the techniques shown in fig. 12 and 13. Fig. 47 shows an example of such a sensor initialization process that measures the baseline background level at the optical detector array each time a fingerprint is obtained. Before actual fingerprint sensing is performed, illumination light or optical detection light for optical sensing is turned on in a darkroom environment without any influence of ambient light, and a finger simulator device is placed on a cover glass to record image data. The finger simulator device is designed to simulate finger skin reflection behaviour, but without any fingerprint pattern. Image data obtained from the finger simulator device is saved as base 1 data into a memory to perform backlight reduction processing in a real sensing operation. This process may be a device calibration process that is done at the factory before shipping the device.

In real-time fingerprint sensing, there are environmental effects. In operation, the illumination light or optical probe light is first turned off to record the image data as a base number 2, the base number 2 being obtained under ambient light conditions. This base number 2 represents the total effect of all ambient light residues. The sum of radix 1 and radix 2 is the real-time radix. Next, the illumination light or the optical detection light is turned on to perform fingerprint sensing to collect a real-time signal that mixes the real fingerprint signal from the fingerprint and the real-time base. A difference between the signal mix and the real-time base may be made as part of the signal processing to reduce the signal contribution from ambient light so that the image quality of the fingerprint image may be obtained.

The above example of fig. 47 illustrates a method for operating an electronic device capable of detecting fingerprints by optical sensing by operating an optical sensor module below a touch display panel that provides the device with touch sensing operations to generate probe light that illuminates a top transparent layer of the touch display panel to operate an optical detector array within the optical sensor module to obtain a first image of returned probe light from the top transparent layer. The method includes operating an optical detector array within the optical sensor module while turning off the probe light to obtain a second image under illumination in which only ambient light and no probe light illuminates a top transparent layer of the touch display panel; and processing the first image and the second image to remove the effect of ambient light in the image operation of the device.

Based on the foregoing, various techniques may be used to mitigate the undesirable effects of background light on the performance of the underscreen optical sensor module, including implementing optical filtering in the optical path to the optical detector array to reduce background light, designing receiving optics for the optical detector array to reduce background light, or controlling the operation of the optical sensor module and signal processing to further reduce the effects of background light on the optical sensing performance. These various techniques may be used alone or in combination to meet desired device performance.

The above examples of techniques for reducing ambient or ambient light are typically based on blocking or filtering the ambient or ambient light from reaching the optical sensor module under the LCD. However, a portion of the background light that reaches the top glass surface at the fingerprint sensing area through the finger may carry fingerprint information and may therefore be used for fingerprint sensing. Referring back to fig. 44, wherein the wavelength of the type 1 background light 4410b transmitted through the finger and a portion of such background light 4410b at the top of the optical sensor module 4400 is between 540nm and 950nm as shown in fig. 43. The background light 4410b at the top of the optical sensor module 4400 can be detected for optical fingerprint sensing because background light with wavelengths between 650nm and 950nm can be transmitted into the finger tissue and propagate through the stratum corneum of the finger skin to imprint fingerprint information on the background light by internal structural changes inside the finger skin caused by the ridge and valley areas of the finger and such internal structural changes appear as light signals with different intensities as fingerprint patterns caused by finger tissue absorption, refraction and reflection, finger skin structure shadowing, and/or optical reflection differences at the finger skin, so the background light 4410b at the top of the optical sensor module 4400 can be detected.

This technique of using transmitted light having a wavelength between 650nm and 950nm through finger tissue to obtain fingerprint information is different from the technique based on reflected light caused by surface topography patterns formed by finger ridges and valleys outside the finger as shown in fig. 5A, 5B and 5C. When relying on surface reflection at the finger as shown in fig. 5A, 5B and 5C, the presence of water or sweat between the finger and the top glass surface ("wet finger contact condition") or other material such as dirt or debris ("dirty finger contact condition") can alter the optical reflection and cause undesirable changes in image contrast, thereby adversely affecting fingerprint sensing performance. One of the advantages of optically transmitted light using a fingerprint pattern obtained via structural changes inside the finger is that the optical fingerprint sensing is less sensitive to the surface contact condition of the finger in contact with the top glass surface, compared to optical fingerprint sensing based on reflected light caused by surface topographic patterns formed by the ridges and valleys of the finger outside the finger as shown in fig. 5A, 5B and 5C.

There are technical challenges to using such background light for optical fingerprint sensing. For example, the background light may be intense light, such as sunlight, and may saturate optical detectors in the optical sensor array 4400. However, it is to be appreciated that a device having an LCD display may also be configured, such as an LCD layer with liquid crystal cells, to exhibit certain optical transmission characteristics that help mitigate the adverse effects of ambient or ambient light upon LCD optical sensing, while utilizing ambient light 4410b at the top of the optical sensor module 4400 with an optical wavelength between 650nm and 950nm for fingerprint sensing. For example, an LCD layer with front and back optical polarization layers on two opposite sides of the LC cell layer may be designed to selectively block or reduce optical transmission of some background light in the visible and UV spectral ranges to the LCD lower optical sensor module while allowing a portion of the background light 4410b at the top of the optical sensor module 4400 with optical wavelengths between 650nm and 950nm for fingerprint sensing. As a specific example, an LCD layer with front and back optical polarizing layers on opposite sides of the LC cell layer may be designed to block visible light or light having a wavelength less than 650nm while still transmitting longer wavelengths, such as 800nm or longer, when the LC cell is turned to an off mode to block optical transmission. Such operation of the LCD module in a non-transmissive off mode at the LC cell may be used to reduce the amount of background light reaching the optical sensor module 4400, while allowing the optical sensor module 4400 to use a portion of the background light 440b carrying a fingerprint for optical sensing. With this design, the detection light source 436a for optical sensing should be designed to emit detection light in a longer wavelength range, wherein the LC cell in the off mode to prevent optical transmission may still be transparent to allow transmission of longer wavelength light, such as 800nm or longer. For example, the detection light source 436a for optical sensing may be designed to emit detection light of wavelength 850nm to illuminate a finger through the LCD layer in the off mode and return to the optical sensor module 4400 through the LCD layer in the off mode. With this design, when the optical sensor module 4400 and the probe light module 436a are activated for fingerprint detection or other optical sensing, the respective LCD pixels in the peripheral display area 20 or the fingerprint sensing area of the taskbar may be turned off such that the liquid crystal material and the reflector film block a portion of the light in the visible and near infrared IR bands. This increases the total amount of signal light carrying fingerprint information while reducing the amount of background light at the optical sensor module 4400.

In summary, for optical sensing under LCD, the probing light source 436a for optical sensing may be designed to emit probing light in a longer wavelength range between 650nm and 950nm for fingerprint sensing, e.g., 800nm or longer, to allow optical transmission through finger tissue, such that (1) the light used to obtain transmission of a fingerprint pattern via internal structural changes of the finger as described above and (2) the reflected probing light caused by the surface topography pattern formed by ridges and valleys of the finger outside the finger as shown in fig. 5A, 5B and 5C may be used for optical fingerprint sensing. For example, the detection light source 436a for optical sensing may be designed to emit detection light with a wavelength of 850nm to obtain fingerprint information based on two mechanisms, reducing the effect of an undesired wet or dirty finger contact condition due to the presence of transmitted detection light at the optical sensor module.

Fig. 48 shows an example of a device using an under-LCD optical sensor module based on probe light having a wavelength between 650nm and 950nm that is transparent to finger tissue for fingerprint sensing. Two different

detection light sources

436a and 4661 for optical sensing are provided at two different positions on the top glass cover 431, the two different

detection light sources

436a and 4661 being specifically designed to generate detection light at the same wavelength between 650nm and 950nm for fingerprint sensing. The first detection light source module 436a is disposed adjacent to the optical sensor module 4820 under the LCD module 433 to generate the first detection light directed to the fingerprint sensing area on the top glass cover 431 upward at a relatively small incident angle, thereby generating a reflected detection light signal as shown in fig. 5A, 5B and 5C. The first probe light also generates return probe light caused by a portion of the first probe light transmitted into the finger. This portion of the first probe light inside the finger is scattered back by the finger tissue and passes through the finger again, thereby being printed with a fingerprint pattern by passing through the surface of the finger in contact with the top glass cover 431. These two portions of the first probe light are directed by the receiving optics of optical sensor module 4820 to the optical detector array for optical sensing. The first detection light is not shown in fig. 48.

The second probing light source module 4661 is located under the top glass cover 431 but away from the fingerprint sensing area on the top glass cover 431, so that the second probing light source module 4661 generates second probing light 4661a and directs the second probing light 4661a to the fingerprint sensing area on the top glass cover at a large incident angle, therefore, the optical reflection of the second probing light at the surface does not enter the optical sensor module 4820 to a large extent. However, a part of the second detection light 4461a at the surface penetrates into the finger, and the part of the second detection light 4461a inside the finger is scattered back by the finger tissue as 4661b and passes through the finger again, thereby being printed with a fingerprint pattern by passing through the surface of the finger in contact with the top glass cover 431. This signal probe light 4661b caused by second probe light 4661 is directed by the receiving optics of optical sensor module 4820 to an optical detector array for optical sensing.

Thus, optical sensor module 4820 receives not only the probe light signals from two different probe light sources, but also the same fingerprint pattern information from two types of optical finger interactions: (1) surface reflections of the ridges and valleys of the finger and (2) interaction with the external structures of the finger associated with the ridges and valleys of the finger. This results in improved optical sensing in a device having an LCD panel configured to provide a main display area and a peripheral display area in the same LCD panel or in a device having an LCD panel configured as a single area display.

Fig. 49 shows a specific example of implementing the probe light design of fig. 48 to the apparatus example of fig. 42 with an LCD display configured to provide a main display area and a peripheral display area in the same LCD panel. Two different

detection light sources

436a and 4661 are provided for optical sensing in the example of the apparatus in fig. 42 with mirror 617p and lens 617q as part of the receiving optics and

light sources

436a and 4661. The first probing light source module 436a is placed adjacent to the optical sensor module 4820 under the LCD module 433 to generate two types of probing light signals: (1) reflected probe light signals (dashed lines) from the outer surface of the finger as described and labeled in fig. 42 and (2) return probe light signals (dashed lines) caused by probe light that penetrates the finger and scatters back through the finger into the receiving optics (mirror 617p and lens 617q) and optical detector array 612 d. A second detection light source module 4661 is placed side-by-side with the LCD module 433 under their common top glass cover 431 to generate second detection light at a large angle of incidence at the sensing region to penetrate into the finger and scatter back, then through the finger into the receiving optics (mirror 617p and lens 617q) and optical detector array 612 d. The optical reflection of the second detection light from the second detection light source module 4461 is not described in fig. 49, and the detection signal caused by the internal scattering of the finger is shown in a solid line.

While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this patent document can also be implemented in combination in a single embodiment, in the context of separate embodiments. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the various individual system components in the embodiments described in this patent document are not to be construed as requiring such separation in all embodiments.

This patent document describes only a few implementations and examples, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. An electronic device capable of detecting a fingerprint by optical sensing, comprising:

a Liquid Crystal Display (LCD) screen providing touch sensing operations and including an LCD display panel structure for displaying images, wherein the LCD screen includes (1) a main display area and a peripheral display area having LCD display pixels, wherein the main display area and the peripheral display area together form a seamless continuous LCD display area, and (2) an LCD backlight module providing backlight that illuminates the main display area and the peripheral display area;

a designated peripheral display area illumination module positioned relative to the LCD screen and configured to generate illumination light and direct the illumination light only to the peripheral display area to enable the peripheral display area to display images or information independently of the main display area and operable to display images or information when the LCD backlight module is off;

and the optical sensor module is arranged below the peripheral display area and used for receiving detection light which comes from an object contacting or approaching the peripheral display area and passes through the LCD display screen to detect fingerprints.

2. The electronic device of claim 1, wherein the optical sensor module comprises:

an optical detector array for receiving the probe light for optical detection directed back from a top transparent layer formed over the LCD screen as a user touch interface for the touch sensing operation and as an interface for transmitting light from the LCD screen to display an image or information to a user.

3. The electronic device of claim 2, wherein the electronic device comprises:

a mirror for positioning in the optical path of a returned probe signal beam caused by the probe beam from the designated peripheral display area illumination module to redirect the returned probe signal beam to be parallel to or at an angle relative to the direction parallel to the cover glass surface and other layers of the top transparent layer.

4. The electronic device of claim 3, wherein the electronic device further comprises:

a lens for positioning in the optical path of the reflected light beam from the mirror and projecting the reflected light beam from the mirror along the cover glass surface of the top transparent layer and other layers onto the optical detector array.

5. The electronic device of any of claims 2-4,

the optical detector array is for being disposed at an angular orientation relative to a cover glass surface or other layer of the top transparent layer.

6. The electronic device of claim 2, wherein the optical detector array is to receive the probe light directed back from a top transparent layer for optical sensing.

7. The electronic device of claim 1, wherein the designated peripheral display area illumination module comprises:

a first detection light source located below the LCD display screen in the vicinity of the optical sensor module within the peripheral display region to generate first detection light in a spectral range at a detection wavelength that is transparent to finger tissue entering a finger as part of the detection light to cause first reflected detection light from a surface reflection of the finger and first return detection light caused by internal scattering of the finger, the first reflected detection light and the first return detection light being directed toward the optical sensor module located below the LCD display panel structure;

a second probe light source located outside the LCD panel structure and below a top transparent layer to generate a second probe light entering the finger in the spectral range to cause a second return probe light caused by scattering inside the finger, the top transparent layer disposed above an LCD display screen, the second return probe light directed toward the optical sensor module located below the LCD display screen,

Wherein the optical sensor module detects a combination of the first reflected probe light, the first return probe light, and the second return probe light for fingerprint sensing.

8. The electronic device of claim 7, wherein:

the wavelength of the detection light is between 650nm and 950 nm.

9. The electronic device of claim 1, further comprising:

a top transparent layer formed over the LCD display screen as a user touch interface for the touch sensing operation and as an interface for transmitting light from the LCD display screen to display images or information to a user.

10. The electronic device of claim 1, wherein the LCD display panel structure comprises a light diffuser layer for diffusing light and a light reflector layer for redirecting illuminating light in the LCD display panel structure back to LCD pixels for display operation,

each of the light diffuser layer and the light reflector layer includes apertures or channels at selected areas over the electronic device and the peripheral display area illumination module to allow light to be transmitted to the electronic device and to allow light from the peripheral display area illumination module to illuminate LCD pixels in the peripheral display area of the LCD display panel structure.

11. The electronic device of claim 1, wherein the LCD display panel structure comprises a light diffuser layer for diffusing light and a light reflector layer for redirecting illuminating light in the LCD display panel structure back to LCD pixels for display operation,

each of the light diffuser layer and the light reflector layer includes a modified region at a selected area above the electronic device and the peripheral display area illumination module other than other portions of the light diffuser layer or the light reflector layer to allow light to be transmitted to the electronic device and to allow light from the peripheral display area illumination module to illuminate LCD pixels in the peripheral display area of the LCD display panel structure.