WO2021199769A1 - 電子機器 - Google Patents

電子機器 Download PDF

Info

Publication number
WO2021199769A1
WO2021199769A1 PCT/JP2021/006197 JP2021006197W WO2021199769A1 WO 2021199769 A1 WO2021199769 A1 WO 2021199769A1 JP 2021006197 W JP2021006197 W JP 2021006197W WO 2021199769 A1 WO2021199769 A1 WO 2021199769A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
electronic device
authentication
light receiving
display
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/006197
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
征志 中田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Semiconductor Solutions Corp
Original Assignee
Sony Semiconductor Solutions Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Semiconductor Solutions Corp filed Critical Sony Semiconductor Solutions Corp
Priority to KR1020227032597A priority Critical patent/KR20220159378A/ko
Priority to US17/802,442 priority patent/US12131574B2/en
Priority to EP21781028.2A priority patent/EP4131919A4/en
Priority to JP2022511653A priority patent/JPWO2021199769A1/ja
Publication of WO2021199769A1 publication Critical patent/WO2021199769A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/12—Fingerprints or palmprints
    • G06V40/13—Sensors therefor
    • G06V40/1318—Sensors therefor using electro-optical elements or layers, e.g. electroluminescent sensing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/1633—Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
    • G06F1/1684—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/31—User authentication
    • G06F21/32—User authentication using biometric data, e.g. fingerprints, iris scans or voiceprints
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q20/00—Payment architectures, schemes or protocols
    • G06Q20/38—Payment protocols; Details thereof
    • G06Q20/40—Authorisation, e.g. identification of payer or payee, verification of customer or shop credentials; Review and approval of payers, e.g. check credit lines or negative lists
    • G06Q20/401—Transaction verification
    • G06Q20/4014—Identity check for transactions
    • G06Q20/40145—Biometric identity checks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0641—Electronic shopping [e-shopping] utilising user interfaces specially adapted for shopping
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/143—Sensing or illuminating at different wavelengths
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/12—Fingerprints or palmprints
    • G06V40/1335—Combining adjacent partial images (e.g. slices) to create a composite input or reference pattern; Tracking a sweeping finger movement
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/12—Fingerprints or palmprints
    • G06V40/1365—Matching; Classification
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/60—Static or dynamic means for assisting the user to position a body part for biometric acquisition
    • G06V40/67—Static or dynamic means for assisting the user to position a body part for biometric acquisition by interactive indications to the user
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/53—Control of the integration time
    • H04N25/532—Control of the integration time by controlling global shutters in CMOS SSIS
    • H—ELECTRICITY
    • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H10K—ORGANIC ELECTRIC SOLID-STATE DEVICES
    • H10K50/00—Organic light-emitting devices
    • H10K50/80—Constructional details
    • H10K50/86—Arrangements for improving contrast, e.g. preventing reflection of ambient light
    • H—ELECTRICITY
    • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H10K—ORGANIC ELECTRIC SOLID-STATE DEVICES
    • H10K59/00—Integrated devices, or assemblies of multiple devices, comprising at least one organic light-emitting element covered by group H10K50/00
    • H10K59/40—OLEDs integrated with touch screens
    • H—ELECTRICITY
    • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H10K—ORGANIC ELECTRIC SOLID-STATE DEVICES
    • H10K59/00—Integrated devices, or assemblies of multiple devices, comprising at least one organic light-emitting element covered by group H10K50/00
    • H10K59/60—OLEDs integrated with inorganic light-sensitive elements, e.g. with inorganic solar cells or inorganic photodiodes
    • H10K59/65—OLEDs integrated with inorganic image sensors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/15—Biometric patterns based on physiological signals, e.g. heartbeat, blood flow

Definitions

  • This disclosure relates to electronic devices.
  • the optical fingerprint sensor had to keep the finger stationary for a predetermined time when acquiring the fingerprint, which was not convenient in terms of UI. Further, in a terminal logged in by fingerprint authentication, subsequent operations are often possible without fingerprint authentication or the like. For example, it is often possible to make a purchase by credit card payment from a browser after performing personal authentication by fingerprint once, and there is also a security problem.
  • One aspect of the present disclosure provides an electronic device that realizes fingerprint authentication during a swipe operation of a display.
  • the electronic device includes a display and an optical fingerprint sensor.
  • the display includes a display surface having light emitting pixels in an array in a first direction and a direction intersecting the first direction.
  • the optical fingerprint sensor has light receiving elements in an array in the first direction and the second direction on the side opposite to the display surface of the display in the third direction intersecting the first direction and the second direction.
  • An image pickup element is provided, and each of the light receiving elements transfers the charge charged by photoelectric conversion at the same timing.
  • the image pickup element may include a pixel memory that temporarily stores the light reception information of each light receiving element, and the transfer of information from the light receiving element to the pixel memory may be executed at the same timing.
  • An information processing unit that executes information processing of fingerprint information read by the image sensor may be further provided.
  • the information processing unit may read the fingerprint information at least during an operation including a finger swipe operation.
  • a touch panel that senses contact information with respect to the display may be further provided, and the speed of the swipe operation may be estimated on the touch panel.
  • the information processing unit may estimate the speed of the swipe operation based on the information read by the image sensor.
  • an instruction to slow down the swipe operation may be output.
  • the instruction of the speed of the swipe operation may be output to the display.
  • the instruction may display a speed guide on the display.
  • the output that the speed is too fast may be at least one of the output to the display, the output by sound, or the output by vibration.
  • the information processing unit may shorten the exposure time of the optical fingerprint sensor when the speed is faster than a predetermined speed.
  • the information processing unit may generate authentication information from the fingerprint information captured at different times.
  • the light emitting pixel may output light having a different wavelength on the display surface side of the light receiving element, and the light receiving element may acquire the fingerprint information based on the reflected light having a different wavelength.
  • a polarizing filter may be provided between the light receiving element and the display surface, and the light receiving element may sense polarized light through the polarizing filter.
  • a filter for acquiring the state of hemoglobin may be provided between the light receiving element and the display surface, and the information processing unit may acquire information on the hemoglobin and perform biometric authentication.
  • the information processing unit may perform biometric authentication based on the information on the temporal shape change of the finger in contact with the display surface.
  • the light receiving element may detect the swipe operation with a plurality of fingers.
  • the information processing unit may execute the fingerprint authentication by using a combination of a plurality of fingers in the swipe operation of a plurality of fingers.
  • the combination of multiple fingers may be different based on the required authentication accuracy.
  • the information processing unit may detect finger information and accumulate the fingerprint information while the authentication is being executed or during the time when the authentication is not being executed.
  • the information processing unit may accumulate changes in the finger to improve the authentication accuracy.
  • the information processing unit may acquire and store the fingerprint information on a finger other than the unregistered finger.
  • the light receiving element may be provided with a larger number of elements in a direction intersecting the direction of the swipe operation than the number of elements in the direction of the swipe operation.
  • the light receiving element may be provided with a number of elements in a direction intersecting the direction of the swipe operation, which is more than twice the number of elements in the direction of the swipe operation.
  • the display may display a guide for executing the swipe operation in a direction intersecting a direction in which many light receiving elements are provided.
  • An area provided with the light receiving element is displayed on the display, product information and a purchase button are arranged so as to pass through the area, and the product information is swiped from the product information to the purchase button.
  • an interface that enables the purchase of the product may be provided, and the purchase information may be transmitted to the server based on the result of the fingerprint authentication.
  • a dynamic object may be displayed on the display so as to include an area provided with the light receiving element.
  • the shape of the object may change when the user's finger touches it.
  • the light emitting state of the object may change when the user's finger touches it.
  • the light emitting state may be changed so as to be suitable for acquiring the fingerprint information.
  • the object may dynamically change based on the fingerprint information acquisition status or the personal authentication status after the user's finger has passed.
  • a tilt detection unit that detects the tilt of the display from the horizontal direction may be further provided, or the fingerprint information may be authenticated based on the tilt detected by the tilt detection unit.
  • the signal to be acquired is described as image information or imaging information, but the image information and imaging information are broadly defined concepts, and one frame in a still image, a moving image, or a moving image. It is a concept that includes images and the like. Further, “greater than” and “less than” may be read as “greater than or equal to” and “less than or equal to”, respectively.
  • FIG. 1 is a diagram schematically showing an electronic device according to an embodiment. Further, FIG. 2 is a diagram schematically showing a cross section of the electronic device according to the embodiment.
  • the electronic device 1 is, for example, an arbitrary device having a display function and a shooting function, such as a smartphone, a mobile phone, a tablet terminal, and a personal computer.
  • the electronic device 1 includes a display unit 2 and a camera module 3.
  • the display screen 1a extends close to the external size of the electronic device 1, and the width of the bezel 1b around the display screen 1a can be reduced to, for example, several mm or less. It is possible.
  • Many electronic devices 1 include a fingerprint authentication unit in the bezel 1b, but in the present embodiment, the camera module 3 is provided in the display screen 1a as shown by the dotted line. As shown in the left figure, the width of the bezel 1b can be narrowed by providing the camera module 3 for taking a picture for fingerprint authentication on the back side of the display screen 1a.
  • the camera module 3 is arranged on the back surface side of the display screen 1a near the center, but the position is not limited to this figure if it is the back surface of the display screen 1a.
  • the camera module 3 may be arranged near the peripheral edge of the display screen 1a, or may be arranged below the center of the right figure.
  • the place may be arranged at a plurality of places instead of one place.
  • the display unit 2 and the camera module 3 are provided on one surface of the electronic device 1, but the present invention is not limited to this.
  • the display unit 2 and the camera module 3 may be provided on both sides of the electronic device 1.
  • the display unit 2 is a structure in which a display panel 4, a circularly polarizing plate 5, a touch panel 6, and a cover glass 7 are laminated as a display optical system.
  • the arrangement of these is not limited, and may be appropriately replaced, two or more of the same configurations may exist, or even another configuration may be included.
  • the display panel 4 may be provided with, for example, an OLED (Organic Light Emitting Diode), a liquid crystal display, a MicroLED, and other light emitting elements based on a display principle in an array.
  • the light emitting elements are provided in an array in a plane including the first direction and the second direction.
  • the display panel 4 such as OLED is composed of a plurality of layers.
  • a member having a low transmittance, such as a color filter layer, is often arranged on the display panel 4.
  • a through hole may be formed in the member having a low transmittance in the display panel 4 according to the arrangement location of the camera module 3. If the subject light passing through the through hole is incident on the camera module 3, the image quality of the image captured by the camera module 3 can be improved.
  • the circularly polarizing plate 5 is provided to reduce glare or enhance the visibility of the display screen 1a even in a bright environment.
  • a touch sensor is incorporated in the touch panel 6. There are various types of touch sensors such as a capacitance type, a resistance film type, and a pressure sensitive type, and any method may be used. Further, the touch panel 6 and the display panel 4 may be integrated.
  • the cover glass 7 is provided to protect the display panel 4 and the like. Each of these elements may be adhered with an adhesive that has less optical effect.
  • the camera module 3 includes an imaging unit 8 and an optical system 9.
  • the optical system 9 is arranged on the incident surface side of the light of the imaging unit 8, that is, on the side close to the display unit 2, and collects the light that has passed through the display unit 2 on the imaging unit 8.
  • the optical system 9 may include one or more lenses.
  • the imaging unit 8 operates as an optical fingerprint sensor to acquire user fingerprint information.
  • the imaging unit 8 that operates as an optical fingerprint sensor includes a plurality of photoelectric conversion units.
  • a lens is arranged in each photoelectric conversion unit. In this lens, the light appropriately emitted from the optical system 9 to the imaging unit 8 is received by the photoelectric conversion unit constituting each pixel.
  • the photoelectric conversion unit photoelectrically converts the light incident on the display unit 2.
  • the photoelectric conversion unit may be a CMOS (Complementary Metal-Oxide-Semiconductor Field-Effect Transistor) sensor or a CCD (Charge Coupled Device) sensor.
  • the photoelectric conversion unit is provided in an array, for example, in a plane including the first direction and the second direction. Further, the photoelectric conversion units may be provided in an array along the first direction and the second direction.
  • the light receiving element of the photoelectric conversion unit is provided as an array of light receiving pixels along the same direction as the light emitting pixels of the display unit.
  • the photoelectric conversion unit may be provided with a photodiode or may be provided with an organic photoelectric conversion film.
  • the plurality of photoelectric conversion units can be arranged by any method.
  • the arrangement method of the plurality of photoelectric conversion units may be, for example, a Bayer arrangement, an interline arrangement, a checkered arrangement, a stripe arrangement, or the like. It may be an array of.
  • the output value of the photoelectric conversion unit or the value obtained by predetermined conversion based on this output value is referred to as a light receiving pixel value.
  • the fingerprint information of the user is acquired at the timing when the finger passes through the area where the camera module 3 of the display exists, and personal authentication is executed using this fingerprint information.
  • the camera module 3 swipes, flicks, and wipes at the timing of dragging the product image to the purchase button. Get fingerprint information from the same finger. This acquisition is performed using the global shutter, as described below. Further, the camera module 3 executes the operation of this global shutter at an appropriate exposure time.
  • FIG. 3 is a diagram showing an example of a block diagram of the electronic device 1 according to the present embodiment.
  • the inside of the display unit 2 shown in FIGS. 1 and 2 is omitted.
  • the electronic device 1 includes an information processing unit 10 and a storage unit 11 that output information from the camera module 3 and the touch panel 6 described above and execute this information processing.
  • the image pickup unit 8 of the camera module 3 is provided with a pixel memory 8b corresponding to each photoelectric conversion unit 8a together with the photoelectric conversion unit 8a (photoelectric conversion element).
  • the information sensed by the photoelectric conversion unit 8a is transferred to the corresponding pixel memory 8b at the same timing.
  • the light receiving result is output from the pixel memory 8b to the information processing unit 10 at an arbitrary timing.
  • the information processing unit 10 is configured to include, for example, a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and the like. As described below, the information processing unit 10 may be divided into blocks that perform a part or all of various operations, and information processing by software is concretely performed by a CPU for a part or all of each operation. It may be realized as a target.
  • a CPU Central Processing Unit
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • the storage unit 11 includes, for example, various memories, storages, and the like.
  • information such as programs and data required for the electronic device 1 to operate may be stored non-temporarily.
  • the memory may include a temporary storage area such as a cache area.
  • the information processing unit 10 includes an A / D conversion unit 100, a clamp unit 101, a color-coded output unit 102, a defect correction unit 103, a linear matrix unit 104, a frame memory 105, an authentication unit 106, and an output unit. It is equipped with 107.
  • the frame memory 105 may be provided as a part of the storage unit 11 instead of the information processing unit 10.
  • the A / D conversion unit 100 converts the data received by the photoelectric conversion unit 8a and converted, that is, the analog signal stored in the pixel memory 8b at a predetermined timing into a digital signal for each pixel.
  • the A / D conversion unit 100 outputs the converted digital signal as image data.
  • the clamp unit 101 defines, for example, a black level, and the specified black level is subtracted from the image data output from the A / D conversion unit 100 and output.
  • the clamp unit 101 may set the ground level for each photoelectric conversion element provided in the pixel. In this case, the ground correction of the signal value is executed based on the ground level set for each acquired photoelectric conversion element. do.
  • the color-coded output unit 102 outputs sub-pixel image data for each color, for example, when the photoelectric conversion unit 8a acquires an analog signal for each color.
  • the photoelectric conversion unit 8a is provided with, for example, R (red), G (green), B (blue) color filters and the like. Each photoelectric conversion unit 8a outputs the sensing information for each color by these filters and the like. Instead of the photoelectric conversion unit 8a being provided with a color filter, the photoelectric conversion unit 8a may acquire an analog signal for each color by the organic photoelectric conversion film.
  • the clamp unit 101 adjusts the ground level based on these filters, and the color-coded output unit 102 outputs the signal output by the clamp unit 101 by color. No color information is added to the analog signal acquired by the photoelectric conversion unit 8a. Therefore, the color-coded output unit 102 may store data on colors arranged for each light receiving element in the photoelectric conversion unit 8a, and output each color based on this data.
  • the photoelectric conversion unit 8a may be provided with a filter for acquiring a complicated spectrum such as an element that receives near-infrared light and a plasmon filter. In these cases, it is difficult to express as mere color information, but the color-coded output unit 102 may process by associating these information with the wavelength information as long as it can be classified by the wavelength of light. In such a case, the color-coded output unit 102 may process the wavelength information as information having a range.
  • the defect correction unit 103 corrects defects in the image data. Defects in image data occur, for example, due to missing pixels or missing information due to defects in the photoelectric conversion element provided in the pixels, or missing information due to saturation of light in the optical system 9.
  • the defect correction unit 103 may execute the defect correction process by interpolating based on, for example, information on surrounding pixels or information on peripheral pixels having the same color information.
  • the linear matrix unit 104 reproduces colors by executing matrix operations on color information.
  • the linear matrix unit 104 obtains desired spectroscopy by, for example, performing calculations on a plurality of wavelengths.
  • the linear matrix unit 104 may execute, for example, an output suitable for detecting skin color. Further, the linear matrix unit 104 may execute the calculation so as to output the output suitable for the detection of the yellow to red wavelength region in order to acquire the biometric authentication information of the vein or the like.
  • the frame memory 105 may be provided, for example, as a part of the storage unit 11.
  • the frame memory 105 temporarily stores the information acquired from the pixel memory 8b output from the photoelectric conversion unit 8a at the same timing after the above processing.
  • the information processing unit 10 performs various above-mentioned processes on the fingerprint information acquired in the frame memory 105 at the same timing, and finally saves the fingerprint information as a single image data color-reproduced by the linear matrix unit 104.
  • the authentication unit 106 executes personal authentication using the fingerprint information stored in the frame memory 105. For example, this personal authentication is executed by extracting the feature points in the fingerprint data and comparing the extracted feature points with the authentication information stored in the storage unit 11. Further, the authentication unit 106 may execute biometric authentication for spoofing prevention based on not only RGB information but also, for example, near-infrared light receiving information and information acquired via a plasmon filter. For example, biometric authentication is executed depending on whether or not the wavelength information corresponding to hemoglobin in the data stored in the frame memory 105 is normally acquired. These may be determined by statistical processing such as feature points, or as another example, authentication may be performed using a neural network model trained by machine learning.
  • the authentication unit 106 stores the fingerprint information of a specific user acquired in advance in the storage unit 11. Personal authentication is executed based on this stored user information.
  • the fingerprint information of the multiple users should be stored appropriately.
  • the stored fingerprint information may be an image of each fingerprint information or the information of feature points acquired from each fingerprint information may be encrypted.
  • the output unit 107 outputs the authentication result by the authentication unit 106. For example, in the case of shopping by e-commerce or the like, it may be transmitted to a server or the like so that payment processing is performed based on the authentication result. In this way, the output unit 107 may transmit the authentication result using wireless communication. Further, as another example, the authentication result such as fingerprint authentication may be used instead of the authentication by the password or the like in the electronic device 1.
  • FIG. 4A is a diagram for explaining the relationship between the camera module 3 and the display panel 4 in FIG. 1 in more detail.
  • the camera module 3 includes, for example, an imaging unit 8 and an optical system 9.
  • the optical system 9 is arranged on the incident surface side of the light of the imaging unit 8, that is, on the side closer to the display unit 2.
  • the light transmitted through the display surface of the display unit 2 is propagated to the image pickup unit 8 by the optical system 9.
  • the imaging unit 8 may include, for example, a photoelectric conversion unit 8a such as a photodiode shown in FIG. 3 and a pixel memory 8b.
  • the light that has been condensed, refracted, diffused, etc. by the optical system 9 and propagated is received by the photoelectric conversion unit 8a provided in the imaging unit 8 and output as an analog signal.
  • the photoelectric conversion unit 8a may be provided with, for example, a color filter such as a Bayer array or a laminated color filter on the incident surface side of each image sensor. Further, as described above, instead of the color filter, another filter may be provided, or the filter may be composed of other elements such as an organic photoelectric conversion film. Further, although not shown, other elements, circuits, etc. necessary for receiving light and outputting analog signals are provided for output.
  • the optical system 9 may be a concept including, for example, a through hole in a member having a low transmittance.
  • the through hole includes, for example, an opening provided in the display panel 4, particularly the substrate a having low transmittance in the display panel 4.
  • the optical system 9 includes, for example, an aperture provided in the display panel 4 and a lens arranged at a position closer to the image pickup unit 8 than the aperture. With this lens and aperture, the optical system 9 may define optical features such as numerical aperture Na (Numerical Aperture) and F-number (F-Number) in the camera module 3.
  • the aperture and the lens are shown as an example, and the configuration of the optical system 9 is not necessarily limited to these combinations. Further, in the figure, one or a plurality of lenses are provided for one aperture, but the present invention is not limited to this.
  • a plurality of openings may be provided for one lens in the optical system 9. In the region where the opening does not exist, for example, the light emitting element of the display panel 4 may be provided, and the opening may be provided so as to sew between these light emitting elements. By arranging in this way, it is possible to provide the camera module 3 without breaking the display.
  • FIG. 5A is a cross-sectional view schematically showing a light receiving pixel according to an embodiment.
  • the light receiving pixel includes, for example, the photoelectric conversion unit 8a and the pixel memory 8b described above.
  • the light receiving pixel in the present disclosure is not limited to the light receiving pixel having the configuration shown in FIG. 5, and may be a configuration that operates as a global shutter.
  • the photoelectric conversion unit 8a receives light in the light receiving region 8R drawn above the photoelectric conversion unit 8a in the figure by the optical system 9 and the lens provided in the light receiving pixel. This light receiving portion is separated from the light receiving region 8R of the adjacent pixel by the shielding portion 8d. As shown in the figure, the shielding unit 8d is provided so that light does not pass through the upper part of the pixel memory 8b, that is, the surface of the photoelectric conversion unit 8a having the light receiving region 8R.
  • the shielding portion 8d is made of metal, for example, and the shielding portion 8d, the photoelectric conversion unit 8a, and the pixel memory 8b are separated by the insulating unit 8e.
  • the insulating portion 8e is, for example, an oxide film.
  • the photoelectric conversion unit 8a When the photoelectric conversion unit 8a receives light in the light receiving region 8R, it accumulates electric charges due to the intensity of the received light. The accumulated charge is transferred to the pixel memory 8b by applying a voltage to the electrode 8c at a predetermined timing. From the pixel memory 8b, an analog signal is output to the A / D conversion unit 100 by another transistor or the like before the transfer of the received charge at the next timing. This transfer to the A / D conversion unit 100 does not have to be performed at the same timing.
  • the transfer of electric charge from the photoelectric conversion unit 8a to the pixel memory 8b is executed at the same timing over a plurality of photoelectric conversion units 8a existing in the light receiving pixel array.
  • the same timing is not exactly the same moment, and there may be a deviation to the extent that there is no shutter distortion.
  • the imaging unit 8 can output the analog signal by the light received at the same timing by executing the operation of the global shutter for the light received by the plurality of photoelectric conversion units 8a.
  • the information processing unit 10 can acquire an image by the light received at the same timing. That is, the image pickup unit 8 can acquire the fingerprint information of the user at the same timing as the optical fingerprint sensor.
  • FIG. 5B is a plan view schematically showing the connection between the photoelectric conversion unit 8a shown in FIG. 5A and the pixel memory 8b in a plurality of pixels.
  • the electrode 8c may be the gate electrode of the transfer transistor.
  • wiring and the like other than the control line between the photoelectric conversion unit 8a and the pixel memory 8b are omitted, but of course, wiring and the like from the pixel memory 8b to the information processing unit 10 are appropriately provided.
  • the connection between the photoelectric conversion unit 8a of each light receiving pixel and the pixel memory 8b is controlled by the transfer transistor. After transferring the electric charge of the pixel memory 8b and discharging the electric charge provided in the pixel memory 8b, the electric charge is transferred from the photoelectric conversion unit 8a to the pixel memory 8b to the respective electrodes 8c of each light receiving pixel at the same timing and for the same time.
  • the voltage to be transferred is applied. This voltage is, for example, sufficiently higher than the threshold voltage for flowing charges from the drain to the source when the transfer transistor is an n-type MOSFET.
  • the electric charge acquired by the photoelectric conversion at the same timing is transferred to each pixel memory 8b.
  • analog information (charge) of the image acquired at the same timing is temporarily stored in each pixel memory 8b.
  • the output from the pixel memory 8b does not have to be at the same timing, and is appropriately transferred to the information processing unit 10. For example, the transfer from the pixel memory 8b to the information processing unit 10 is executed at a predetermined timing for each line.
  • the pixel memory 8b is supposed to store the charge state as an example, but the present invention is not limited to this.
  • the pixel memory 8b may store a voltage state, a digital value, or another state related to the received intensity information.
  • the pixel memory 8b is provided on the same substrate as the light receiving element, but the present invention is not limited to this.
  • the pixel memory 8b may be provided in, for example, a chip stacked and connected to the image pickup device.
  • the transfer transistor may be provided in any layer or between layers. These layers may be connected by, for example, via holes, micro bumps, micro pads, plasma junctions, or the like. These connections may be stacked by, for example, a CoC (Chip on Chip), CoW (Chip on Wafer) or WoW (Wafer on Wafer) method.
  • the image pickup unit 8 has an image pickup element having light receiving elements in an array in the first direction and the second direction on the side opposite to the display surface of the display unit 2 which is a display in the third direction. May be provided. Then, each of these light receiving elements realizes the operation of a global shutter that reads out the light receiving information at the same timing. Reading the light receiving information at the same timing may indicate that the light receiving element (photoelectric conversion unit) transfers the information of the charge converted by photoelectric to the memory provided for each light receiving element at the same timing. .. It should be noted that, not shown in FIGS. 5A and 5B, the light receiving pixel may adopt another method that realizes the same operation as the global shutter.
  • the information processing unit 10 executes signal processing of the fingerprint information read by the imaging unit 8, that is, the optical fingerprint sensor. Then, for example, by sliding a finger on the display surface of the display unit 2 (performing a swipe operation), the user can acquire fingerprint information by this optical fingerprint sensor and execute personal authentication or the like.
  • the swipe operation will be described, but the swipe is not limited to the swipe, and any movement such as flicking or wiping may be used as long as it slides on the touch panel.
  • FIG. 6 is a flowchart of the personal authentication process according to the embodiment.
  • the electronic device 1 determines whether or not to start personal authentication (S100). If it is not the time to start personal authentication (S100: NO), the standby state is continued.
  • the authentication may be started by using the sensing information of the touch panel 6, for example, as shown by the dotted line in FIG. For example, after the sensed information on the touch panel 6 reaches the range of the camera module 3, the process may shift to the fingerprint information acquisition process.
  • the electronic device 1 may be notified that the authentication process will be executed on the website or the like for authentication, and the electronic device 1 may be in the standby state.
  • the imaging unit 8 starts receiving light for personal authentication (S102). For example, by activating the camera module 3 for fingerprint authentication, it shifts to a standby state for acquiring fingerprint information and receives light at a required timing.
  • the photoelectric conversion unit 8a of the imaging unit 8 transfers the received intensity to the pixel memory 8b at the same timing as above (S102). This transfer is performed, for example, by applying a voltage for transfer to a plurality of electrodes 8c at the same timing.
  • the analog signal acquired by the photoelectric conversion unit 8a and output by the pixel memory 8b is converted into a digital signal.
  • the information processing unit 10 executes signal processing and image processing other than A / D conversion of the converted digital signal (S108).
  • This process is, for example, a clamp, a color-coded output, a defect correction, a color reproduction process, and a storage process in the frame memory 105, as described above.
  • the above is given as an example, and in addition to these, more appropriate processing may be executed.
  • the authentication unit 106 executes authentication using the image including the acquired fingerprint information (S110).
  • the authentication may be performed by extracting feature points and performing comparison processing, or by using a trained neural network.
  • the electronic device 1 may include a chip capable of executing the above-mentioned information processing and neural network processing on the same substrate as the camera module 3.
  • the output unit 107 outputs the authentication result to the required location (S112). For example, in the case of personal authentication when shopping on the web, the personal authentication result may be notified to the browser that opens the website or the application that accepts the authentication result. In this way, the output unit 107 can execute the output at the required location.
  • the output unit 107 may notify, for example, that the authentication has failed in order to perform the authentication operation again when the authentication cannot be performed. Further, the output unit 107 may instruct the image pickup unit 8 to take an image again.
  • the authentication unit 106 may execute the authentication process using a plurality of fingerprint information. For example, if one of the plurality of fingerprint information can be authenticated, the output unit 107 may be notified at that timing that the authentication is successful.
  • the authentication unit 106 may change the threshold value of authentication success or failure based on the authentication accuracy. For example, when high authentication accuracy is required, the authentication accuracy can be improved by increasing the authentication threshold value, for example, the fingerprint matching degree threshold value. To improve the authentication accuracy, for example, the threshold value of the matching degree of the feature points may be raised. Further, when using a neural network model, the threshold value of the degree of agreement may be increased.
  • the case where authentication with high accuracy is required is, for example, when trying to purchase an expensive product, when trying to access highly sensitive information, and the like.
  • the imaging unit 8 when the imaging unit 8 has a near-infrared filter, a plasmon filter, or the like, it may be authenticated whether or not it is a living body in addition to personal authentication. In this authentication, for example, based on information acquired by near infrared rays or the like, it is authenticated whether or not the wavelength of light is transmitted through hemoglobin such as a vein. When executing this authentication, light such as infrared rays may be emitted from the display unit 2, and this reflected state may be acquired for determination.
  • the display panel 4 may emit various colors and authenticate that it is a living body based on the reflection of the emitted light.
  • a filter for acquiring various colors may be provided as described above, or so-called multispectral or hyperspectral information may be acquired and analyzed. For example, it is possible to make a judgment based on the skin peculiar to a living body, particularly a human being, or information indicating reflection from the inside of the skin.
  • the authentication unit 106 may determine whether or not the sensing region has elasticity on the touch panel 6 and authenticate that it is a living body.
  • the imaging unit 8 may be used, and it may be determined whether or not the finger image acquired by the imaging unit 8 has elasticity in terms of time. The elasticity may be determined, for example, based on a change in the shape or size of the area occupied by the finger.
  • the touch panel 6 may be provided with a pressure-sensitive sensor, and the elasticity may be determined based on the sensing information of the pressure-sensitive sensor.
  • the authentication accuracy may be improved by acquiring fingerprint information at a plurality of timings.
  • the imaging unit 8 may acquire fingerprint information at a plurality of timings, and the fingerprint information may be stored in the frame memory 105. Then, authentication may be executed using the accumulated fingerprint information. For example, by referring to the fingerprint information, defects of the image generated at the time of image acquisition may be corrected. More simply, after adjusting the position and angle of the fingerprint, the weighted average of the fingerprint information acquired for each time may be obtained. Then, the fingerprint information acquired by this averaging calculation may be the target of authentication.
  • the imaging unit 8 may include a filter other than this. For example, by equipping the light receiving pixel with a polarizing filter, polarized fingerprint information can be acquired. By acquiring the polarized information, it becomes possible to acquire the unevenness of the fingerprint more clearly.
  • the information processing unit 10 may synthesize such polarized information to extract feature points.
  • the information processing unit 10 may include an image correction unit that corrects deterioration of the image provided in the frame memory 105.
  • the present embodiment it is possible to realize highly accurate personal authentication by using the camera module under the display. Furthermore, spoofing can be prevented by authenticating that the person is a human body. Further, since the image pickup unit 8 can acquire an image by the operation of the global shutter, it is possible to realize robust fingerprint authentication against distortion or the like depending on the shutter of the image.
  • the swipe speed of the user was not considered.
  • the speed of the user's finger is determined, and the fingerprint information is corrected based on the speed.
  • FIG. 7 is a diagram showing an example of a block diagram of the electronic device 1 according to the present embodiment.
  • the electronic device 1 further includes a speed detection unit 108 in addition to the components described with reference to FIG.
  • the speed detection unit 108 acquires the speed of the user's finger movement at the timing when the camera module 3 acquires the fingerprint image, for example, based on the detection information of the touch panel 6. That is, when the user swipes, the camera module 3 acquires fingerprint information from the received light information, and the touch panel 6 acquires finger speed information from the sensing information. For example, the speed detection unit 108 acquires the finger speed based on the amount of movement of the user's finger area detected by the touch panel 6 within a predetermined time.
  • the speed detection unit 108 acquires information for speed detection from the touch panel 6 when, for example, a user's finger touches the touch screen at the timing of authentication.
  • the speed detection unit 108 acquires, for example, the detection information of the touch panel 6 at a certain time and the detection information of the touch panel 6 after a predetermined time when the user's finger moves.
  • the sensing information of the touch panel 6 may be acquired from the center point of the finger region, for example, the position of the center of gravity when the finger region is sensed.
  • the position of the center of gravity can be obtained by calculating, for example, the averaging of the points in the sensing region in the first direction and the second direction, respectively.
  • the speed detection unit 108 detects the swipe speed of the user's finger by obtaining the position of the center of gravity after a certain time and a predetermined time.
  • the speed detection unit 108 When the speed of the finger is faster than the predetermined speed, the speed detection unit 108 outputs to the display unit 2 to urge the user to slow down the swipe speed, for example, as shown in FIG. May be good. Further, as another example, the speed detection unit 108 may notify the output unit 107 that the speed is too high and output the speed to the display unit 2 via the output unit 107. In this case, the output unit 107 is not limited to displaying on the display unit 2, and may, for example, notify the user by sound from the speaker or vibrate the vibrator.
  • the output unit 107 may determine the speed based on the desired authentication accuracy. In such a case, for example, when high authentication accuracy is required, the output unit 107 lowers the threshold value for the speed detected by the speed detection unit 108, and the user swipes slowly in a slower speed state. Can be output to.
  • FIG. 9 shows another example of output.
  • the output unit 107 may display that the number of triangular marks displayed within a predetermined time gradually increases, for example, from the figure on the left to the figure on the right.
  • the speed at which the triangular marks increase may be the speed at which the imaging unit 8 acquires highly accurate fingerprint information.
  • the output unit 107 may output an index (speed guide) indicating speed to the display. This index is not limited to the triangle as shown in FIG. 9, and may be, for example, a diagram in which an arrow extends. There may be.
  • the speed detection unit 108 detects the speed based on the detection information of the touch panel 6, but the speed is not limited to this.
  • the speed may be detected based on the imaging information acquired by the imaging unit 8.
  • the speed detection unit 108 refers to the frame memory 105, and based on the timing from the timing when the image of the user's finger is acquired to the timing when the image of the user's finger cannot be acquired, the time taken between these timings , The speed of the user's finger may be detected based on the size of the fingerprint acquisition area.
  • the speed is not limited to this, and the speed may be detected by estimating how much the finger is moving one frame or a plurality of frames after a certain time.
  • the present embodiment it is possible to improve the accuracy of fingerprint acquisition or the accuracy of authentication of the acquired fingerprint by detecting the swipe speed of the user's finger. It is also possible to output a prompt to the user so that the speed is suitable for fingerprint acquisition, if necessary.
  • the speed detected by the speed detection unit 108 can be used for other purposes.
  • the camera module 3 may acquire fingerprint information at an appropriate exposure time based on this speed information.
  • the camera module 3 may have a short exposure time when the speed is slow, and a long exposure time as the speed is high.
  • FIG. 10 is a diagram showing an example of a block diagram of the electronic device 1 according to the present embodiment.
  • the electronic device 1 further includes a correction amount calculation unit 109 in addition to the components described with reference to FIG.
  • the correction amount calculation unit 109 acquires a correction amount based on the speed detected by the speed detection unit 108. Based on this correction amount, image processing may be performed on the fingerprint information stored in the frame memory 105. This image processing is, for example, image processing based on motion blur. A point image distribution function (PSF: Point Spread Function) may be generated based on the finger speed and the shutter speed, and deconvolution filter processing may be performed on the fingerprint information.
  • PSF Point Spread Function
  • authentication is realized with one finger.
  • index finger authentication may be used.
  • authentication using a plurality of fingers may be performed.
  • the imaging unit 8 may acquire not only the information of the index finger but also the information of the middle finger at the same swipe timing, and execute personal authentication based on the two fingers. Further, 3 to 5 fingers may be used, or authentication may be performed with distant fingers such as the index finger and the ring finger. Of course, the combination of these plurality of fingers does not have to include the index finger.
  • the timing is the same in the above, different timings, for example, the first authentication may be executed with the index finger and then the second authentication may be executed with the middle finger. In this case, the information of both the right hand and the left hand may be used.
  • the information obtained by swiping multiple fingers may be acquired as one authentication information instead of performing authentication for each finger.
  • authentication using any number and combination of fingers may be executed at any timing. By increasing the authentication information in this way, it is possible to prevent false detection and spoofing with higher accuracy.
  • Multiple finger authentication may be modified based on the required high accuracy of authentication. For example, when high authentication accuracy is required, a plurality of fingers may be used, or a specific combination of fingers may be used.
  • the electronic device 1 in each of the above-described embodiments is, for example, a device such as a smartphone.
  • a finger may pass over the camera module 3 at a timing other than the fingerprint authentication timing.
  • the electronic device 1 acquires such finger information at an arbitrary timing by the imaging unit 8. Then, when the authentication unit 106 can authenticate with the registered user, the photographed fingerprint information may be stored in the storage unit 11. However, even in the case of authentication, if the authentication is successful, the acquired fingerprint information may be stored in the storage unit 11.
  • the information processing unit 10 may update the information used for authentication based on the information accumulated at an arbitrary timing. For example, when fingerprint information is acquired as image information, the information stored in the same direction and at the same position is calibrated, and the weighted average is calculated so that the newer the information, the heavier the weighting, and the authentication is performed. It may be used as information.
  • this finger may be acquired as information of the other finger of the user at the timing when the user uses the other finger and stored in the storage unit 11. good. By taking such measures, it is possible to improve the authentication accuracy even in the case of the above-mentioned plurality of fingers.
  • the information used for authentication can be updated.
  • updating it becomes possible to acquire information used for authentication, which is applied to subtle changes in fingers (fingerprints) due to aging, changes due to the environment, and the like. As a result, the accuracy of personal authentication can be maintained high.
  • FIGS. 11 and 12 are diagrams schematically showing the electronic device 1 according to the present embodiment.
  • the camera module 3 is arranged so as to be longer in the direction perpendicular to the swipe direction.
  • FIG. 11 when swiping from top to bottom or from bottom to top (second direction), a camera module having a wide area of the imaging unit 8 laterally (first direction).
  • FIG. 12 when swiping from left to right or from right to left (first direction), the camera module 3 having a wide area of the imaging unit 8 in the vertical direction (second direction) is used.
  • second direction when swiping from left to right or from right to left (first direction), the camera module 3 having a wide area of the imaging unit 8 in the vertical direction (second direction) is used.
  • the imaging unit 8 can be mounted in this way by providing the photoelectric conversion unit 8a (light receiving pixel) in the range shown by the dotted line in FIGS. 11 and 12.
  • the size of the light receiving pixels may be, for example, twice or more the number of arrangements on the wider side as the number of arrangements on the narrow side.
  • the number of light receiving pixels provided along the first direction may be twice or more the number of light receiving pixels provided along the second direction.
  • a guide indicating the swipe direction may be displayed so as to cross a certain range of the light receiving pixels.
  • the range in which fingerprint authentication can be performed by swiping that is, the range shown by the dotted line in FIGS. 11 and 12, may be displayed on the display. By displaying in this way, it is also possible to show the user where to swipe to execute personal authentication.
  • the swipe area may be made easy for the user to understand by the object displayed on the display unit 2 of the electronic device 1. This object may be dynamically changing.
  • FIG. 13 is a diagram showing an example of a GUI (Graphical User Interface) displayed on the display unit 2 of the electronic device 1.
  • GUI Graphic User Interface
  • the object in the left figure may be trembling or moving within a predetermined area, for example, so as to show the swipe area more clearly.
  • the shape and size may be changed by touching the user's finger. It may also disappear so that it fades out after swiping.
  • the display of the object is not limited to the polka dots as shown in FIG. 13, and for example, an object having a flow such as a river may be displayed. By displaying such a display, the swipe direction and the like may be indicated to the user. Not limited to these examples, for example, a predetermined character may be displayed, something like a target may be displayed, or a display that is easy for the user to understand may be used.
  • the light emitting state of the light emitting element may be changed.
  • the wavelength of the reflected light on the display panel 4 touched by the user's finger may be changed, and the change in the wavelength may facilitate the acquisition of fingerprint information.
  • the wavelength of the emitted light to a wavelength of yellow to red
  • the user's skin color or the reflection from the vein may be easily understood, or conversely, the emission of cyan or the like having a wavelength close to the complementary color may be obtained. By doing so, the wavelength of the reflected light can be made suitable for acquiring fingerprint information.
  • the object may change dynamically based on the acquisition status of the fingerprint information. For example, when the fingerprint information is normally acquired, the color of the object may change, the shape may change, or the object may disappear so as to fade out. If the fingerprint information is not acquired normally, the object in the initial state may be displayed, or the color and shape may be changed to those different from those acquired normally. Further, as in the above-described embodiment, the speed guide and the location guide may be displayed.
  • the information processing unit 10 may include a fingerprint information acquisition determination unit that determines the acquisition of fingerprint information. The acquisition of fingerprint information may be based on, for example, whether or not a predetermined number or more of irregularities are detected in a predetermined area.
  • This may be based on authentication information.
  • the object may be changed as described above depending on whether the personal authentication is successful or the personal authentication is unsuccessful.
  • two steps may be changed based on the fingerprint information acquisition status and the authentication success / failure status.
  • the present embodiment it is possible to widen the area where the image pickup unit 8 exists in the camera module 3 in the direction intersecting the swipe direction.
  • the optical fingerprint sensor By providing the optical fingerprint sensor in this way, it is possible to mount the optical fingerprint sensor having robustness against the positional deviation.
  • FIG. 14 is a block diagram showing an example of the electronic device 1 according to the present embodiment.
  • the electronic device 1 further includes a tilt detection unit 12.
  • the tilt detection unit 12 includes, for example, a gyroscope or an acceleration sensor.
  • the tilt detection unit 12 detects the tilt of the electronic device 1 at the timing when the fingerprint information is acquired.
  • the tilt is, for example, the tilt of the display with respect to gravity (vertical direction) or horizontal direction.
  • 15 and 16 are diagrams showing an example of the inclination of the electronic device 1.
  • the first direction and the second direction in the electronic device 1 are represented as the sum of the components in the gravitational direction and the horizontal direction intersecting the gravitational direction.
  • the certification unit 106 is used for matching such that the finger is peeled upward in the drawing shown in the right figure. Perform fingerprint authentication using data preferentially.
  • the authentication unit 106 when the absolute value of the gravity direction component in the first direction in the electronic device 1 is larger than the absolute value of the gravity direction component in the second direction, the authentication unit 106 is shown in the right figure. Fingerprint authentication is performed by preferentially using the matching data such that the finger is pointing to the left in the drawing shown.
  • fingerprint authentication can be executed by preferentially using the matching data in which the fingerprint is rotated by 90 degrees depending on which of the four sides is below.
  • the matching accuracy is improved and the matching speed is improved by selecting the matching data according to the inclination with respect to gravity or preferentially selecting and using the data. It becomes possible to do. As an example, every 180 degrees or every 90 degrees has been described, but the span of angles may be smaller.
  • the personal authentication process may be executed by rotating the corrected fingerprint information based on the result of the tilt detection.
  • the display includes a display surface having light emitting pixels in an array in a first direction and a second direction intersecting the first direction.
  • the optical fingerprint sensor has light receiving elements in an array in the first direction and the second direction on the side opposite to the display surface of the display in the third direction intersecting the first direction and the second direction.
  • An image pickup element is provided, and each of the light receiving elements transfers the charge charged by photoelectric conversion at the same timing. Electronics.
  • the image pickup element includes a pixel memory that temporarily stores the light receiving information of each of the light receiving elements, and the transfer of information from the light receiving element to the pixel memory is executed at the same timing.
  • An information processing unit that executes information processing of fingerprint information read by the image sensor.
  • the information processing unit reads out the fingerprint information during an operation including at least a finger swipe operation.
  • the information processing unit estimates the speed of the swipe operation based on the information read by the image sensor.
  • the electronic device according to (5) or (6).
  • the instruction displays a speed guide on the display.
  • the output of being too fast is at least one of the output to the display, the output by sound, or the output by vibration.
  • the information processing unit shortens the exposure time of the optical fingerprint sensor when the speed is faster than a predetermined speed.
  • the electronic device according to (6) The electronic device according to (6).
  • the information processing unit generates authentication information from the fingerprint information captured at different times.
  • the electronic device according to any one of (5) to (13).
  • the light emitting pixel outputs light having a different wavelength on the display surface side of the light receiving element.
  • the light receiving element acquires the fingerprint information based on reflected light of different wavelengths.
  • a polarizing filter is provided between the light receiving element and the display surface.
  • the light receiving element senses polarized light through the polarizing filter.
  • the electronic device according to (14) or (15).
  • a filter for acquiring the state of hemoglobin is provided between the light receiving element and the display surface.
  • the information processing unit acquires the hemoglobin information and performs biometric authentication.
  • the electronic device according to any one of (14) to (16).
  • the information processing unit performs biometric authentication based on the information of the temporal shape change of the finger in contact with the display surface.
  • the electronic device according to any one of (14) to (17).
  • the light receiving element detects the swipe operation on a plurality of fingers.
  • the electronic device according to any one of (5) to (18).
  • the information processing unit executes the fingerprint authentication by using a combination of the plurality of fingers in the swipe operation of the plurality of fingers.
  • the electronic device according to (19).
  • the information processing unit detects finger information and accumulates the fingerprint information while the authentication is being executed or during the time when the authentication is not being executed.
  • the electronic device according to any one of (4) to (21).
  • the information processing unit accumulates changes in the finger to improve authentication accuracy.
  • the information processing unit acquires and accumulates the fingerprint information on a finger other than the unregistered finger.
  • the light receiving element is provided with a larger number of elements in a direction intersecting the direction of the swipe operation than the number of elements in the direction of the swipe operation.
  • the light receiving element is provided with a number of elements in a direction intersecting the direction of the swipe operation, which is more than twice the number of elements in the direction of the swipe operation.
  • a guide for executing the swipe operation is displayed on the display in a direction intersecting with a direction in which many light receiving elements are provided.
  • An area provided with the light receiving element is displayed on the display, product information and a purchase button are arranged so as to pass through the area, and the product information is swiped from the product information to the purchase button.
  • An interface that enables the purchase of the product, With The purchase information is transmitted to the server based on the result of the fingerprint authentication.
  • the electronic device according to any one of (5) to (27).
  • a dynamic object is displayed on the display so as to include an area in which the light receiving element is provided.
  • the electronic device according to any one of (1) to (27).
  • the light emitting state of the object changes when the user's finger touches it.
  • the electronic device The electronic device.
  • the light emitting state changes so as to be suitable for acquiring the fingerprint information.
  • the object dynamically changes based on the fingerprint information acquisition status or the personal authentication status.
  • the electronic device according to any one of (29) to (32).
  • a tilt detection unit for detecting the tilt of the display from the horizontal direction is further provided. Authentication of the fingerprint information is executed based on the inclination detected by the inclination detection unit.
  • the electronic device according to any one of (1) to (33).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Computer Security & Cryptography (AREA)
  • Accounting & Taxation (AREA)
  • Signal Processing (AREA)
  • Finance (AREA)
  • Computer Hardware Design (AREA)
  • General Business, Economics & Management (AREA)
  • Strategic Management (AREA)
  • Software Systems (AREA)
  • Inorganic Chemistry (AREA)
  • Chemical & Material Sciences (AREA)
  • Economics (AREA)
  • Development Economics (AREA)
  • Marketing (AREA)
  • Optics & Photonics (AREA)
  • Sustainable Development (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Input (AREA)
  • Collating Specific Patterns (AREA)
PCT/JP2021/006197 2020-03-30 2021-02-18 電子機器 Ceased WO2021199769A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
KR1020227032597A KR20220159378A (ko) 2020-03-30 2021-02-18 전자기기
US17/802,442 US12131574B2 (en) 2020-03-30 2021-02-18 Electronic equipment
EP21781028.2A EP4131919A4 (en) 2020-03-30 2021-02-18 ELECTRONIC INSTRUMENT
JP2022511653A JPWO2021199769A1 (https=) 2020-03-30 2021-02-18

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-061485 2020-03-30
JP2020061485 2020-03-30

Publications (1)

Publication Number Publication Date
WO2021199769A1 true WO2021199769A1 (ja) 2021-10-07

Family

ID=77868352

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/006197 Ceased WO2021199769A1 (ja) 2020-03-30 2021-02-18 電子機器

Country Status (7)

Country Link
US (1) US12131574B2 (https=)
EP (1) EP4131919A4 (https=)
JP (1) JPWO2021199769A1 (https=)
KR (1) KR20220159378A (https=)
CN (1) CN215910907U (https=)
TW (1) TWI874604B (https=)
WO (1) WO2021199769A1 (https=)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07334646A (ja) * 1994-06-08 1995-12-22 Nippon Telegr & Teleph Corp <Ntt> 指紋撮像装置
JP2008006146A (ja) * 2006-06-30 2008-01-17 Oki Electric Ind Co Ltd 指紋認証方法とその装置
JP2009064262A (ja) * 2007-09-06 2009-03-26 Mitsumi Electric Co Ltd 画像補正装置及び画像補正方法及びこれらを用いた指紋画像読取装置及び指紋画像読取方法
US20120258773A1 (en) 2011-04-07 2012-10-11 Apple Inc. Devices and methods for providing access to internal component
JP2015023250A (ja) * 2013-07-23 2015-02-02 ソニー株式会社 固体撮像素子及びその駆動方法、並びに電子機器
JP2017538225A (ja) * 2014-12-18 2017-12-21 フィンガープリント カーズ アーベー タッチセンサのデータを使用する指紋認証
US20180074627A1 (en) * 2016-09-13 2018-03-15 Samsung Electronics Co., Ltd. Electronic device including display emitting light to provide optics-based fingerprint detection
JP2019070790A (ja) * 2017-08-11 2019-05-09 イソルグ 画像センサを備えた表示システム
JP2019128961A (ja) * 2018-01-24 2019-08-01 三星電子株式会社Ｓａｍｓｕｎｇ Ｅｌｅｃｔｒｏｎｉｃｓ Ｃｏ．，Ｌｔｄ． 指紋認識のための方法、電子装置及び格納媒体

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07334646A (ja) * 1994-06-08 1995-12-22 Nippon Telegr & Teleph Corp <Ntt> 指紋撮像装置
JP2008006146A (ja) * 2006-06-30 2008-01-17 Oki Electric Ind Co Ltd 指紋認証方法とその装置
JP2009064262A (ja) * 2007-09-06 2009-03-26 Mitsumi Electric Co Ltd 画像補正装置及び画像補正方法及びこれらを用いた指紋画像読取装置及び指紋画像読取方法
US20120258773A1 (en) 2011-04-07 2012-10-11 Apple Inc. Devices and methods for providing access to internal component
JP2015023250A (ja) * 2013-07-23 2015-02-02 ソニー株式会社 固体撮像素子及びその駆動方法、並びに電子機器
JP2017538225A (ja) * 2014-12-18 2017-12-21 フィンガープリント カーズ アーベー タッチセンサのデータを使用する指紋認証
US20180074627A1 (en) * 2016-09-13 2018-03-15 Samsung Electronics Co., Ltd. Electronic device including display emitting light to provide optics-based fingerprint detection
JP2019070790A (ja) * 2017-08-11 2019-05-09 イソルグ 画像センサを備えた表示システム
JP2019128961A (ja) * 2018-01-24 2019-08-01 三星電子株式会社Ｓａｍｓｕｎｇ Ｅｌｅｃｔｒｏｎｉｃｓ Ｃｏ．，Ｌｔｄ． 指紋認識のための方法、電子装置及び格納媒体

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4131919A4

Also Published As

Publication number Publication date
KR20220159378A (ko) 2022-12-02
JPWO2021199769A1 (https=) 2021-10-07
CN113468942A (zh) 2021-10-01
US12131574B2 (en) 2024-10-29
US20230144032A1 (en) 2023-05-11
TW202207690A (zh) 2022-02-16
TWI874604B (zh) 2025-03-01
EP4131919A4 (en) 2023-08-23
EP4131919A1 (en) 2023-02-08
CN215910907U (zh) 2022-02-25

Similar Documents

Publication Publication Date Title
KR102775935B1 (ko) 언더-디스플레이 이미지 센서
CN108369135B (zh) 用于显示器中集成的光学传感器
KR101924916B1 (ko) 온-스크린 지문 감지를 위한 언더-스크린 광학 센서 모듈
US10410036B2 (en) Under-screen optical sensor module for on-screen fingerprint sensing
CN108291838B (zh) 用于显示底板上的集成的光学传感器
KR102695040B1 (ko) 표시 장치 및 그 제어 방법
US20210342566A1 (en) Under-display-type fingerprint authentication sensor module and under-display-type fingerprint authentication device
CN110088769B (zh) 指纹采集的方法、系统和电子设备
EP3352221A2 (en) Array substrate and fabricating method thereof, display panel, display apparatus, electronic device, computer program and recording medium
US11210491B2 (en) Fingerprint sensor under a display module with tilted receiving optics
WO2017036079A1 (zh) 指纹识别元件、指纹识别方法、显示器件及显示装置
CN112602112B (zh) 显示器下方的光学生物特征成像装置中的传感器位移补偿
WO2021056318A1 (zh) 指纹识别的方法、装置和电子设备
US11948397B2 (en) Electronic device
EP4209940A1 (en) Electronic device
US12131574B2 (en) Electronic equipment
CN113468942B (zh) 电子设备
CN111868736B (zh) 带有倾斜接收光学元件的显示模块下的指纹传感器
CN119156608A (zh) 使用多视角图像传感器进行形貌特征认证

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21781028

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022511653

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021781028

Country of ref document: EP

Effective date: 20221031

WWW Wipo information: withdrawn in national office

Ref document number: 2021781028

Country of ref document: EP