KR20220154390A - Electronic device and method including fingerprint sensor - Google Patents

Abstract

An electronic device according to various embodiments of the present disclosure comprises: a housing including a front surface and a rear surface facing in a direction opposite to the front surface; a display included within the housing and including at least one pixel; a fingerprint recognition sensor disposed on at least one area of the display; an illuminance sensor disposed on the display to overlap at least one area of the display; and at least one processor electrically connected to the display, the fingerprint recognition sensor, and the illuminance sensor. The at least one processor may detect an input related to an external object through the at least one area, check output setting information of the display in response to the detection, determine the sensor exposure time per row unit of the fingerprint recognition sensor to acquire data of the at least one pixel in row units based on the checked output setting information of the display, acquire a fingerprint image through the fingerprint recognition sensor based on the determined sensor exposure time, and acquire fingerprint information on the external object through the acquired fingerprint image. Accordingly, a uniform fingerprint image can be obtained.

Description

Electronic device and method including a fingerprint sensor {ELECTRONIC DEVICE AND METHOD INCLUDING FINGERPRINT SENSOR}

The present disclosure relates to an electronic device and method including a fingerprint sensor.

In order to increase a screen display area on the front side of an electronic device such as a smart phone, an area of a display disposed on the front side of the electronic device has increased. As the area of the display increases, the illuminance sensor or camera module disposed at the edge of the display may be disposed adjacent to the display. An illuminance sensor may be disposed on a rear surface of the display to measure external illuminance of the electronic device.

In addition, an electronic device such as a smart phone may include an authentication function using biometric information such as a user's fingerprint or iris as a means for financial transaction or personal authentication. The electronic device may include an in-display type fingerprint sensor disposed on the rear surface of the display to recognize a user's fingerprint in the screen area of the front display.

The illuminance sensor and the fingerprint sensor disposed on the front display of the electronic device may be performed in different ways. For example, an ambient light sensor measures the intensity of ambient light to determine the state of ambient light and adjusts the brightness of a display, whereas an optical fingerprint sensor detects a user's fingerprint in the fingerprint sensing area of the display. It operates in such a way that the display of the light is emitted and the display light source is reflected by the finger and incident on the fingerprint sensor to obtain a fingerprint image.

Light output from the display acts as a signal component for the fingerprint recognition sensor, whereas it may act as a noise component for an illuminance sensor that measures external light to measure external light. In addition, while an external light source acts as a signal component for the illuminance sensor, it may act as a noise component for the fingerprint sensor since the external light source may pass through a finger and be incident on the fingerprint sensor.

In order to minimize the noise that the display light source has on the illuminance sensor, it is recommended to operate the illuminance sensor in the interval where the display does not emit light by increasing the time of the off period in which the plurality of light emitting elements do not emit light between frames output through the display. It is possible. In the case of increasing the display on-off ratio to minimize the effect of the display light source on the illuminance sensor, the difference in the amount of light from the display light source on the fingerprint sensor inevitably increases according to the operating time of the fingerprint sensor. . For example, a fingerprint sensor of a rolling shutter type that acquires a fingerprint image in units of rows instead of a method in which all pixels are sensed at the same time produces different amounts of light depending on the light exposure timing for each row. In this case, there is a problem in that line noise occurs due to a difference in light amount for each row. The position of such line noise changes each time it is measured, and it is also difficult to remove it by image processing because it includes a narrow high-frequency component.

According to various embodiments of the present disclosure, in an electronic device, a housing including a front surface and a rear surface facing in a direction opposite to the front surface, a display included in the housing and including at least one pixel, and at least one region of the display a fingerprint recognition sensor disposed on the display, an illuminance sensor disposed on the display to overlap at least one area of the display, and at least one processor electrically connected to the display, the fingerprint recognition sensor, and the illuminance sensor; The processor detects an input related to an external object through the at least one area, checks output setting information of the display in response to the detection, and based on the checked output setting information of the display, determines the output setting of the at least one pixel. A sensor exposure time per row unit of the fingerprint recognition sensor is determined to acquire data in row units, and a fingerprint image is acquired through the fingerprint recognition sensor based on the determined sensor exposure time, , Fingerprint information on the external object may be obtained through the acquired fingerprint image.

According to various embodiments disclosed in this document, in an environment where the off ratio of a display is high, a fingerprint detected in a fingerprint sensing area through a rolling shutter type fingerprint sensor that acquires a fingerprint image in units of rows is detected. When obtaining an image, a uniform fingerprint image may be obtained.

According to various embodiments disclosed in this document, when the intensity of external light is measured through the illuminance sensor to determine the illuminance of the external environment, the off ratio of the display is increased to minimize noise generated by the display light source. By operating the illuminance sensor in a section where the display does not emit light, an electronic device capable of accurately measuring the illuminance of an external environment even in a low-light environment can be provided.

According to various embodiments disclosed in this document, an electronic device capable of increasing a recognition rate of a fingerprint recognition sensor even in an environment where an off ratio of a display is increased may be provided.

In addition, various effects identified directly or indirectly through this document may be provided.

1 is a front perspective view of an electronic device according to an exemplary embodiment;
2 is an equivalent circuit diagram illustrating an example of a pixel PX included in a display according to an exemplary embodiment.
3 is a diagram illustrating an operating state of a display according to time according to an exemplary embodiment.
4 shows a block diagram of an electronic device according to an embodiment.
5 illustrates an off ratio (OR) of a display and an operating state according to time according to an exemplary embodiment.
6 illustrates an output value of an illuminance sensor in a low-illuminance environment according to an off ratio (OR) of a display according to an exemplary embodiment.
7 illustrates an operation of a fingerprint recognition sensor according to a first sensor exposure time of the fingerprint recognition sensor.
8A is a flowchart illustrating an operation of a fingerprint recognition sensor of an electronic device according to an embodiment.
8B illustrates a sensor exposure time determination and an operation of a fingerprint recognition sensor according to the sensor exposure time, according to an embodiment.
9 illustrates an operation of a fingerprint recognition sensor according to sensor exposure time according to another embodiment.
10 illustrates an off ratio (OR) for each region in a display of an electronic device according to an embodiment.
11 illustrates an amount of display light predicted by a fingerprint recognition sensor of an electronic device according to an embodiment.
12 illustrates control according to an amount of light predicted for each row of pixels (px) corresponding to an area where a fingerprint recognition sensor is disposed, according to an exemplary embodiment.
13 illustrates control according to an amount of light predicted for each row of pixels (px) corresponding to an area where a fingerprint recognition sensor is disposed, according to another embodiment.
14 is a block diagram of an electronic device in a network environment according to various embodiments.

1 is a front perspective view of an electronic device 101 according to an embodiment.

The electronic device 101 of FIG. 1 may correspond to the electronic device 1401 of FIG. 14 to be described later. For example, the electronic device 101 may include some or all of the components constituting the electronic device 1401 of FIG. 14 .

Referring to FIG. 1 , an electronic device 101 according to an embodiment of the present invention discloses an example of a bar-type electronic device, but is slidable, rollable, and foldable. ) can also be applied to electronic devices such as type.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of this document is not limited to the aforementioned devices.

Referring to FIG. 1 , an electronic device 101 according to an embodiment includes a first surface (or front surface) 110A, a second surface (or rear surface) 110B, and a first surface 110A and a second surface. It may include a housing 110 including a side (or side wall) 110C surrounding the space between (110B). In another embodiment, the housing 110 may refer to a structure forming some of the first surface 110A, the second surface 110B, and the side surface 110C of FIG. 1 .

According to one embodiment, the first surface 110A may be formed by a front plate 121 (eg, a glass plate or a polymer plate including various coating layers) that is substantially transparent at least in part. In one example, the front plate 121 may include a curved portion that is bent toward the rear plate 111 from the first surface 110A and extends seamlessly at at least one side edge portion.

In one embodiment, the second surface 110B may be formed by a substantially opaque back plate (not shown). In one example, the back plate (not shown) may be formed of coated or tinted glass, ceramic, polymer, metal (eg, aluminum, stainless steel (STS), or magnesium), or a combination of at least two of the foregoing. can In one example, the back plate 111 may include a curved portion that extends seamlessly by being bent toward the front plate 121 from the second surface 110B at at least one end.

In one embodiment, the side surface 110C may be formed by a side member (or side bezel structure or side wall) including metal and/or polymer by combining with the front plate 121 and the rear plate (not shown). . In one example, the side surface 110C is located on the right side of the electronic device 101 (eg, +x direction in FIG. 1 ) and extends along a first direction (eg, +y direction in FIG. 1 ). 1111), a second side surface 1112 parallel to the first side surface 1111 and extending along the first direction, and extending along a second direction perpendicular to the first direction (eg, +x direction in FIG. 1) , The third side surface 1113 connecting one end of the first side surface 1111 (eg, one end in the +y direction of FIG. 1) and one end of the second side surface 1112 (eg, one end in the +y direction of FIG. 1). ), and/or parallel to the third side surface 1113, the other end of the first side surface 1111 (eg, one end in the -y direction of FIG. 1) and the other end of the second side surface 1112 (eg, FIG. 1 It may include a fourth side surface 1114 connecting one end in the -y direction).

According to an embodiment, the electronic device 101 may include at least one of the display 120, the connector hole 104, the audio module 105, the illuminance sensor 140, and the camera module 150. In one example, the electronic device 101 may omit at least one or additionally include other elements. For example, the electronic device 101 may further include a sensor module (not shown). The electronic device 101 may include a key input device (not shown).

In one embodiment, a sensor such as a proximity sensor, an image sensor, or an iris sensor may be integrated into the display 120 or disposed adjacent to the display 120 within an area provided by the front plate 121 .

In one embodiment, the display 120 may be exposed through a substantial portion of the front plate 121 . In one example, the display 120 may be combined with or disposed adjacent to a touch sensing circuit, a pressure sensor capable of measuring the intensity (pressure) of a touch, and/or a digitizer that detects a magnetic field type stylus pen. In one example, the edge of the display 120 may be formed substantially the same as an adjacent outer shape (eg, a curved surface) of the front plate 121 .

In an embodiment, the electronic device 101 may obtain fingerprint information of an external object detected through the fingerprint sensing area 131 through a fingerprint recognition sensor (eg, the fingerprint recognition sensor 130 of FIG. 4 ). In one example, the fingerprint sensing area 131 may be an area overlapping the fingerprint recognition sensor 130 disposed below the display 120 . The fingerprint sensing area 131 may or may not be directly displayed on the display 120 .

In one embodiment, the electronic device 101 may be disposed in an area where the illuminance sensor 140 and/or the camera module 150 overlap at least one area of the display 120 . The illuminance sensor 140 and/or the camera module 150 may be embedded and disposed in the display 120 . In one embodiment, the illuminance sensor 140 may be disposed so that the front surface of the housing 110 overlaps at least one area of the display 120 when viewed from above. The illuminance sensor 140 may be disposed on the rear surface (eg, -z direction) of the display 120 . In one example, the illuminance sensor 140 may be disposed adjacent to the camera module 150 . In FIG. 1 , the illuminance sensor 140 is disposed at the upper center of the display 120, but is not limited thereto. For example, the illuminance sensor 140 may be disposed on the upper right or left side of the display 120 .

In one embodiment, the illuminance sensor 140 may measure external illuminance using the intensity of light incident from the outside. The illuminance sensor 140 may receive external light to measure the illuminance of the environment where the electronic device 101 is located.

In one embodiment, the camera module 150 may include a plurality of cameras. In one example, camera module 150 may include one or a plurality of lenses, an image sensor, and/or an image processor.

In one embodiment, the connector hole 104 is a connector for transmitting and receiving power and/or data to and from an external electronic device (eg, the electronic devices 1402 and 1404 of FIG. 14 ) and/or to transmit and receive audio signals to and from the external electronic device. The following connectors may be accommodated: For example, the connector hole 104 may include a USB connector or an earphone jack (not shown) (or "earphone interface"). The jack may be implemented as a single hole, and in another embodiment, the electronic device 101 may transmit and receive power and/or data or audio signals to and from an external device without a separate connector hole.

In one embodiment, the audio module 105 may include a microphone hole and a speaker hole. In the microphone hole, a microphone for acquiring external sound may be disposed therein, and a plurality of microphones may be disposed therein to sense the direction of the sound. In another example, a speaker hole and a microphone hole may be implemented as one hole, or a speaker (eg, a piezo speaker) may be included without a speaker hole. The speaker hole may include an external speaker hole and a receiver hole for communication.

In one embodiment, the electronic device 101 may generate an electronic signal or data value corresponding to an internal operating state or an external environmental state by including a sensor module (not shown). The electronic device 101 may include a sensor module not shown, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a color sensor, an IR (infrared) sensor, a temperature sensor, or a humidity sensor. At least one more may be included.

2 is an equivalent circuit diagram illustrating an example of a pixel PX included in the display 120 according to an exemplary embodiment.

Referring to FIG. 2 , each pixel PX among the plurality of pixels PX disposed on the display 120 may include a light emitting element EL, two transistors, and a storage capacitor (Cst). Such a sub-pixel structure is referred to as a 2T (transistor) 1C (capacitor) structure.

In one embodiment, the two transistors DT and SCT may include a driving transistor DT and a switching transistor SCT.

In an embodiment, the pixel PX includes a light emitting element EL, a switching transistor SCT that is turned on by a scan signal applied from a scan line SL and transmits a data signal, and transmitted data. It may include a storage capacitor Cst to charge a voltage value corresponding to the signal, and a driving transistor DT to control the amount of current flowing to the light emitting element EL according to the voltage value charged in the storage capacitor Cst.

In one embodiment, the light emitting element EL may include a first electrode, a second electrode, and a light emitting layer. In one example, the first electrode may be an anode electrode or a cathode electrode, and the second electrode may be a cathode or anode electrode.

In an embodiment, in the light emitting element EL, the first electrode is an anode electrode corresponding to an electrode present in each pixel PX, and the second electrode is applied with the second power source voltage ELVSS corresponding to the common voltage. is the cathode electrode. In one example, a light emitting layer may be disposed between the first electrode and the second electrode.

In an embodiment, the driving transistor DT is a transistor for driving the light emitting element EL, and may include a first node N1, a second node N2, and a third node N3. In one example, the first node N1 of the driving transistor DT may be a gate node. The first node N1 of the driving transistor DT may be connected to one terminal of the storage capacitor Cst and may be electrically connected to a source node or a drain node of the switching transistor SCT. The second node N2 of the driving transistor DT may be a source node or a drain node, and may also be electrically connected to the first electrode of the light emitting element EL. The third node N3 of the driving transistor DT may be electrically connected to the first power voltage line supplying the first power voltage ELVDD.

In an embodiment, the driving transistor DT is connected to the driving voltage line PL, and the driving current flows from the driving voltage line PL to the light emitting element EL in response to the value of the voltage stored in the storage capacitor Cst. can control. In one example, the light emitting element EL may emit light having a predetermined luminance to the outside by the driving current. That is, the driving transistor DT receives the light emitting element from the first power supply voltage ELVDD in response to the gate-source voltage Vgs (that is, the voltage value corresponding to the data signal) of the driving transistor DT. The amount of current flowing through (EL) can be controlled. Therefore, the voltage level of the first power supply voltage ELVDD should have a higher value than the gate-source voltage Vgs of the driving transistor DT. That is, the maximum value of the voltage level of the first power voltage ELVDD may vary according to the data signal input to the driving transistor DT.

In an embodiment, the switching transistor SCT is turned on or turned off according to the first gate signal supplied from the first gate line SL to output the data line DL. A connection between the first node N1 of the driving transistor DT and the driving transistor DT may be controlled. In one example, the switching transistor SCT is turned on by the first gate signal having the transistor on voltage VGL, and the data voltage Vdata supplied from the data line DL is applied to the driving transistor DT. 1 can be delivered to node N1.

In an embodiment, the storage capacitor Cst may be connected between the first node N1 and the second node N2 of the driving transistor DT. The storage capacitor Cst is charged with an amount of charge corresponding to a voltage difference between both ends, and may play a role of maintaining the voltage difference between both ends for a predetermined frame time. Accordingly, the corresponding pixel PX may emit light during a predetermined frame time.

In one embodiment, in FIG. 2 , the equivalent circuit diagram of the pixel PX included in the display 120 has been described as including two transistors and one storage capacitor, but the present invention is not limited thereto. For example, an equivalent circuit diagram may include a plurality of transistors and two or more capacitors.

3 is a diagram illustrating an operating state of the display 120 according to time according to an exemplary embodiment.

In one embodiment, the display 120 may have an operating state of repeating turn-on and turn-off during one frame.

In one embodiment, the display 120 is turned on or turned off according to the gate-to-source voltage (Vgs) of the driving transistor DT supplied to the display 120. It can be. One frame may be a time period in which scan lines of the display 120 sequentially operate. The display refresh time 310 may include one turn-on period 320 and one turn-off period 330 . The display reproduction time may mean one duty period. For example, when the display playback time 310 is from t0 to t3, the turn-off period 330 is from t0 to t2 and the turn-on period 320 is from t2 to t3. The turn-off period 330 includes an initialization period (to to t1) 340 and a data reading period (t1 to t2) when voltage for turning on the display 120 is supplied. ) (350). The initialization period may refer to an initialization period for initializing a gate-source voltage of the driving transistor DT.

4 shows a block diagram of an electronic device 101 according to an embodiment.

Referring to FIG. 4 , the electronic device 101 may include a display 120, a fingerprint recognition sensor 130, an illuminance sensor 140, or a processor 160.

In one embodiment, the display 120 may visually output information to the user through the display 120 based on an organic light emitting diode (OLED), but is not limited thereto, a quantum dot light emitting diode (QLED), Information may be visually output to a user based on at least one of a liquid crystal display (LCD) and light emitting diodes (LEDs).

In one embodiment, the electronic device 101 is an add-on type disposed on the display 120 to more intuitively control a user interface output through the display 120. A touch sensor panel (TSP) may be included. The electronic device 101 is not limited thereto, and the touch sensor panel may be implemented as an on-cell type or an in-cell type inserted into the display 120 . In one example, the touch sensor panel touches or displays the display 120 using at least one of a resistive film, capacitive components, surface acoustic waves, and infrared rays. (120) A position of an object (eg, a user's finger or a stylus pen) hovering above may be detected.

In one embodiment, the illuminance sensor 140 may measure the illuminance of the environment in which the electronic device 101 is located.

In one embodiment, the illuminance sensor 140 may be disposed to overlap at least one area of the display 120 . The illuminance sensor 140 may be disposed on the rear surface of the display 120 .

In one embodiment, the illuminance sensor 140 is any type of sensor using the intensity of light incident from the outside, such as a visible light sensor, a proximity illuminance sensor, a spectrometer sensor, or an ultraviolet (UV) sensor. of sensors may be included. The illuminance sensor 140 may include a light receiving element such as a photo diode (PD) capable of receiving light incident from the outside.

In one embodiment, the illuminance sensor 140 may be influenced by a change in transmittance of external light by the display 120 and a screen displayed on the display 120 when measuring the illuminance. For example, when the display 120 is a liquid crystal display (LCD) including a back light, transmittance of external light incident on the illuminance sensor 140 may be reduced by the back light. In another example, organic light emitting diodes (OLEDs) or quantum dot light emitting diodes (QLEDs) that each pixel PX individually emits light illuminance values measured by the illuminance sensor 140 by light emitted from the pixels this may increase

In one embodiment, the electronic device 101 may obtain fingerprint information of the user using the fingerprint recognition sensor 130 .

In an embodiment, the fingerprint recognition sensor 130 may obtain fingerprint information of an external object detected through a fingerprint sensing area (eg, the fingerprint sensing area 131 of FIG. 1 ) of the display 120 . The display 120 may output a guide to the fingerprint sensing area 131 . The guide may be a mark for inducing a user to touch the display 120 with a finger. The guide output through the display 120 may include a fingerprint form, but is not limited thereto.

In one embodiment, the fingerprint recognition sensor 130 may be disposed on the rear surface of the display 120 or inside the display 120 . In one example, the fingerprint identification sensor 130 may be disposed parallel to the display 120 .

In an embodiment, the fingerprint recognition sensor 130 may detect a pattern of an external object using light emitted from the display 120 . The display 120 may function as a light source for operation of the fingerprint recognition sensor 130 .

In one embodiment, the fingerprint recognition sensor 130 may include a CMOS image sensor (CIS), a charge coupled device image sensor (CCD), a TFT amorphous silicon image sensor, and an organic photodiode (OPD). ), or at least one of an organic image sensor.

In an embodiment, the fingerprint recognition sensor 130 may be driven using at least one method of a global shutter or a rolling shutter. The global shutter method may refer to a method in which all pixels are simultaneously sensed. The rolling shutter method may refer to a method of controlling an exposure time in units of pixel rows.

In an embodiment, the fingerprint recognition sensor 130 may determine whether an external object corresponds to a fingerprint. In one example, the fingerprint recognition sensor 130 may be electrically connected to the processor 160 . The fingerprint recognition sensor 130 may be controlled by the processor 160 .

In an embodiment, the fingerprint recognition sensor 130 may detect light emitted by the display 120 and reflected by an external object. The processor 160 may process data sensed by the fingerprint recognition sensor 130 .

In one embodiment, the processor 160 may be electrically or operatively connected to the display 120 , the illuminance sensor 140 and the fingerprint recognition sensor 130 . In one example, the processor 160 may execute calculations or data related to control and/or communication of at least one other component of the electronic device 101 by using the structures stored in a memory (not shown). In an example, the processor 160 may include a central processing unit (CPU), a graphics processing unit (GPU), a micro controller unit (MCU), a sensor hub, a supplementary processor, a communication processor, and an application processor. processor), an application specific integrated circuit (ASIC), and field programmable gate arrays (FPGAs), and may have a plurality of cores.

In an embodiment, the processor 160 may control the luminance of the display 120 based on illuminance data obtained through the illuminance sensor 140 .

In an embodiment, the processor 160 may detect an input related to an external object through the fingerprint sensing area 131 disposed on at least one area of the display 120 . For example, the processor 160 may determine whether an external object detected through the fingerprint sensing area 131 corresponds to a part of the user's body (eg, a finger).

In one embodiment, the processor 160 may determine whether the fingerprint image acquired through the fingerprint recognition sensor 130 matches fingerprint information stored in a memory (not shown). The processor 160 may unlock the electronic device 101 or perform a security-required application function when the fingerprint image acquired through the fingerprint recognition sensor 130 matches the fingerprint information stored in the memory. .

5 illustrates an off ratio (OR) of the display 120 and an operating state according to time according to an exemplary embodiment.

Hereinafter, a display off ratio (OR) may mean a ratio of a time when a light emitting element is turned off and does not emit light to a time when the display 120 reproduces one frame. A time during which one frame is reproduced by the display 120 may be a sum of a time when the light emitting device is turned on and emits light and a time when the light emitting element is turned off and does not emit light.

Referring to (a) of FIG. 5 , when the OR (off raito) of the display 120 corresponds to about 0.5% (about 0.5% of the frame period of the display), the illuminance sensor 140 responds with time. Indicates a change in the amount of light of the display 120 that is sensed.

Referring to (b) of FIG. 5 , when the OR of the display 120 corresponds to about 15% (about 15% of the frame period of the display), the display detected by the illuminance sensor 140 over time ( 120) indicates the change in light quantity.

In one embodiment, when the OR (off ratio) of the display 120 is relatively low, the difference between the high light intensity part and the low light intensity part of the OLED is not large. For example, when the OR of the display 120 is about 15%, the difference between a high and low light intensity portion is relatively greater than when the OR of the display 120 is about 0.5%. Therefore, when the OR of the display 120 corresponds to about 15%, the illuminance sensor utilizes the time when the amount of light of the OLED decreases to reduce the noise caused by the external light of the display 120 to the illuminance sensor 140, and the illuminance sensor Although the intensity of external light can be measured using 140, the difference in the amount of light incident through the fingerprint recognition sensor 130 can be large according to the light exposure time of the display 120.

6 illustrates an output value of the illuminance sensor 140 in a low-illuminance environment according to an off ratio (OR) of the display 120 according to an exemplary embodiment.

Referring to FIG. 6 , although the environment to be measured through the illuminance sensor 140 is a low-light environment, the light emitted from the display 120 may increase the illuminance value measured by the illuminance sensor 140. As the luminance (unit: nit) setting value of the display 120 increases, the output value (unit: au) measured through the illuminance sensor 140 may increase.

In one embodiment, when the display 120 is driven according to a display off ratio (OR) corresponding to the first size (eg, about 15%) 610, the illuminance of external light is measured through the illuminance sensor 140. can do. In another embodiment, when the display 120 is driven according to a display off ratio (OR) corresponding to the second size (eg, about 0.5%) 620, the illuminance of external light is measured through the illuminance sensor 140. can be measured

In an embodiment, when the illuminance sensor 140 is embedded in the display 120 driven according to the first size 610, the illuminance is measured at a time point when noise caused by external light of the display 120 is minimized. Therefore, it is possible to obtain an illuminance data value closer to the actual illuminance of external light than when the illuminance sensor 140 is built into the display 120 driven according to the second size 620 .

7 illustrates an operation of a fingerprint recognition sensor according to a first sensor exposure time of the fingerprint recognition sensor.

The sensor exposure time may mean an exposure time of a plurality of pixels in a row unit. The plurality of pixels PX may be arranged in n pixel rows and m pixel columns. (However, n and m are integers greater than 1)

The fingerprint recognition sensor may be driven according to a rolling shutter method for controlling an exposure start time in units of pixel rows.

The first sensor exposure time 710 may refer to an arbitrary time during which a plurality of pixels are exposed in a row unit. For example, the pixel may include row x, row y, and row z, and the first sensor sequentially applies to row x, row y, and row z, respectively. During the exposure time 710, the fingerprint recognition sensor may be exposed. For example, fingerprint data 711 may be obtained by photographing row x from time t1 to the first sensor exposure time 710 through the fingerprint recognition sensor. Acquiring fingerprint data 712 by capturing row y from time t2 to first sensor exposure time 710 while obtaining fingerprint data 711 for row x from time t1 to first sensor exposure time 710 can do. Acquisition of fingerprint data 713 by photographing row z from time t3 to first sensor exposure time 710 while acquiring fingerprint data 712 for row y from time t2 to first sensor exposure time 710 can do.

Fingerprint data for row x, fingerprint data for row y, and fingerprint data for row z may be acquired through the fingerprint recognition sensor. Although a plurality of pixels are illustrated as including row x, row y, and row z in FIG. 7, the present invention is not limited thereto, and a plurality of rows may be included, and the fingerprint recognition sensor may include a plurality of fingerprint data corresponding thereto. can be obtained.

Referring to (a) of FIG. 7 , the fingerprint recognition sensor may operate in an environment where the OR of the display corresponds to a first size (eg, about 15%). When fingerprint data is obtained for each row of pixels by being exposed during the first sensor exposure time 710 for each row of pixels through a fingerprint recognition sensor operating according to a rolling shutter method, the fingerprint recognition sensor is Data reflecting the difference in the amount of light exposed to each star can be obtained. The fingerprint data 711 for row x may include fingerprint data by a relatively low amount of OLED light than the fingerprint data 712 for row y and/or the fingerprint data 713 for row z. Accordingly, fingerprint data 711 for row x may include fingerprint data that is relatively darker than fingerprint data 712 for row y and/or fingerprint data 713 for row z. There may be a difference in the amount of light exposed for each row between the fingerprint data 711 for row x, the fingerprint data 712 for row y, and/or the fingerprint data 713 for row z. A fingerprint image 720 may be obtained based on fingerprint data 711 for row x, fingerprint data 712 for row y, and fingerprint data 713 for row z. The fingerprint image 720 may include an image in which a difference in the amount of light exposed to each row is reflected. In one example, the fingerprint image 720 may include line noise 721 according to a difference in the amount of light exposed for each row.

Referring to FIG. 8B , a method of controlling the exposure time of the sensor to obtain a fingerprint image of uniform brightness through the fingerprint recognition sensor 130 operating according to the rolling shutter method will be described in detail.

8A is a flowchart illustrating an operation of the fingerprint recognition sensor 130 of the electronic device 101 according to an embodiment.

According to an embodiment, the electronic device 101 (eg, the processor 160) may detect an input related to an external object through at least one area of the display 120 in operation 801 . In one example, the processor 160 may detect an input related to an external object through a touch sensor panel (TSP) of the display 120 .

In one embodiment, a fingerprint sensing area (eg, the fingerprint sensing area 131 of FIG. 1 ) may be included in at least one area of the display 120 . A fingerprint recognition sensor 130 may be disposed in an area corresponding to the fingerprint sensing area 131 .

In an embodiment, the processor 160 may detect an input related to an external object (eg, a user's body part) through the fingerprint sensing area 131 .

According to an embodiment, the electronic device 101 (eg, the processor 160) may check output setting information of the display 120 in response to the detection in operation 803 .

In an embodiment, the processor 160 may check output setting information of the display 120 when an input related to an external object is sensed through the fingerprint sensing area 131 .

In one embodiment, the output setting information of the display 120 may include information about a refresh time of the display 120 . In one example, the playback time of the display 120 may include one turn-on period and one turn-off period of the display 120 .

According to an embodiment, the electronic device 101 (eg, the processor 160) converts at least one pixel data into rows of pixels based on the output setting information of the display 120 identified in operation 805. A sensor exposure time per row of the fingerprint recognition sensor 130 may be determined.

In an embodiment, since the fingerprint recognition sensor 130 is driven in a rolling shutter method, data for at least one pixel may be obtained in units of pixel rows.

In one embodiment, the processor 160 may determine a sensor exposure time based on the output setting information. The sensor exposure time may mean the time exposed through the fingerprint recognition sensor 130 per pixel row unit.

In an embodiment, a method of determining a sensor exposure time based on output setting information will be described in detail with reference to FIG. 8B.

According to an embodiment, the electronic device 101 (eg, the processor 160) may obtain a fingerprint image through the fingerprint recognition sensor 130 based on the sensor exposure time determined in operation 807.

In an embodiment, the processor 160 may obtain fingerprint data based on the exposure time of the sensor through the fingerprint recognition sensor 130 for each row of pixels. In one example, the processor 160 may obtain a fingerprint image through fingerprint data obtained for each pixel row through the fingerprint recognition sensor 130 .

In an embodiment, the processor 160 may acquire ambient illumination data of the electronic device 101 through the illumination sensor 140 while obtaining a fingerprint image through the fingerprint recognition sensor 130 . In one example, the processor 160 may control the luminance of the display 120 based on illuminance data acquired through the illuminance sensor 140 .

According to an embodiment, the electronic device 101 (eg, the processor 160) may obtain fingerprint information on the external object through the obtained fingerprint image in operation 809 .

In one embodiment, the processor 160 may extract fingerprint information on the external object through the fingerprint image. The processor 160 may perform various functions of the electronic device 101 by utilizing the fingerprint image. For example, the processor 160 may release the lock function of the electronic device 101 by utilizing fingerprint information obtained through the fingerprint recognition sensor 130 .

8B shows the determination of the sensor exposure time 810 and the operation of the fingerprint recognition sensor 130 according to the sensor exposure time 810 according to an embodiment.

In FIG. 8B , the OR of the display 120 corresponds to a first size (eg, about 15%), and the fingerprint recognition sensor 130 acquires a fingerprint image by varying the sensor exposure time for each row. It is assumed to work as

Referring to FIG. 8B , a sensor exposure time 810 of the fingerprint recognition sensor 130 operating according to the rolling shutter method may be controlled by a processor (eg, the processor 160 of FIG. 4 ). The processor 160 may control the sensor exposure time 810 for each row in order to obtain a fingerprint image having uniform brightness through the fingerprint recognition sensor 130 .

In an embodiment, the sensor exposure time 810 may mean a time during which the fingerprint recognition sensor 130 is exposed in units of a plurality of rows of pixels.

In an embodiment, the sensor exposure time of the fingerprint recognition sensor 130 may be controlled such that the amount of light incident on the fingerprint recognition sensor 130 is the same for each row of pixels.

In one embodiment, the sensor exposure time 810 of the fingerprint recognition sensor 130 operating according to the rolling shutter method may correspond to an integer multiple of the display refresh time 820 . For example, the processor 160 may control the sensor exposure time 810 to correspond to an integer multiple of the display refresh time. In this case, data acquired for each row of pixels may be data obtained according to the same amount of light. In one example, the screen playback time 820 may mean a time during which one frame is reproduced by the display 120 . The time during which one frame is reproduced by the display 120 may be the sum of the time when the pixel (px) is turned on and emits light and the time when it is turned off and does not emit light. have.

In an embodiment, data obtained for each row of pixels may include data according to the same amount of light. Accordingly, since a fingerprint image acquired through data obtained for each row of pixels does not include line noise caused by a difference in the amount of light exposed to each row of pixels, it may correspond to a fingerprint image of uniform brightness.

For example, the fingerprint recognition sensor 130 of the electronic device 101 may be driven according to a rolling shutter method. If the rows of each pixel are sequentially arranged in the order of row x, row y, and row z, the processor 160 captures row x through the fingerprint recognition sensor 130 from time t1 to sensor exposure time 810. Thus, fingerprint data 811 may be obtained. The processor 160 may obtain the fingerprint data 812 by capturing row y during the sensor exposure time 810 from time t2. The processor 160 may acquire the fingerprint data 813 by capturing the row z from time t3 to the sensor exposure time 810 . The sensor exposure time 810 may be set as an integer multiple of the screen reproduction time 820 . Accordingly, the fingerprint recognition sensor 130 may acquire a plurality of pieces of fingerprint data captured at different sensor exposure times during the sensor exposure time 810 . A plurality of pieces of fingerprint data acquired by photographing during the sensor exposure time 810 may have the same cumulative light intensity.

In an embodiment, the processor 160 acquires fingerprint data 811 for row x, fingerprint data 812 for row y, and fingerprint data 813 for row z through the fingerprint recognition sensor 130. can Referring to FIG. 8B , it is illustrated that a plurality of pixels include row x, row y, and row z to obtain fingerprint data for row x, row y, and row z, but it is not limited thereto, and a plurality of pixels A plurality of rows may be included in the fields, and the fingerprint recognition sensor 130 may acquire a plurality of fingerprint data corresponding thereto.

In one embodiment, the processor 160 may obtain fingerprint data 811 for row x, fingerprint data 812 for row y, and fingerprint data 813 for row z with substantially equal amounts of light being exposed. have.

In one embodiment, the processor 160 may obtain illuminance data by measuring the intensity of external light through the illuminance sensor 140 for each section in which the display 120 is turned off.

9 illustrates an operation of the fingerprint recognition sensor 130a according to a sensor exposure time 910 of the fingerprint recognition sensor 130a according to another embodiment.

In an embodiment, the fingerprint recognition sensor 130a may be operated in a global shutter method in which all pixels are simultaneously sensed.

9(a) shows an operation according to the fingerprint recognition sensor 130a when the OR of the display 120 corresponds to about 0.5%.

9(b) shows an operation according to the fingerprint recognition sensor 130a when the OR of the display 120 corresponds to about 15%.

In one embodiment, since the fingerprint recognition sensor 130a operates according to a global shutter method in which all pixels are sensed simultaneously, a fingerprint image can be always obtained based on the same amount of light. Accordingly, a fingerprint image of uniform brightness may be obtained regardless of the OR of the display 120 .

In one embodiment, when the fingerprint recognition sensor 130a driven by the global shutter method is disposed in the electronic device 101, it is higher than the case where the fingerprint recognition sensor 130 driven by the rolling shutter method is disposed in the electronic device 101. Batch driving circuits are complex and can be larger in size.

10 illustrates an off ratio (OR) for each region on the display 120 of the electronic device 101 according to an exemplary embodiment.

Referring to (a) of FIG. 10 , the processor 160 sets the OR of the display area 1010 where the illuminance sensor 140 is disposed to have a first size, and the fingerprint sensing area 131 capable of detecting a fingerprint. ) (and/or the fingerprint recognition sensor 130) may be set to have a second size. In one example, the OR of the display 120 having the first size may be greater than the OR of the display 120 having the second size.

In an embodiment, the processor 160 may set the OR of the display area 1010 where the illuminance sensor 140 is disposed to be greater than the OR of the display area 1020 where the fingerprint recognition sensor 130 is disposed. The illuminance sensor 140 can obtain a more accurate illuminance data value during the turn-off period of the display 120, and the fingerprint recognition sensor 130 minimizes noise that may occur depending on the difference in light amount to obtain fingerprints. image can be obtained.

Referring to (b) of FIG. 10 , the processor 160 continuously operates from an upper area of the display 120 where the illuminance sensor 140 is disposed to a lower area of the display 120 where the fingerprint sensing area 131 is disposed. The OR of the display 120 can be varied. For example, the OR of the display 120 may be continuously decreased from the upper region of the display 120 to the lower region of the display 120 .

11 illustrates the amount of light of the display 120 predicted by the fingerprint recognition sensor 130 of the electronic device 101 according to an embodiment.

In one embodiment, the display 120 may include a plurality of pixels PX. The plurality of pixels PX may be arranged in n pixel rows and m pixel columns. (However, n and m are integers greater than 1)

In one embodiment, the display 120 may display an image in units of frames. A time required to display one frame may be defined as a vertical period. The vertical period may be determined by a scan rate of the display 120 . In one example, when the refresh rate of the display 120 is 60 Hz, the vertical period may be 1/60 second, or about 16.7 msec.

In one embodiment, the screen reproduction time 1110 may mean a time during which one frame is reproduced by the display 120 . The time during which one frame is reproduced by the display 120 may be the sum of the time when the pixel PX is turned on and emits light and the time when the pixel PX is turned off and does not emit light. When the refresh rate of the display 120 is 60 Hz, the screen refresh time 1110 may be about 16.7 msec.

In one embodiment, the display 120 may sequentially output signals from the uppermost scan line to the lowermost scan line of the display 120 in order to output an image.

In one embodiment, the illuminance sensor 140 may be disposed to overlap at least one area of the display 120 . Therefore, the illuminance sensor 140 always operates and can measure the light emission state of one area of the display 120 corresponding to the area where the illuminance sensor 140 is disposed.

Referring to (a) of FIG. 11 , the illuminance sensor 140 reaches the maximum amount of light or the minimum amount of light for each row of pixels px corresponding to the area where the illuminance sensor 140 is disposed. Data for a corresponding point in time can be obtained. In one example, the processor 160 uses the illuminance sensor 140 at a point in time when the amount of light of the pixel px for each row of the pixel px in the area where the illuminance sensor 140 is disposed corresponds to the minimum ( Data for t1) can be obtained.

Referring to (b) of FIG. 11 , the processor 160 predicts the time point t2 when the amount of light of the pixel px corresponding to the area where the fingerprint recognition sensor 130 is disposed is minimum, and the fingerprint recognition sensor ( 130) may be obtained for each row of pixels (px) corresponding to the area where the amount of light is the minimum.

Referring to Equation 1, P0 may mean the number of rows of pixels (px) from the illuminance sensor 140 to the fingerprint recognition sensor 130. Py may mean the number of rows in all pixels of the display 120 . t1 may mean a time point at which the amount of light of the pixel px corresponding to the area where the illuminance sensor 140 is disposed corresponds to the minimum. t2 may mean a time point at which the amount of light of the pixel (px) corresponding to the area where the fingerprint recognition sensor 130 is disposed corresponds to the minimum.

In an embodiment, the processor 160 may use Equation 1 to determine the fingerprint recognition sensor 130 based on the amount of light of a pixel (px) corresponding to the area where the illuminance sensor 140 is disposed and the data of the corresponding point in time. ) can be predicted.

In an embodiment, the amount of light of the pixel (px) corresponding to the area where the illuminance sensor 140 is disposed and the data about the corresponding time are the first area of the display 120 corresponding to the area where the illuminance sensor 140 is disposed. It may include a light emitting state for.

In an embodiment, the data on the amount of light of the pixel (px) corresponding to the area where the fingerprint recognition sensor 130 is disposed may include a light emission state of the second area of the display 120 .

12 illustrates control according to an amount of light predicted for each row of pixels (px) corresponding to an area where the fingerprint recognition sensor 130 is disposed, according to an exemplary embodiment.

In one embodiment, the fingerprint recognition sensor 130 of the electronic device 101 may be driven according to a rolling shutter method, and accordingly, fingerprint data for a plurality of rows may be sequentially acquired. For example, a plurality of pixels (px) may be sequentially arranged in the order of row x, row y, and row z in the area where the fingerprint recognition sensor 130 is disposed, but is not limited thereto, and row 1, row 2, ??. Row n can be arranged.

In one embodiment, the processor 160 arranges the fingerprint recognition sensor 130 according to time based on the amount of light per row of pixels (px) corresponding to the area where the illuminance sensor 140 is placed and data about the corresponding point in time. Data on the amount of light of the pixel (px) corresponding to the area that has been detected can be predicted. The processor 160 may store an arbitrary sensor exposure time 1210 through a memory (eg, the memory 1434 of FIG. 14 ). When the processor 160 captures images for each row of pixels (px) corresponding to the fingerprint recognition sensor 130 through the fingerprint recognition sensor 130 during the sensor exposure time 1210, the pixel corresponding to the fingerprint recognition sensor 130 The amount of light exposed for each row of (px) can be predicted.

Referring to (a) of FIG. 12 , when a plurality of pixels px are sequentially arranged in the order of row x, row y, and row z in the area where the fingerprint recognition sensor 130 is disposed, the processor 160 Through the fingerprint recognition sensor 130, row x, row y, and row z are sequentially photographed during the sensor exposure time 1210 pre-stored in memory, and fingerprint data 1201 for row x and fingerprint data for row y In step 1202, fingerprint data 1203 for row z may be acquired. Fingerprint data 1201 for row x, fingerprint data 1202 for row y, and fingerprint data 1203 for row z may be data according to different amounts of light for each row.

In one embodiment, the processor 160 may predict the amount of light exposed for each row of pixels (px) corresponding to the fingerprint recognition sensor 130 during the sensor exposure time 1210 . Therefore, when the amount of light exposed to each row of the pixel (px) corresponding to the fingerprint recognition sensor 130 is different during the sensor exposure time 1210, the processor 160 may change the amount of light exposed to each row through the fingerprint recognition sensor 130. In order to obtain uniform fingerprint data by calibrating the amount of light, an electrical gain value may be set for each row. For example, the amount of light exposed to row x during the sensor exposure time 1210 stored in memory from time tx is the amount of light exposed to row y during the sensor exposure time 1210 from time ty and the sensor exposure time 1210 from time tz If the amount of light exposed to row z is predicted to be relatively higher than the amount of light exposed to row z during And a relatively high gain value can be set for the data for row z.

In one embodiment, the processor 160 may store settings for gain values that are differentially applied for each row in a memory (eg, the memory 1434 of FIG. 14 ).

In an embodiment, the processor 160 sets a differential gain value for each row to correct the difference in light amount, providing fingerprint data 1211 for row x, fingerprint data 1212 for row y, and fingerprint data 1212 for row z. Data 1213 can be obtained. Since the processor 160 may obtain fingerprint data for each row in which a difference in the amount of light exposed to each row is corrected, a uniform fingerprint image may be obtained based on the fingerprint data for each row in which the difference in light amount is corrected.

13 illustrates control according to an amount of light predicted for each row of pixels (px) corresponding to an area where the fingerprint recognition sensor 130 is disposed, according to another embodiment.

Referring to (a) of FIG. 13 , when a plurality of pixels px are sequentially arranged in the order of row x, row y, and row z in the area where the fingerprint recognition sensor 130 is disposed, the processor 160 Through the fingerprint recognition sensor 130, row x, row y, and row z are sequentially photographed during the sensor exposure time 1210 pre-stored in memory, and fingerprint data 1201 for row x and fingerprint data for row y In step 1202, fingerprint data 1203 for row z may be acquired. Fingerprint data 1201 for row x, fingerprint data 1202 for row y, and fingerprint data 1203 for row z may be data according to different amounts of light for each row.

In an embodiment, the processor 160 may predict the amount of light exposed for each row of pixels (px) corresponding to the fingerprint recognition sensor 130 during the sensor exposure time 1210 . Therefore, when the amount of light exposed to each row of the pixel (px) corresponding to the fingerprint recognition sensor 130 is different during the sensor exposure time 1210, the processor 160 may change the amount of light exposed to each row through the fingerprint recognition sensor 130. Differential sensor exposure times may be set for each row in order to obtain uniform fingerprint data by calibrating the amount of light. For example, the amount of light exposed to row x during a certain sensor exposure time 1210 pre-stored in memory from time tx is the amount of light exposed to row y from time ty to sensor exposure time 1210 and the sensor exposure time from time tz It may be relatively less than the amount of light exposed to row z during (1210). In this case, the sensor exposure time for row x is the sensor exposure time 1210b obtained by maintaining the sensor exposure time 1210 previously stored in memory and reducing the sensor exposure time for row y from the sensor exposure time 1210 previously stored in memory. ), and the sensor exposure time for row z is reduced from the sensor exposure time 1210, so that it can be captured during the sensor exposure time 1210c. The processor 160 may obtain fingerprint data 1221 for row x, fingerprint data 1222 for row y, and fingerprint data 1223 for row z to which the same amount of light is exposed. In another embodiment, the sensor exposure time for row x is increased from sensor exposure time 1210 to increase the amount of light exposed by row x, and the sensor exposure time for row y and the sensor exposure time for row z are increased by the sensor exposure time 1210. By maintaining the amount of light exposed by row y and row z at time 1210, fingerprint data for row x, row y, and row z exposed to the same amount of light can be obtained. In another embodiment, the processor 160 calculates the electrical gain and sensor exposure time for each row according to the amount of light predicted for each row of pixels (px) corresponding to the area where the fingerprint recognition sensor 130 is disposed. By controlling the sensor exposure time, the processor 160 may set conditions for acquiring optimal fingerprint data.

14 is a block diagram of an electronic device 1401 within a network environment 1400, according to various embodiments.

Referring to FIG. 14 , in a network environment 1400, an electronic device 1401 communicates with an electronic device 1402 through a first network 1498 (eg, a short-range wireless communication network) or through a second network 1499. It may communicate with at least one of the electronic device 1404 or the server 1408 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 1401 may communicate with the electronic device 1404 through the server 1408. According to an embodiment, the electronic device 1401 includes a processor 1420, a memory 1430, an input module 1450, an audio output module 1455, a display module 1460, an audio module 1470, a sensor module ( 1476), interface 1477, connection terminal 1478, haptic module 1479, camera module 1480, power management module 1488, battery 1489, communication module 1490, subscriber identification module 1496 , or an antenna module 1497. In some embodiments, in the electronic device 1401, at least one of these components (eg, the connection terminal 1478) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 1476, camera module 1480, or antenna module 1497) are integrated into a single component (eg, display module 1460). It can be.

The processor 1420, for example, executes software (eg, the program 1440) to cause at least one other component (eg, hardware or software component) of the electronic device 1401 connected to the processor 1420. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 1420 transfers instructions or data received from other components (eg, sensor module 1476 or communication module 1490) to volatile memory 1432. , process commands or data stored in the volatile memory 1432, and store resultant data in the non-volatile memory 1434. According to an embodiment, the processor 1420 may include a main processor 1421 (eg, a central processing unit or an application processor) or a secondary processor 1423 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 1401 includes a main processor 1421 and a co-processor 1423, the co-processor 1423 may use less power than the main processor 1421 or be set to be specialized for a designated function. can The auxiliary processor 1423 may be implemented separately from or as part of the main processor 1421 .

The secondary processor 1423 may, for example, take the place of the main processor 1421 while the main processor 1421 is in an inactive (eg, sleep) state, or when the main processor 1421 is active (eg, running an application). ) state, together with the main processor 1421, at least one of the components of the electronic device 1401 (eg, the display module 1460, the sensor module 1476, or the communication module 1490) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 1423 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 1480 or communication module 1490). have. According to an embodiment, the auxiliary processor 1423 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 1401 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 1408). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 1430 may store various data used by at least one component (eg, the processor 1420 or the sensor module 1476) of the electronic device 1401 . The data may include, for example, input data or output data for software (eg, the program 1440) and commands related thereto. The memory 1430 may include a volatile memory 1432 or a non-volatile memory 1434 .

The program 1440 may be stored as software in the memory 1430 and may include, for example, an operating system 1442 , middleware 1444 , or an application 1446 .

The input module 1450 may receive a command or data to be used by a component (eg, the processor 1420) of the electronic device 1401 from an outside of the electronic device 1401 (eg, a user). The input module 1450 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 1455 may output sound signals to the outside of the electronic device 1401 . The sound output module 1455 may include, for example, a speaker or receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 1460 can visually provide information to the outside of the electronic device 1401 (eg, a user). The display module 1460 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display module 1460 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 1470 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 1470 acquires sound through the input module 1450, the sound output module 1455, or an external electronic device connected directly or wirelessly to the electronic device 1401 (eg: Sound may be output through the electronic device 1402 (eg, a speaker or a headphone).

The sensor module 1476 detects an operating state (eg, power or temperature) of the electronic device 1401 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 1476 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 1477 may support one or more specified protocols that may be used to directly or wirelessly connect the electronic device 1401 to an external electronic device (eg, the electronic device 1402). According to one embodiment, the interface 1477 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 1478 may include a connector through which the electronic device 1401 may be physically connected to an external electronic device (eg, the electronic device 1402). According to one embodiment, the connection terminal 1478 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 1479 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 1479 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 1480 may capture still images and moving images. According to one embodiment, the camera module 1480 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 1488 may manage power supplied to the electronic device 1401 . According to one embodiment, the power management module 1488 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 1489 may supply power to at least one component of the electronic device 1401 . According to an embodiment, the battery 1489 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module 1490 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 1401 and an external electronic device (eg, the electronic device 1402, the electronic device 1404, or the server 1408). Establishment and communication through the established communication channel may be supported. The communication module 1490 may include one or more communication processors that operate independently of the processor 1420 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 1490 is a wireless communication module 1492 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1494 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 1498 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 1499 (eg, legacy It may communicate with the external electronic device 1404 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, LAN or WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 1492 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 1496 within a communication network such as the first network 1498 or the second network 1499. The electronic device 1401 may be identified or authenticated.

The wireless communication module 1492 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 1492 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 1492 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 1492 may support various requirements defined for the electronic device 1401, an external electronic device (eg, the electronic device 1404), or a network system (eg, the second network 1499). According to an embodiment, the wireless communication module 1492 may be used to realize peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency (for realizing URLLC). Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 1497 may transmit or receive signals or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 1497 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to an embodiment, the antenna module 1497 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 1498 or the second network 1499 is selected from the plurality of antennas by, for example, the communication module 1490. can be chosen A signal or power may be transmitted or received between the communication module 1490 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 1497 in addition to the radiator.

According to various embodiments, the antenna module 1497 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 1401 and the external electronic device 1404 through the server 1408 connected to the second network 1499 . Each of the external electronic devices 1402 or 1404 may be the same as or different from the electronic device 1401 . According to an embodiment, all or part of operations executed in the electronic device 1401 may be executed in one or more external electronic devices among the external electronic devices 1402 , 1404 , or 1408 . For example, when the electronic device 1401 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 1401 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 1401 . The electronic device 1401 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 1401 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 1404 may include an internet of things (IoT) device. Server 1408 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 1404 or server 1408 may be included in the second network 1499. The electronic device 1401 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of this document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeably interchangeable with terms such as, for example, logic, logic blocks, components, or circuits. can be used A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document describe one or more instructions stored in a storage medium (eg, internal memory 1436 or external memory 1438) readable by a machine (eg, electronic device 1401). It may be implemented as software (eg, the program 1440) including them. For example, a processor (eg, the processor 1420) of a device (eg, the electronic device 1401) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the components described above may include a single object or a plurality of objects, and some of the multiple objects may be separately disposed in other components. . According to various embodiments, one or more components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by modules, programs, or other components are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

According to various embodiments, in an electronic device, a housing including a front surface and a rear surface facing in a direction opposite to the front surface, a display included in the housing and including at least one pixel, and disposed on at least one area of the display A fingerprint recognition sensor, an illuminance sensor disposed on the display to overlap at least one area of the display, and at least one processor electrically connected to the display, the fingerprint recognition sensor, and the illuminance sensor, the at least one processor comprising: An input related to an external object is detected through at least one area, output setting information of the display is checked in response to the detection, and data of the at least one pixel is processed based on the checked output setting information of the display. A sensor exposure time for each row of the fingerprint recognition sensor is determined to be acquired in units of (row), and a fingerprint image is obtained through the fingerprint recognition sensor based on the determined sensor exposure time, and the Fingerprint information on the external object may be acquired through the acquired fingerprint image.

According to an embodiment, the output setting information of the display may include information about a refresh time of the display.

According to an embodiment, the sensor exposure time may correspond to an integer multiple of the playback time of the display.

According to an embodiment, the playback time of the display may include a turn-on period and a turn-off period of the display.

According to an embodiment, the fingerprint recognition sensor may be a CMOS image sensor (CIS), a charge coupled device image sensor (CCD), a TFT amorphous silicon image sensor, or an organic photodiode (OPD). ), and an organic image sensor.

According to an embodiment, the fingerprint recognition sensor may operate in a rolling shutter method for obtaining data of the at least one pixel in a row unit.

According to an embodiment of the present disclosure, the at least one processor may obtain illuminance data of the periphery of the electronic device through the illuminance sensor while acquiring the fingerprint image based on the sensor exposure time through the fingerprint recognition sensor. have.

According to an embodiment, the at least one processor determines a light emitting state of a first region of the display corresponding to the region where the illuminance sensor is disposed through the illuminance sensor. Based on the light emitting state of the first region of the display and the output setting information, a light emitting state of the second region of the display corresponding to the region where the fingerprint recognition sensor is disposed is determined, and the light emitting state of the second region of the display is determined. An electrical gain value of the fingerprint recognition sensor may be set on a row-by-row basis based on a state, and a fingerprint image may be obtained through the fingerprint recognition sensor based on the sensor exposure time and the electrical gain value.

According to an embodiment, the at least one processor determines a light emitting state of a first region of the display corresponding to the region where the illuminance sensor is disposed through the illuminance sensor. determining the light emitting state of the first region of the display and the light emitting state of the second region of the display corresponding to the region where the fingerprint recognition sensor is disposed based on the output setting information; The sensor exposure time may be differentially determined on a row-by-row basis based on a light emission state, and a fingerprint image may be acquired through the fingerprint recognition sensor based on the determined sensor exposure time.

According to an embodiment, the at least one processor determines a light emitting state of the first region of the display corresponding to the region where the illuminance sensor is disposed through the illuminance sensor, and determines the light emitting state of the first region of the display. , Based on the output setting information, determining a light emitting state of the second region of the display corresponding to the region where the fingerprint recognition sensor is disposed, and recognizing a fingerprint row by row based on the light emitting state of the second region of the display. An electrical gain value of a sensor and an exposure time of the sensor may be determined, and a fingerprint image may be acquired through the fingerprint recognition sensor based on the electrical gain value and the sensor exposure time.

According to an embodiment, the at least one processor determines an off period, which is a period in which the display is deactivated through the illuminance sensor while acquiring the fingerprint image based on the sensor exposure time through the fingerprint recognition sensor. Illuminance data can be obtained within

According to an embodiment, a memory is further included, and the at least one processor determines whether the fingerprint information matches information stored in the memory, and if the fingerprint information matches information stored in the memory, The electronic device may be unlocked.

In the method of an electronic device including a display, a fingerprint recognition sensor, and an illuminance sensor, according to various embodiments, an operation of sensing an input related to an external object through at least one area of the display, in response to the detection, the An operation of checking the output setting information of the display, based on the checked display output setting information, in a row unit of the fingerprint recognition sensor to obtain at least one pixel data included in the one area in a row unit An operation of determining a sensor exposure time for the object, an operation of obtaining a fingerprint image through the fingerprint recognition sensor based on the sensor exposure time, and an operation of obtaining fingerprint information on the external object through the acquired fingerprint image. It may include an operation to acquire.

According to an embodiment, the display output setting information may include information about a refresh time of the display.

According to an embodiment, the sensor exposure time may correspond to an integer multiple of the playback time of the display.

According to an embodiment, the playback time of the display may include a turn-on period and a turn-off period of the display.

According to an embodiment, the fingerprint recognition sensor may be a CMOS image sensor, a charge coupled device image sensor (CCD), a TFT amorphous silicon image sensor, or an organic photodiode (OPD), An organic image sensor may be included.

According to an embodiment, while acquiring the fingerprint image based on the exposure time of the sensor through the fingerprint recognition sensor, the operation may further include acquiring illuminance data of a periphery of the electronic device through the illuminance sensor. .

According to an embodiment, the method may further include controlling luminance of the display based on the illuminance data.

According to an embodiment, while acquiring the fingerprint image based on the exposure time of the sensor through the fingerprint recognition sensor, the illuminance data is obtained through the illuminance sensor in an off period in which the display is black. An acquiring operation may be further included.

Claims (20)

In electronic devices,
a housing including a front surface and a rear surface facing in a direction opposite to the front surface;
a display included in the housing and including at least one pixel;
a fingerprint recognition sensor disposed on at least one area of the display;
an illuminance sensor disposed on the display to overlap at least one area of the display; and
at least one processor electrically connected to the display, the fingerprint recognition sensor, and the illuminance sensor; The at least one processor is:
detecting an input related to an external object through the at least one region;
In response to the detection, check output setting information of the display;
Based on the identified output setting information of the display, determining a sensor exposure time for each row of the fingerprint recognition sensor to obtain data of the at least one pixel in a row unit;
Obtaining a fingerprint image through the fingerprint recognition sensor based on the determined sensor exposure time;
Acquiring fingerprint information on the external object through the acquired fingerprint image. The method of claim 1,
The electronic device of claim 1 , wherein the output setting information of the display includes information about a refresh time of the display. The method of claim 2,
The sensor exposure time is an integer multiple of the playback time of the display, the electronic device. The method of claim 2,
The playback time of the display includes a turn-on period and a turn-off period of the display. The method of claim 1,
The fingerprint recognition sensor may be a CMOS image sensor (CIS), a charge coupled device image sensor (CCD), a TFT amorphous silicon image sensor, an organic photodiode (OPD), or an organic image sensor ( An electronic device, including an organic image sensor. The method of claim 1,
The electronic device of claim 1 , wherein the fingerprint recognition sensor operates in a rolling shutter method for obtaining data of the at least one pixel in a row unit. The method of claim 1,
The electronic device of claim 1 , wherein the at least one processor acquires illuminance data of a periphery of the electronic device through the illuminance sensor while acquiring the fingerprint image based on the sensor exposure time through the fingerprint recognition sensor. The method of claim 1,
The at least one processor is:
determining a light emitting state of a first area of the display corresponding to the area where the illuminance sensor is disposed through the illuminance sensor;
determining a light emitting state of a first region of the display and a light emitting state of a second region of the display corresponding to the region where the fingerprint recognition sensor is disposed based on the output setting information;
Setting an electrical gain value of the fingerprint recognition sensor on a row-by-row basis based on a light emitting state of the second region of the display;
An electronic device that obtains a fingerprint image through the fingerprint recognition sensor based on the sensor exposure time and the electrical gain value. The method of claim 1,
The at least one processor is:
determining a light emitting state of a first area of the display corresponding to the area where the illuminance sensor is disposed through the illuminance sensor;
determining a light emitting state of the first region of the display and a light emitting state of a second region of the display corresponding to the region where the fingerprint recognition sensor is disposed based on the output setting information;
Differentially determining the exposure time of the sensor in units of rows based on a light emitting state of the second region of the display;
An electronic device that obtains a fingerprint image through the fingerprint recognition sensor based on the determined sensor exposure time. The method of claim 1,
The at least one processor is:
determining a light emitting state of a first region of the display corresponding to the region where the illuminance sensor is disposed through the illuminance sensor;
determining a light emitting state of a first region of the display and a light emitting state of a second region of the display corresponding to the region where the fingerprint recognition sensor is disposed based on the output setting information;
determining an electrical gain value of a fingerprint recognition sensor and an exposure time of the sensor on a row-by-row basis based on a light emitting state of the second region of the display;
An electronic device that obtains a fingerprint image through the fingerprint recognition sensor based on the electrical gain value and the sensor exposure time. The method of claim 1,
The at least one processor acquires illuminance data through the illuminance sensor within an off period, which is a period in which the display is deactivated, while acquiring the fingerprint image based on the sensor exposure time through the fingerprint recognition sensor. to do, electronic devices. The method of claim 1,
contain more memory;
The at least one processor is:
determining whether the fingerprint information matches information stored in the memory;
An electronic device that unlocks the electronic device when the fingerprint information matches information stored in the memory. A method of an electronic device including a display, a fingerprint recognition sensor, and an illuminance sensor,
sensing an input related to an external object through at least one area of the display;
In response to the detection, checking output setting information of the display;
Based on the checked display output setting information, a sensor exposure time for each row of the fingerprint recognition sensor is determined to obtain at least one pixel data included in the one area in a row unit. action to decide;
obtaining a fingerprint image through the fingerprint recognition sensor based on the sensor exposure time; and
And acquiring fingerprint information on the external object through the acquired fingerprint image. The method of claim 13,
The display output setting information includes information about a refresh time of the display. The method of claim 13,
Wherein the sensor exposure time is an integer multiple of the refresh time of the display. The method of claim 13,
The playback time of the display includes one turn-on period and one turn-off period of the display. The method of claim 13,
The fingerprint recognition sensor may be a CMOS image sensor, a charge coupled device image sensor (CCD), a TFT amorphous silicon image sensor, an organic photodiode (OPD), or an organic image sensor. A method, including senor). The method of claim 13,
The method of claim 1, further comprising obtaining illuminance data of a periphery of the electronic device through the illuminance sensor while acquiring the fingerprint image based on the sensor exposure time through the fingerprint recognition sensor. The method of claim 18
The method further comprises controlling luminance of the display based on the illuminance data. The method of claim 13,
Further comprising obtaining illuminance data through the illuminance sensor in an off period in which the display is black while acquiring the fingerprint image based on the sensor exposure time through the fingerprint recognition sensor. How to.

Images