KR20220126546A - Electronic device and face recognition method thereof - Google Patents

Abstract

Disclosed are an electronic device and a face recognition method thereof. The electronic device may include at least one sensor, a memory, and a processor. The electronic device may detect external illuminance through the at least one sensor, and determine the number of target frames based on the external illuminance. The electronic device may acquire a frame set according to the number of target frames through a camera, remove noise from the frame set, and perform face recognition using the frame set from which the noise has been removed. The number of target frames may be set to correspond to the external illuminance. Alternatively, the number of target frames may be set based on the external illumination and brightness information of a frame. Besides, various other embodiments are possible. Therefore, the present invention is capable of increasing the efficiency of face recognition.

Description

ELECTRONIC DEVICE AND FACE RECOGNITION METHOD THEREOF

This document relates to an electronic device and a face recognition method thereof.

Recently, the use of biometric (eg, fingerprint, face, iris, and voice) recognition technology in personal electronic devices (eg, smart phones, mobile terminals, tablets, and notebooks) having portability and mobility is increasing.

For example, face recognition technology may be applied in relation to user authentication or a security function of the electronic device. When the face recognition technology is applied, the user can easily determine whether or not authentication succeeds with a simple act of the user staring at the electronic device, which can have advantages in intuition and usability.

In a low light environment, it may be difficult to normally perform biometric recognition of an electronic device. For example, when disturbance light (eg, sunlight, lighting) is weak in a low-light environment, noise may be reflected in the image or feature points in the image may be substantially lost. The failure probability or frequency of failure of biometric recognition may increase, which may decrease usability.

With respect to the face recognition technology, a technology for increasing the success rate of face recognition in a low-light environment can be divided into a technology using an optical device and a technology not using an optical device.

In the case of a technology using an optical device, the optical device may include a light source of a specific wavelength band and a means for detecting reflected light of the light source. Electronic devices include specific sensors that can ensure robust image quality even in extreme lighting environments (e.g., structured light-type three-dimensional (3D) sensors that are not dependent on the influence of ambient light, and IR (IR) that can identify objects even in dark lighting conditions. Infrared) sensor) can be used to extract feature points from facial images. The electronic device may determine whether authentication is successful by determining the degree of face similarity based on the extracted feature points.

A technology for removing low-light noise without using an optical device can be largely classified into three methods. First, spatial (2D) denoising technology, second temporal denoising technology, and thirdly, three-dimensional (3D) denoising technology combining spatial/temporal denoising technology.

When an optical device is used, hardware complexity may increase or aesthetics may be impaired due to the addition of mechanical structures (various sensors and driving circuits). A part of the screen may be obscured, which may reduce visibility or usability.

Even if an optical device is not used, a large amount of resource consumption (eg, memory usage, calculation amount) may be inefficient in terms of recognition speed or performance when low-light noise reduction (NR) is applied.

SUMMARY OF THE INVENTION An object of the present disclosure is to provide an electronic device and a face recognition method capable of securing reliability and accuracy of a face recognition technology in a simple and intuitive manner in a low-light environment.

Another object of the present disclosure is to provide an electronic device capable of reducing resource consumption or improving face recognition speed or performance (facial recognition success rate) when processing an image for face recognition, and a face recognition method thereof.

Another object of the present disclosure is to provide an electronic device capable of removing low-light noise or increasing the efficiency of face recognition only by software processing without relying on a separate mechanical structure, and a face recognition method thereof.

An electronic device according to various embodiments may include at least one sensor, a memory, and at least one processor connected to the at least one sensor and the memory. When the memory is executed, the at least one processor detects external illuminance through the at least one sensor, determines a target frame number based on the external illuminance, and uses a camera according to the target frame number Instructions for acquiring a frame set, performing noise reduction (NR) on the frame set, and performing face recognition using the noise-removed frame set may be stored.

The method for recognizing a face of an electronic device according to various embodiments of the present disclosure includes the steps of detecting external illuminance, determining the number of target frames based on the external illuminance, acquiring a frame set according to the number of target frames through a camera, and It may include an operation of performing noise reduction (NR) on a frame set, and an operation of performing face recognition using the noise reduction frame set.

According to various embodiments, reliability and accuracy of the face recognition technology may be secured in a simple and intuitive manner in a low-light environment.

According to various embodiments, it is possible to reduce resource consumption during image processing for face recognition or to improve speed or performance (facial recognition success rate) of face recognition.

According to various embodiments, it is possible to increase the efficiency of low-light noise removal or face recognition only by software processing without adding a separate mechanical structure.

In addition, various effects directly or indirectly identified through this document may be provided.

1 is a block diagram of an electronic device according to an exemplary embodiment.
2 is a block diagram illustrating a partial configuration of an electronic device according to an exemplary embodiment.
3 is a flowchart illustrating a face recognition method of an electronic device according to an exemplary embodiment.
4 is a flowchart illustrating a face recognition method of an electronic device according to another exemplary embodiment.
5 is an example of frames for explaining a face recognition method of an electronic device according to an embodiment.
6 is a screen example illustrating an example of using an electronic device according to an embodiment.
7 is a screen example illustrating another use example of an electronic device according to an embodiment.
8 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure;
9 is a block diagram illustrating a camera module according to various embodiments.

Hereinafter, various embodiments are described with reference to the accompanying drawings.

1 is a block diagram of an electronic device according to an exemplary embodiment.

Referring to FIG. 1 , an electronic device 100 according to an embodiment may include at least one processor 110 , a memory 120 , and a sensor module 150 . The electronic device 100 may further include one or more of a display 130 , a camera module 140 , and a communication module 160 .

The processor 110 , the display 130 , the memory 120 , the sensor module 150 , the camera module 140 , and the communication module 160 are electrically and/or operatively connected to each other to transmit signals (eg, commands or data).

The electronic device 100 may include at least a part of the electronic device 801 illustrated in FIG. 8 . For example, the processor 110 may correspond to the processor 820 or 823 of FIG. 8 . The display 130 may correspond to the display module 860 of FIG. 8 . The memory 120 may include at least a portion of the memory 830 of FIG. 8 . The sensor module 150 may correspond to the sensor module 876 of FIG. 8 . The camera module 140 may correspond to the camera module 880 of FIG. 8 .

The processor 110 may include at least one processor. For example, the processor 110 may include an application processor (AP) (eg, main processor 821 ), an image signal processor (ISP) (eg, image signal processor 860 ), and a communication processor (CP) (eg: at least one of the coprocessors 823).

The processor 110 may execute and/or control various functions supported by the electronic device 100 . The processor 110 may control at least some of the display 130 , the memory 120 , the sensor module 150 , the camera module 140 , and the communication module 160 . The processor 110 may execute an application by executing a code written in a programming language stored in the memory 120 of the electronic device 100 , and may control various hardware.

As instructions stored in the memory 120 are executed, an operation of the processor 110 may be performed. Information for face recognition (eg, information about one or more frames and/or images, user information, authentication information, and application-related information) for face recognition may be stored in the memory 120 .

The processor 110 may acquire (eg, capture) one or more frames (or images) through the camera module 140 . A frame may be referred to as an image or still image. One or more frames (or images) may be converted into a database and stored in the memory 120 . The memory 120 may store frames, images, and other various information or data.

The processor 110 may perform a face recognition operation by controlling at least some of the memory 120 , the display 130 , the camera module 140 , and/or the sensor module 150 . The processor 110 may perform a face recognition operation in a normal illuminance state and/or a low illuminance state.

The display 130 may visually output a user interface (UI) or display a screen according to the control of the processor 110 .

The processor 110 may control the display 130 . The processor 110 may display a screen or a user interface for face recognition through the display 130 .

The camera module 140 may acquire a still image or a moving image and transmit it to the processor 110 . The camera module 140 may also be referred to as a camera. The camera module 140 may capture (or photograph) a still image or a moving image of a subject (eg, a specific object, or a person or a background). Still images or moving images acquired through the camera module 140 may be displayed in real time through the display 130 or stored in the memory 120 . For example, the camera module 140 may be included in the electronic device 100 . As another example, the camera module 140 may be electrically and/or operatively connected to the processor 110 in the electronic device 100 while installed in the external electronic device (eg, a cloud camera).

The camera module 140 may include at least a part of FIG. 9 . For example, the camera module 140 may include an image sensor 930 , an image signal processor 960 , and a lens assembly 910 . The image sensor 930 may perform a function of converting an optical signal of a subject input through the lens assembly 910 into an electrical signal.

The lens assembly 910 may perform a function of projecting a still image or a moving image. Lens assembly 910 may be configured to include one or more optical lenses. The image signal processor 960 may receive data input from the image sensor 930 and perform overall image processing. The image signal processor 960 may calculate an auto exposure target of the automatic face recognition module 210 or measure the brightness of a subject under the control of the processor 110 . The image signal processor 960 may set exposure or remove noise according to the calculated or measured result.

The image signal processor 960 may be integrated into the image sensor 930 or configured outside the camera module 140 (eg, a part of the processor 110 ).

The sensor module 150 may include at least one sensor (eg, an illuminance sensor, an optical sensor, or an IR sensor) for detecting external illuminance.

The processor 110 may control the camera module 140 . For example, the processor 110 may activate (eg, on) or deactivate (eg, off) the camera module 140 .

The processor 110 may control the face recognition operation.

The processor 110 may detect external illuminance through at least one sensor (eg, illuminance sensor) in the sensor module 150 . The processor 110 may determine the number of target frames based on the sensed external illuminance.

In an embodiment, the processor 110 may determine whether the low-illuminance state is in the low-illuminance state based on the external illuminance and, if the low-illuminance state, may determine the number of target frames in response to the external illuminance.

For example, the processor 110 may measure an external illuminance value through the sensor module 150 to determine the low illuminance state when the measured external illuminance value is equal to or less than a threshold (eg, 0). When the measured external illuminance value exceeds a threshold value, it may be determined as a normal illuminance state. For example, the processor 110 may obtain an input frame through the camera module 140 in a normal illuminance state, and perform face recognition using the obtained input frame. The processor 110 may dynamically set the number of frames required for low-illuminance face recognition according to external illumination in a low-illuminance state.

In an embodiment, the target frame number may be dynamically set according to external illumination.

The number of target frames may be inversely proportional to external illuminance. As the external illuminance decreases, the number of target frames may increase. For example, the number of target frames may be determined according to a range to which external illuminance measured by subdividing a range below a threshold value belongs. For example, when the external illuminance falls within a first range (eg, 0 to -200), the number of target frames may be determined as the first number (eg, 2 frames). When the external illuminance falls within the second range (eg, -200 to -400), the number of target frames may be determined as the second number (eg, 3 frames).

The processor 110 may acquire a frame set according to the target number of frames through the camera module 140 . The frame set may include one or more frames (or images) to be face recognition targets. Each frame in the frame set may be a frame including a face region (or a frame in which a face region is detected) among input frames acquired through the camera module 140 .

For example, when a face region is detected from an input frame acquired through the camera module 140 , the input frame may be included in a frame set and used as a face recognition target. When the face region is not detected in the input frame, the input frame may be excluded from the frame set and may be skipped from the face recognition target.

The processor 110 may perform noise reduction (NR) on the frame set. For example, noise removal may be performed on the entire area of each frame in the frame set. As another example, noise removal may be performed only on a face region excluding the surrounding region in the frame set, and face recognition may be performed using the noise-removed face region.

For example, the processor 110 may perform noise reduction on the frame set by a temporal noise reduction (TNR) technique. One or more frames or images from which noise has been removed may be obtained through temporal noise removal of the frame set, and face recognition may be performed using the one or more frames or images from which noise has been removed.

In an embodiment, the processor 110 may determine the number of target frames by considering both external illuminance and frame brightness information. The processor 110 may obtain brightness information of one of the frame sets. The processor 110 may adjust the preset number of first target frames (eg, the number of initial target frames) to the second target number of frames according to external illuminance based on the brightness information of the frame. The number of first target frames (eg, the number of initial target frames) may be changed to the number of second target frames by the adjusting operation.

The brightness information of the frame may include at least one of brightness information of a face region of the frame, brightness information of an entire region of the frame, and information indicating a difference in brightness between the face region and surrounding regions of the frame.

The processor 110 may perform face recognition using the noise-removed frame set. For example, face recognition may be performed using the noise-removed face region. The face recognition function can be used in various ways. As an example, face authentication of the user may be performed using face recognition. As another example, face recognition may be utilized in connection with various applications such as an authentication application, a security application, a photographing application, a camera application, a person search application, and an editing application.

In a normal illuminance state, the processor 110 may perform a face recognition operation regardless of external illuminance. The processor 110 may obtain an input frame through the camera module 140 and perform face recognition using the obtained input frame.

In an embodiment, the electronic device 100 may further include a communication module 160 . When necessary, the processor 110 communicates with an external electronic device (eg, a cloud server) (the server 808 or the electronic device 802 of FIG. 8 ) through the communication module 160 to process frames or images during the face recognition operation. All related data can be transmitted and received. The processor 110 may transmit/receive information transmitted from an external electronic device or information according to a user reaction through the communication module 160 . The processor 110 may update the transmitted/received information in real time. The processor 110 may communicate with an external electronic device through the communication module 160 to support various services related to face recognition desired by a user.

In an embodiment, at least a portion of the processor 110 or the memory 120 may be accessible through an authentication (eg, face authentication) procedure. Facial recognition may be used for the authentication process.

As an example, the memory 120 may include a shared memory area accessible in both a general environment (eg, a rich execution environment (REE) and a trusted environment (eg, a trusted execution environment (TEE)). For example, the shared memory The region may be a memory region in which data shareable by the first processor operating in the general environment and the second processor operating in the trusted environment among the processors included in the processor 110 is stored.

As another example, the memory 120 may include a secure memory area accessible only in a trusted environment. For example, the secure memory area may be a memory that cannot be accessed by a first processor operating in a normal environment but accessible by a second processor operating in a trusted environment.

As another example, the memory 120 may include a secure memory area that can only be accessed by a designated/authorized process through face recognition. For example, the secure memory area may be designated by a physical or logical address.

As another example, the memory 120 in the electronic device 100 may include a plurality of physically separated memories. At least one memory among the plurality of memories may be used as a secure memory (or a shared memory).

The memory 120 may store authentication information. For example, the authentication information may be stored in a secure memory area. However, the information stored in the memory 120 is not limited to the authentication information, and the region in which the authentication information is stored is not limited to the secure memory region.

A secure memory region may provide a trusted environment. The trusted environment may be provided through a virtual (or logical) core or may be operated by one core and a plurality of virtual cores. At least one processor in the processor 110 may perform a task or a thread in a general environment. At least a portion of the processor 110 or the memory 120 may be a protected area that can be used only through an authentication procedure using face recognition.

2 is a block diagram illustrating a partial configuration of an electronic device according to an exemplary embodiment.

In an embodiment, the electronic device 100 may include a security area 200 . For example, the secure region 200 may be part of the processor 110 and/or the memory 120 . The security area 200 may include a face recognition module 210 . For example, the face recognition module 210 may be a face recognition model implemented in the form of an application module.

The electronic device 100 may perform a face recognition (eg, face authentication) function using the face recognition module 210 .

The face recognition module 210 includes an image processing module 211 , a face detection module 212 , a feature information extraction module 213 , a biometric identification module 214 , a face similarity determination module 215 , and a low-light determination module 216 . and a low-light face similarity determination module 217 .

The image processing module 211 may process (eg, synthesize, adjust exposure, and remove noise) one or more frames (or images) acquired through the camera module 140 . For example, the image processing module 211 may correspond to the image signal processor 960 of FIG. 9 .

The face detection module 212 may detect a face region from a frame (or image) processed by the image processing module 211 . For example, the face detection module 212 may identify a face region in a frame or image, extract a size of the face region, or calculate brightness information (eg, brightness histogram information) of the frame or image.

The feature information extraction module 213 may extract feature information (eg, feature points) of a face region from a frame or an image. For example, the characteristic information may refer to unique characteristic information possessed by an individual so that a person's face can be distinguished from other people. For example, the facial feature information may be information about a landmark of the face (eg, eyes, nose, mouth, jaw line, or a specific area of the skin).

The liveness determination module 214 may be a module for determining whether a face region in a frame or image is alive. For example, whether a pattern according to the extracted feature information corresponds to a unique pattern related to a person's face may determine whether or not the living body is present. As another example, the biometric determination module 214 determines whether the body is a living body based on additional information (eg, pre-registered biometric model information, reference information for determining whether an image is authentic or forged, multi-biometric information such as body temperature and voice) can be

The face similarity determining module 215 may perform a face recognition operation in a normal illuminance state. The face similarity determining module 215 may determine the similarity by comparing the feature information extracted from the face region with pre-stored facial feature information when it is determined that the face region in the frame or image is a living body in the normal illuminance state. As a result of the determination, if the degree of similarity is greater than or equal to a certain percentage (eg, 90%), face recognition (eg, authentication) may be successful. As a result of the determination, if the similarity is less than a certain percentage (eg, 90%), face recognition (eg, authentication) may fail.

The low-illuminance determination module 216 may determine whether the electronic device 100 is in a low-illuminance state.

When the electronic device 100 is in a low-illuminance state, the low-illuminance face similarity determining module 217 may operate.

The low-illuminance face similarity determining module 217 may perform a face recognition operation in a low-illuminance state. The low-illuminance face similarity determining module 217 may determine the similarity by comparing the feature information extracted from the face region with pre-stored facial feature information when it is determined that the face region in the frame or image is a living body in a low-illuminance state. As a result of the determination, if the degree of similarity is greater than or equal to a certain percentage (eg, 90% or 80%), the face recognition (eg, authentication) operation may be successful. As a result of the determination, if the similarity is less than a certain percentage (eg, 90% or 80%), the face recognition (eg, authentication) operation may fail.

In another embodiment, at least a portion of the security area 200 may be included in an external electronic device (the server 808 or the electronic device 802 of FIG. 8 ). For example, the electronic device 100 may partially interwork with an external electronic device to perform a face recognition (eg, face authentication) function.

The configuration of the face recognition module 210 shown in FIG. 2 is merely an example, and may be variously modified, applied, or modified. For example, some of the components of the face recognition module 210 may be included in the open area instead of the secure area 200 . For example, some of the components of the face recognition module 210 may be integrated with each other, each component may be separated, some components may be omitted, or other components may be added. For example, although the face similarity determining module 215 and the low-illuminance face similarity determining module 217 are separately implemented in FIG. 2 , they may be integrated with each other. In this case, the face recognition operation in the normal illuminance and/or low illuminance state may be performed through the same module.

Hereinafter, a face recognition method according to various embodiments will be described with reference to FIGS. 3 to 7 . At least one of the operations of the illustrated face recognition method may be omitted, the order of some operations may be changed, or other operations may be added. Alternatively, the operations of each embodiment may be selectively combined and performed.

The face recognition method according to various embodiments may be performed by an electronic device (eg, the electronic device 100 or the processor 110 of FIG. 1 , or the electronic device 801 or the processor 820 of FIG. 8 ). . For convenience of description, it is assumed that each operation of the face recognition method according to various embodiments is performed by the processor 110 or the electronic device 100 of FIG. 1 .

3 is a flowchart illustrating a face recognition method of an electronic device according to an exemplary embodiment.

Referring to FIG. 3 , a method for recognizing a face of an electronic device according to an embodiment may include operations 310 to 350 .

In operation 310 , the processor 110 of the electronic device 100 may detect external illuminance through at least one sensor (eg, illuminance sensor) in the sensor module 150 . The sensor module 150 may measure an external illuminance value (eg, a brightness value (BV)) in the current lighting environment and transmit it to the processor 110 .

In operation 320, the processor 110 may determine (eg, set or adjust) the target frame number (N-frame value) based on the external illumination.

The number of target frames (eg, the number of initial target frames) may be dynamically set according to external illumination. For example, the number of target frames may be inversely proportional to external illuminance. As the external illuminance decreases, the number of target frames may increase. As the external illuminance increases, the number of target frames may decrease.

Information on the number of target frames for each illuminance range may be stored in advance. The number of target frames may be set according to a range to which external illuminance belongs. For example, when the external illuminance falls within the first range (eg, 0 > BV > -200), the number of target frames may be set to the first number (eg, 2 frames). When the external illuminance falls within the second range (eg -200 > BV > -400), the number of target frames may be set to the second number (eg, 3 or 4 (k=2, 2 ^k = 4)). can When the external illuminance falls within the third range (eg -400 > BV > -600), the target frame number can be set to the third number (eg 4 or 8 (k=3, 2 ^k = 8) have.

In an embodiment, the target frame number may be determined based on external illuminance and frame brightness information. For example, the frame may be a frame including a face region (or a frame in which a face region is detected). As another example, the frame may be an arbitrary frame included in a frame set that is a face recognition target. As another example, the frame may be a frame obtained by synthesizing (or accumulating) a plurality of frames to be face recognition.

For efficient face recognition, when determining the number of target frames, brightness information (eg, brightness histogram information) of a frame may be considered together with external illuminance. For example, even if the external illuminance is the same, the relative brightness of the face region in the frame may vary depending on the shooting environment (eg, backlight, non-uniform lighting environment). Even if the overall external illuminance is bright, the face area within the frame may be darker than the surrounding area, or vice versa. For example, when the number of target frames required for low-illuminance face recognition is two when external illumination (eg, 0 > BV > -200) is considered, and the face area within the frame is darker than the surrounding area, the processor 110 sets the target The number of frames can be increased by 1 to adjust to 3 frames. Considering the external illumination (e.g. 0 > BV > -200), if the target number of frames required for low-light face recognition is 2 and the face area within the frame is brighter than the surrounding area, the target number of frames is maintained at 2 without adjustment can be

In operation 330, the processor 110 may obtain a frame set according to the target number of frames (N-frame value). A frame set may be understood as one or more frames or images. The frame set may be a face recognition target. Noise removal and face recognition may be performed on the frame set. A frame set may include as many frame(s) as the target frame number. A frame belonging to a frame set may be a frame including a face region.

Each frame in the frame set may be a frame including a face region (or a frame in which a face region is detected) among input frames acquired through the camera module 140 . When a face region is detected from the input frame (eg, when a face object exists or the average brightness value of the face region is equal to or greater than a threshold value), the input frame may be included in a frame set. When a face region is not detected in the input frame (eg, when a face object does not exist or the average brightness value of the face region is less than a threshold value), the input frame may be excluded from the frame set.

Among the input frames, an input frame that does not include a face region may be skipped without being included in a frame set that is a face recognition target. Noise removal and face recognition operations may not be performed on the skipped input frame.

When face recognition is attempted for a frame that does not include a face region, unnecessary time delay and resource consumption may occur, which may decrease performance or speed of face recognition.

According to various embodiments, by selecting only a frame including a face (or a frame in which a face is detected) as a face recognition target, fast and efficient face recognition may be performed and a face recognition success rate may be increased.

In operation 340, the processor 110 may perform noise reduction (NR) on the frame set.

In an embodiment, the processor 110 may perform noise reduction on the frame set by a temporal noise reduction (TNR) technique.

The temporal noise removal technique may be a technique for removing noise by using a high correlation between adjacent frames of a frame set (eg, a composite image) based on a specific time point. For example, noise may be removed by calculating a weighted average of two pixels corresponding to each other in the current frame and the previous frame. When the temporal noise technique is applied, when noise is mixed in any pixel in the current frame, it is based on the assumption that there is a high probability that the corresponding pixel in an adjacent frame (e.g., a pixel at the same location or substantially the same pixel) does not contain noise. A function of a filter that suppresses noise through values of two pixels (eg, a weighted average) by motion estimation may be performed.

For example, noise removal may be performed on the entire region of a frame set (eg, a composite image). As another example, noise removal may be performed only on a face region excluding a peripheral region in a frame set (eg, a composite image). A method of performing noise removal limited to the face region in the image may be effective in removing low-light noise. In this case, the amount of computation/computation time can be reduced compared to the case of performing filtering on the entire image, and noise removal filtering can be omitted for unnecessary pixels.

In operation 350 , the processor 110 may perform face recognition using the noise-removed frame set in operation 340 . For example, a noise-removed composite frame may be obtained through temporal noise removal of a face region of a frame set (eg, a composite image), and face recognition (eg, authentication) may be performed using the composite frame.

4 is a flowchart illustrating a face recognition method of an electronic device according to another exemplary embodiment.

Referring to FIG. 4 , a method for recognizing a face of an electronic device according to another embodiment may include operations 410 to 470 . Some of the operations of FIG. 4 may correspond to the operations of FIG. 3 . For example, operation 410 may correspond to operation 310 . Operation 440 may correspond to operation 320 . Operations 451 , 453 , and 455 may correspond to operation 330 . Operation 457 may correspond to operation 340 . Operation 470 may correspond to operation 350 .

In operation 410 , the processor 110 may detect external illuminance through at least one sensor (eg, illuminance sensor) in the sensor module 150 . The processor 110 may measure an external illuminance value (eg, BV) using the sensor module 150 .

In operation 420 , the processor 110 may activate the camera module 140 for photographing. When the camera module 140 is already activated, operation 420 may be omitted.

For example, as the operation mode of the electronic device 100 is switched from the standby mode (or normal mode) to the face recognition mode, the sensor module 150 and the camera module 140 may be activated. The standby mode may be an operation mode in a state before performing face recognition. For example, the processor 110 may enter the face recognition mode in response to an event related to face recognition. For example, the event is a user input (eg, a key input requiring user authentication) for a screen (or a user interface, eg, a lock screen, a home screen, a login screen, an application execution screen) being displayed on the display 130 . , swipe input for unlocking) may be an event.

In operation 430 , the processor 110 may determine whether the electronic device 100 is in a low illuminance state based on the external illuminance sensed in operation 410 . The processor 110 may determine whether the illuminance is in a low illuminance state or a normal illuminance state based on the external illuminance.

For example, when the measured external illuminance value is less than or equal to a threshold (eg, 0), the processor 110 may determine that the illuminance is in a low illuminance state. When the measured external illuminance value exceeds a threshold value, the processor 110 may determine that it is in a normal illuminance state.

In the low-illuminance state, the processor 110 may perform a face recognition procedure according to the low-illuminance mode. The face recognition procedure according to the low light mode may include operations 440 , 451 , 453 , 455 , 457 , 459 and 470 . In the low light state, the processor 110 may obtain an input frame (or a captured image) (451), and detect a face region from the input frame (453). The processor 110 may perform face recognition after removing noise on the detected face region ( 457 ) ( 470 ). The processor 110 may calculate brightness information (eg, brightness histogram information) of the input frame. The processor 110 may extract and filter noise information of the input frame.

In the case of a normal illuminance state, the processor 110 may perform a face recognition procedure according to the normal illuminance mode. The face recognition procedure according to the normal illumination mode may include operations 461 and 470 . In the case of a normal illuminance state, the processor 110 may obtain an input frame (or a captured image) ( 461 ), and perform face recognition using the input frame ( 470 ).

If it is determined in operation 430 that the low illuminance state (operation 430 - low illuminance), operation 440 may be performed. In the case of low illumination, the processor 110 may determine the number of target frames according to the external illumination and perform a face recognition procedure based on the number of target frames.

Operation 440 may be an operation of determining (eg, setting or adjusting) the target frame number (N-frame value) for low-light face recognition.

In operation 440, the processor 110 may set the number of target frames (eg, the initial target frame number) required for low-illuminance face recognition based on the external illumination.

In an embodiment, the target frame number may be dynamically set based on external illumination. Information on the number of target frames for each illuminance range may be stored in advance. The number of target frames may be set according to a range to which external illuminance belongs. For example, when the external illuminance falls within a first range (eg, 0 to -200), the number of target frames may be set to the first number (eg, two). When the external illuminance falls within the second range (eg, -200 to -400), the number of target frames may be set to the second number (eg, 3 or 4 frames (k=2, 2 ^k =4)).

A target frame number (eg, an initial target frame number) may be set according to external illuminance, and the target frame number may be adjusted according to frame brightness information. For example, the first target frame number (or initial target frame number) is set according to the external illuminance, and the first target frame number is adjusted to the second target frame number based on the frame brightness information (eg, brightness histogram information). can The number of first target frames may be changed to the number of second target frames by the adjusting operation. The brightness information (eg, brightness histogram information) of the frame includes at least one of brightness information of a face region of the frame, brightness information of an entire region of the frame, and information indicating a difference in brightness between the face region and surrounding regions within the frame. may include

Operations 451 to 459 may be operations of acquiring a frame set according to the target number of frames (N-frame value).

The processor 110 may obtain a frame set according to the target number of frames by repeatedly performing operations 451 to 459 one or more times. A frame set may be understood as one or more frames or images. The frame set may be a face recognition target. Noise removal and face recognition may be performed on the frame set. A frame set may include as many frame(s) as the target frame number. A frame belonging to a frame set may be a frame including a face region.

In operation 451 , the processor 110 may acquire an input frame (or a captured image, for example, frame 1) through the camera module 140 .

In operation 453 , the processor 110 may determine whether a face region is detected in the input frame (eg, frame 1) (or whether the face region is included in the input frame).

Among the input frames acquired through the camera module 140 , only frame(s) including the face region may be selectively included in the frame set. The frame set may be a face recognition target. Noise removal and face recognition may be performed on the frame set.

When a face region is detected from the input frame (eg, when a face object exists or the average brightness value of the face region is equal to or greater than a threshold value), the input frame may be included in a frame set. When a face region is not detected in the input frame (eg, when a face object does not exist or the average brightness value of the face region is less than a threshold value), the input frame may be excluded from the frame set.

If it is determined in operation 453 that no face region is detected in the input frame (operation 453 - NO), the processor 110 returns to operation 451 and re-creates the input frame (eg, frame 2) through the camera module 140 . can be obtained

When a face region is detected in the input frame (eg, frame 1) (operation 453 - Yes), the input frame is included as a frame set, and operations 455 and 457 may be performed.

In operation 455 , the processor 110 may update the current frame number n. The current frame number n may be the number of frame(s) belonging to the frame set or the number of frame(s) including the face region.

When a face region is detected in an input frame (eg, frame 1) (operation 453 - Yes), the input frame is counted as a frame belonging to a frame set, and the current frame number n may increase by one (n). = n+1). When a face region is not detected in the input frame (operation 453 - NO), the input frame may be skipped without being counted as a frame belonging to a frame set. In this case, the current frame number n may be maintained without being updated.

The processor 110 waits without performing the face recognition 470 until the current frame number n reaches the target frame number (N-frame value) to obtain a frame (451), detect a face region (453) and noise removal 457 may be repeatedly performed.

If it is determined in operation 453 that a face region is detected in the input frame (eg, frame 1) (operation 453 - Yes), a face region is extracted from the frame, and noise removal may be performed on the extracted face region. For example, a temporal noise removal technique may be applied to the extracted face region.

Operation 457 may be a noise reduction (NR) operation. The processor 110 may perform a noise removal function by itself or through the camera module 140 .

If it is determined in operation 453 that the face region is not detected in the input frame (eg, frame 1) (operation 453 - NO), the processor 110 may acquire the next frame (eg, frame 2).

In operation 457 , the processor 110 may perform noise removal on an input frame (eg, frame 1 or a composite image of frames in a frame set). For example, the processor 110 may perform noise reduction on the frame set by a temporal noise reduction (TNR) technique. For example, noise removal may be performed on the entire region of a frame set (eg, a composite image). As another example, noise removal may be performed only on a face region excluding a peripheral region in a frame set (eg, a composite image).

In operation 459, the processor 110 may determine whether the current frame number n exceeds the target frame number (N-frame value).

As an example, if it is determined in operation 459 that the current frame number n is less than or equal to the target frame number (N-frame value) (operation 459 - NO), as shown in the figure, the processor 110 returns to operation 451 and returns to the next frame (eg, frame 2) can be obtained. The processor 110 may determine whether the current frame number n reaches a set target frame number while applying frame acquisition 451 , face detection 453 , and noise removal 457 ( 459 ). . When the current frame number n reaches the target frame number, a face recognition operation 470 may be performed.

As another example, if it is determined in operation 459 that the current frame number n is less than or equal to the target frame number (N-frame value) (operation 459 - NO), the processor 110 may return to operation 440 to adjust the target frame number. .

In operation 440 , the processor 110 may adjust the preset number of target frames (eg, the number of initial target frames) based on brightness information of a frame (eg, frame 1) in the frame set.

The processor 110 may obtain brightness information (eg, brightness histogram information) of a frame (eg, frame 1). The processor 110 may adjust the number of target frames based on the brightness information of the frame. The number of first target frames (eg, the number of initial target frames) may be changed to the number of second target frames by the adjusting operation.

The brightness information of a frame (eg, brightness histogram information) includes at least one of brightness information of a face region of the frame, brightness information of an entire region of the frame, and information indicating a difference in brightness between the face region and surrounding regions of the frame. may include

For example, when the number of target frames required for low-illuminance face recognition is two when external illumination is considered, and the face area within the frame is darker than the surrounding area, the processor 110 may increase the number of target frames to adjust the number of frames to three. have. When the number of target frames required for low-illuminance face recognition is two when external illuminance is considered, and the face area within the frame is brighter than the surrounding area, the number of target frames may be maintained at two without adjustment.

If it is determined in operation 459 that the number of frames (n) in the frame set exceeds the target frame number (N-frame value) (operation 459 - Yes), operation 470 may be performed.

Operations 451 to 459 may be repeatedly performed until a frame set according to the target frame number is obtained or the number of frame(s) in the frame set reaches the target frame number. For example, when the target frame number (N-frame value) is 3, operations 451 to 459 may be repeatedly performed a certain number of times (eg, 3 or more) until three frames in which a face region is detected are obtained. have.

In operation 470 , the processor 110 may perform face recognition using the noise-removed frame set in operation 455 .

The face recognition function in operation 470 may be used in various ways. As an example, user authentication using face recognition may be performed. The processor 110 may perform face authentication of the user using a noise-removed frame set (eg, a composite image or a composite frame). The processor 110 may determine the similarity between the face region in the frame set and the user's face registered in advance for authentication, and determine whether authentication succeeds or fails according to the similarity determination result. As another example, face recognition may be utilized in connection with various applications such as an authentication application, a security application, a photographing application, a camera application, a person search application, and an editing application.

If it is determined in operation 430 that the normal illuminance is in the normal illuminance state (operation 430 - normal illuminance), the processor 110 may perform a face recognition operation regardless of the external illuminance.

In the case of a normal illumination state (operation 430 - normal illumination), operations 461 and 470 may be performed.

In operation 461 , the processor 110 may obtain an input frame through the camera module 140 . In operation 470 , the processor 110 may perform face recognition using the frame obtained in operation 461 .

If the face recognition fails, the processor 110 may acquire the next input frame to retry face recognition or display a pop-up window informing of the face recognition failure, and then terminate the face recognition operation.

5 is an example of frames for explaining a face recognition method of an electronic device according to an embodiment.

For example, the electronic device 100 may synthesize (or accumulate) a plurality of frames during a face recognition operation in a low light state and perform face recognition using the synthesized image.

The first frame 510 to the sixth frame 560 may be input frames obtained through the camera module 140 .

The first frame 510 , the second frame 520 , and the third frame 530 may be frames in which a face region is not detected. The first frame 510 , the second frame 520 , and the third frame 530 may be frames that do not include a face region.

The fourth frame 540 , the fifth frame 550 , and the sixth frame 560 may be frames in which a face region is detected. The fourth frame 540 , the fifth frame 550 , and the sixth frame 560 may be frames including a face region.

The electronic device 100 may synthesize a plurality of frames in which the face region is detected, and may determine how many frames to synthesize based on external illuminance.

As an example, the electronic device 100 determines whether a frame (or image) suitable for face recognition can be obtained by synthesizing about how many frames if the current external illuminance level is present, and determines the number of frames as the target frame number. have. The target frame number may be selected based on external illumination. As another example, the electronic device 100 may select the target number of frames in consideration of external illuminance and brightness information of a frame (eg, the fourth frame 540 ).

For example, the number of target frames may be set to three based on external illuminance. When the target frame number is 3, the electronic device 100 performs face recognition using three frames (eg, the fourth frame 540 , the fifth frame 550 , and the sixth frame 560 ) in which the face region is detected. can be done The number of target frames set based on external illuminance may be adjusted based on brightness information of a frame (eg, the fourth frame 540 ). For example, if the number of target frames required for optimized face recognition is 3 when external illumination is taken into consideration, the number of target frames increases (e.g., 3 to 4) or decreases (e.g., 3) according to the brightness information of the frame. to Chapter 2) or maintained (eg Chapter 3).

In FIG. 5 , it is assumed that the target frame number is 3 for convenience. In this case, the electronic device 100 may repeat the capturing (or photographing) operation through the camera module 140 until three frames in which the face region is detected are obtained.

Among the six frames 510 to 560 , the first frame 510 , the second frame 520 , and the third frame 530 are frames in which a face region is not detected, and synthesis may be meaningless. Synthesis may be performed from a frame in which the face region is detected (eg, the fourth frame 540 ).

The three frames 510 , 520 , and 530 in which the face region is not detected are excluded from the face recognition target, and face recognition processing (eg, synthesis, noise removal, and face recognition) may not be performed on the corresponding frames. The three frames 540 , 550 , and 560 in which the face region is detected are used as face recognition targets, and face recognition processing (eg, synthesis, noise removal, and face recognition) may be performed on the frames.

When face recognition is attempted on a frame(s) in which a face region is not detected, unnecessary time delay and resource consumption may occur, thereby degrading performance or speed of face recognition.

According to an embodiment, during the face recognition operation, a frame in which a face region is not detected may be skipped, and face recognition may be performed only on a frame in which a face region is detected. Thereby, the performance (facial recognition success rate) or speed of low-light face recognition can be improved.

The electronic device 100 extracts a face image by synthesizing the fourth frame 540, the fifth frame 550, and the sixth frame 560 for face recognition, and performs noise removal on the extracted face image. can do. For example, a fourth frame 540 , a fifth frame 550 , and a sixth frame 560 are synthesized (or accumulated) by a temporal noise removal technique, a face image is extracted from the synthesized image, and the extracted The noise of the removed face image may be removed, so that the noise-removed face image may be output. The electronic device 100 may perform face recognition (eg, authentication) using the noise-removed face image.

In the case of the sixth frame 560 , the noise level may be relatively lower than that of the fourth frame 540 . For example, if the user's face is red and noise is seen in the fourth frame 540 , while the redness or noise is relatively less in the sixth frame 560 , when the sixth frame 560 is used, fast and accurate face recognition will be performed. can

In the example of FIG. 5 , the electronic device 100 uses three frames (eg, a fourth frame 540 , a fifth frame 550 , and a sixth frame 560 ) to thereby use one frame (eg, the fourth frame 560 ). Face recognition can be performed more efficiently than when the frame 540) is used alone or two frames (eg, the fourth frame 540 and the fifth frame 550) are used.

The electronic device 100 quickly extracts a frame set optimized for face recognition by skipping a frame in which a face region is not detected, prevents wasted time for face recognition due to unnecessary frames, and optimizes frames according to external illuminance Face recognition can be performed by acquiring three.

When the target frame number is 3, the fourth frame 540 is the first frame (n=1) in the frame set, the fifth frame 550 is the second frame (n=2) in the frame set, and the sixth frame ( 560 may be the third frame (n=3) in the frame set.

When the fourth frame 540 and the fifth frame 550 are not used alone for face recognition, and the number of frames in which the face region is detected reaches the target frame number, a composite image (or composite frame) of the corresponding frames is displayed on the face. can be used for recognition.

For example, the electronic device 100 may perform a face recognition operation using a face recognition model. The face recognition model may include a learning model trained using a rule model or an artificial intelligence algorithm. For example, face recognition (eg, authentication) may be performed by applying one or more frames or images having a predetermined standard (eg, resolution, quality, or brightness greater than or equal to a specified reference standard) to the face recognition model.

The electronic device 100 may first generate a composite image (or composite frame) at a level that the face recognition model can recognize by using the target number of frames without having to apply the face recognition model to all input frames. The electronic device 100 may improve the performance or speed of face recognition by determining the number of target frames, synthesizing frames equal to the number of target frames, and driving the face recognition model using the synthesized frames (or images).

According to various embodiments, as shown in the figure, low-illuminance noise may be effectively removed by applying a noise removing technique (eg, a temporal noise removing technique) in the low-illuminance face image. Frames in which no face region is detected (eg, the first frame 510 , the second frame 520 , and the third frame 530 ) are not used for face recognition, so that the time required for face recognition may be reduced. Face recognition may be performed based on a frame (or image) from which noise is removed by applying a noise removal technique to one or more frames in which a face region is detected.

6 is a screen example illustrating an example of using an electronic device according to an embodiment.

For example, face authentication of the user may be required to use a specific function (eg, unlock, log in, security, access) while a screen is displayed on the electronic device 100 . For example, the screen may be a home screen, a lock screen, a login screen, or an execution screen of an application (eg, an authentication application, a security application, a photographing application, a camera application).

The first screen 610 may be an authentication failure screen. Reference numeral 611 denotes a face photographing area. For example, the user may adjust the position of the head so that the face enters the face photographing area 611 while the camera module 140 of the electronic device 100 is activated. Reference numeral 615 may be a message window informing of authentication failure and re-search status.

When authentication fails by performing a face recognition procedure (eg, 461 and 470 of FIG. 4 ) regardless of external illumination in a low light state, the first screen 610 may be displayed. In this case, face authentication may be unnecessarily attempted or authentication failure may be repeated for frames in which the face region is not detected.

When authentication fails, the electronic device 100 may detect external illuminance and determine whether it is in a low illuminance state. In the case of low light, a face recognition procedure (eg, 440 to 459 and 470 of FIG. 4 ) according to the low light mode may be performed.

As an example, the electronic device 100 may determine the number of target frames based on external illuminance. As another example, the electronic device 100 may determine the number of target frames based on external illuminance and frame brightness information.

The electronic device 100 may obtain a frame set according to the target number of frames. Among input frames acquired through the camera module 140 , a frame in which a face region is not detected may be skipped, and only a frame in which a face region is detected may be included in the frame set. The frame set may acquire frames in which face regions are detected as many as the target frame number. The electronic device 100 may perform face recognition using the frame set.

The second screen 620 may be an authentication success screen. Reference numeral 625 may be a message window informing of successful authentication.

For example, the electronic device 100 may perform user authentication by comparing a face region in a frame (eg, the first frame) with a pre-stored face object.

When authentication is successful by performing a face recognition procedure (eg, 440 to 459 and 470 in FIG. 4 of FIG. 4 ) based on external illumination in a low light state, a second screen may be displayed. In this case, successful face authentication can be performed in a short time without face recognition attempts for unnecessary frames or noise removal.

7 is a screen example illustrating another use example of an electronic device according to an embodiment.

For example, a face recognition function may be used while a specific application (eg, a photographing application, a camera application, or an editing application) is executed in the electronic device 100 .

The processor 110 of the electronic device 100 may display the first screen 710 and the second screen 720 through the display 130 . The first screen 710 and the second screen 720 may be screens for recognizing faces of multiple (eg, two) users.

The first screen 710 may be a tuning failure screen. Reference numeral 711 denotes a face region of the first user. Reference numeral 713 may be a face region of the second user.

For example, the processor 110 extracts a face region within the frame and applies a correction item (eg, augmented reality data, facial expression data, beauty data, editing data, tuning data) according to the user's selection to the face region to the frame. can be updated. Reference numeral 715 may be a message window informing of a tuning failure and a re-search state.

When a tuning failure occurs by performing a face recognition procedure (eg, 461 and 470 of FIG. 4 ) regardless of external illumination in a low light state, the first screen 710 may be displayed. For example, the face of at least one of the two users may not be tuned to a desired standard standard (eg, image quality, resolution, or brightness). In this case, tuning may be unnecessarily attempted or tuning failure may be repeated for frames in which the face region is not detected.

When tuning fails, the electronic device 100 may detect external illuminance to determine whether the illuminance is in a low illuminance state. In the case of a low light state, a face recognition procedure (eg, 440 to 459 and 470 of FIG. 4 of FIG. 4 ) according to the low light mode may be performed.

As an example, the electronic device 100 may determine the number of target frames based on external illuminance. As another example, the electronic device 100 may determine the number of target frames based on external illuminance and frame brightness information.

The electronic device 100 may obtain a frame set according to the target number of frames. Among input frames acquired through the camera module 140 , a frame in which a face region is not detected may be skipped, and only a frame in which a face region is detected may be included in the frame set. The frame set may acquire frames in which face regions are detected as many as the target frame number. The electronic device 100 may perform face recognition using the frame set.

The second screen 720 may be a tuning success screen. Reference numeral 725 may be a message window informing of successful tuning.

For example, the electronic device 100 performs tuning successfully when the face regions 721 and 723 of the two users within the frame (eg, the first frame) satisfy the reference standard (eg, image quality, resolution, or brightness). can be performed. For example, upon successful tuning, an effect (eg, image quality improvement) to the face regions 721 and 723 may be applied or item synthesis (eg, augmented reality emoji) to the face regions 721 and 723 may be applied. have.

When tuning is successful by performing a face recognition procedure (eg, 440 to 459 and 470 in FIG. 4 of FIG. 4 ) based on external illuminance in a low illuminance state, the second screen 720 may be displayed. In this case, successful tuning can be performed within a short time without tuning attempts or noise removal for unnecessary frames.

8 is a block diagram of an electronic device 801 in a network environment 800 , according to various embodiments. Referring to FIG. 8 , in a network environment 800 , the electronic device 801 communicates with the electronic device 802 through a first network 898 (eg, a short-range wireless communication network) or a second network 899 . It may communicate with at least one of the electronic device 804 and the server 808 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 801 may communicate with the electronic device 804 through the server 808 . According to an embodiment, the electronic device 801 includes a processor 820 , a memory 830 , an input module 850 , a sound output module 855 , a display module 860 , an audio module 870 , and a sensor module ( 876), interface 877, connection terminal 878, haptic module 879, camera module 880, power management module 888, battery 889, communication module 890, subscriber identification module 896 , or an antenna module 897 . In some embodiments, at least one of these components (eg, the connection terminal 878 ) may be omitted or one or more other components may be added to the electronic device 801 . In some embodiments, some of these components (eg, sensor module 876 , camera module 880 , or antenna module 897 ) are integrated into one component (eg, display module 860 ). can be

The processor 820, for example, executes software (eg, a program 840) to execute at least one other component (eg, a hardware or software component) of the electronic device 801 connected to the processor 820 . It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 820 converts commands or data received from other components (eg, the sensor module 876 or the communication module 890 ) to the volatile memory 832 . may store the command or data stored in the volatile memory 832 , and store the resulting data in the non-volatile memory 834 . According to an embodiment, the processor 820 is a main processor 821 (eg, a central processing unit or an application processor) or a secondary processor 823 (eg, a graphics processing unit, a neural network processing unit) that can be operated independently or together with the main processor 821 (eg, a central processing unit or an application processor) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 801 includes a main processor 821 and a sub-processor 823 , the sub-processor 823 uses less power than the main processor 821 or is set to be specialized for a specified function. can The coprocessor 823 may be implemented separately from or as part of the main processor 821 .

The coprocessor 823 may, for example, act on behalf of the main processor 821 while the main processor 821 is in an inactive (eg, sleep) state, or when the main processor 821 is active (eg, executing an application). ), together with the main processor 821, at least one of the components of the electronic device 801 (eg, the display module 860, the sensor module 876, or the communication module 890) It is possible to control at least some of the related functions or states. According to one embodiment, the coprocessor 823 (eg, image signal processor or communication processor) may be implemented as part of another functionally related component (eg, camera module 880 or communication module 890 ). have. According to an embodiment, the auxiliary processor 823 (eg, a neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 801 itself on which the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 808). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 830 may store various data used by at least one component (eg, the processor 820 or the sensor module 876 ) of the electronic device 801 . The data may include, for example, input data or output data for software (eg, the program 840 ) and commands related thereto. The memory 830 may include a volatile memory 832 or a non-volatile memory 834 .

The program 840 may be stored as software in the memory 830 , and may include, for example, an operating system 842 , middleware 844 , or an application 846 .

The input module 850 may receive a command or data to be used in a component (eg, the processor 820 ) of the electronic device 801 from the outside (eg, a user) of the electronic device 801 . The input module 850 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 855 may output a sound signal to the outside of the electronic device 801 . The sound output module 855 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 860 may visually provide information to the outside (eg, a user) of the electronic device 801 . The display module 860 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 860 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 870 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 870 acquires a sound through the input module 850 or an external electronic device (eg, a sound output module 855 ) directly or wirelessly connected to the electronic device 801 . Sound may be output through the electronic device 802 (eg, a speaker or headphones).

The sensor module 876 detects an operating state (eg, power or temperature) of the electronic device 801 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to one embodiment, the sensor module 876 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 877 may support one or more designated protocols that may be used for the electronic device 801 to directly or wirelessly connect with an external electronic device (eg, the electronic device 802 ). According to an embodiment, the interface 877 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 878 may include a connector through which the electronic device 801 can be physically connected to an external electronic device (eg, the electronic device 802 ). According to an embodiment, the connection terminal 878 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 879 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 879 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 880 may capture still images and moving images. According to one embodiment, the camera module 880 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 888 may manage power supplied to the electronic device 801 . According to an embodiment, the power management module 888 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 889 may supply power to at least one component of the electronic device 801 . According to one embodiment, battery 889 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 890 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 801 and an external electronic device (eg, the electronic device 802, the electronic device 804, or the server 808). It can support establishment and communication performance through the established communication channel. The communication module 890 operates independently of the processor 820 (eg, an application processor) and may include one or more communication processors that support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 890 is a wireless communication module 892 (eg, a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 894 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 898 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 899 (eg, legacy). It may communicate with the external electronic device 804 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 892 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 896 within a communication network, such as the first network 898 or the second network 899 . The electronic device 801 may be identified or authenticated.

The wireless communication module 892 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 892 may support a high frequency band (eg, mmWave band) in order to achieve a high data rate, for example. The wireless communication module 892 uses various technologies for securing performance in a high-frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 892 may support various requirements specified in the electronic device 801 , an external electronic device (eg, the electronic device 804 ), or a network system (eg, the second network 899 ). According to an embodiment, the wireless communication module 892 may include a peak data rate (eg, 20 Gbps or more) for realization of eMBB, loss coverage for realization of mMTC (eg, 164 dB or less), or U-plane latency (for URLLC realization) ( Example: Downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 897 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 897 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 897 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network such as the first network 898 or the second network 899 is connected from the plurality of antennas by, for example, the communication module 890 . can be selected. A signal or power may be transmitted or received between the communication module 890 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 897 .

According to various embodiments, the antenna module 897 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 801 and the external electronic device 804 through the server 808 connected to the second network 899 . Each of the external electronic devices 802 or 804 may be the same as or different from the electronic device 801 . According to an embodiment, all or a part of operations executed by the electronic device 801 may be executed by one or more external electronic devices 802 , 804 , or 808 . For example, when the electronic device 801 needs to perform a function or service automatically or in response to a request from a user or other device, the electronic device 801 may instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 801 . The electronic device 801 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 801 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 804 may include an Internet of things (IoT) device. The server 808 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 804 or the server 808 may be included in the second network 899 . The electronic device 801 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C," each of which may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may simply be used to distinguish an element from other elements in question, and may refer elements to other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively”. When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 836 or external memory 838) readable by a machine (eg, electronic device 801). may be implemented as software (eg, the program 840) including For example, a processor (eg, processor 820 ) of a device (eg, electronic device 801 ) may call at least one of one or more instructions stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided in a computer program product (computer program product). Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly, online between smartphones (eg: smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. have. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. , or one or more other operations may be added.

9 is a block diagram 900 illustrating a camera module 880, in accordance with various embodiments. Referring to FIG. 9 , the camera module 880 includes a lens assembly 910 , a flash 920 , an image sensor 930 , an image stabilizer 940 , a memory 950 (eg, a buffer memory), or an image signal processor. (960). The lens assembly 910 may collect light emitted from a subject, which is an image to be captured. Lens assembly 910 may include one or more lenses. According to an embodiment, the camera module 880 may include a plurality of lens assemblies 910 . In this case, the camera module 880 may form, for example, a dual camera, a 360 degree camera, or a spherical camera. Some of the plurality of lens assemblies 910 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may be a different lens assembly. It may have one or more lens properties that are different from the lens properties of . The lens assembly 910 may include, for example, a wide-angle lens or a telephoto lens.

The flash 920 may emit light used to enhance light emitted or reflected from the subject. According to an embodiment, the flash 920 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp. The image sensor 930 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 910 into an electrical signal. According to an embodiment, the image sensor 930 is, for example, one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, the same It may include a plurality of image sensors having properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 930 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 940 moves at least one lens or the image sensor 930 included in the lens assembly 910 in a specific direction in response to the movement of the camera module 880 or the electronic device 801 including the same. Operation characteristics of the image sensor 930 may be controlled (eg, read-out timing may be adjusted, etc.). This makes it possible to compensate for at least some of the negative effects of the movement on the image being taken. According to an embodiment, the image stabilizer 940 is, according to an embodiment, the image stabilizer 940 is a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 880 . Such a movement of the camera module 880 or the electronic device 801 may be detected using . According to an embodiment, the image stabilizer 940 may be implemented as, for example, an optical image stabilizer. The memory 950 may temporarily store at least a portion of the image acquired through the image sensor 930 for a next image processing operation. For example, when image acquisition is delayed according to the shutter or a plurality of images are acquired at high speed, the acquired original image (eg, a Bayer-patterned image or a high-resolution image) is stored in the memory 950 and , a copy image corresponding thereto (eg, a low-resolution image) may be previewed through the display module 860 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a portion of the original image stored in the memory 950 may be obtained and processed by, for example, the image signal processor 960 . According to an embodiment, the memory 950 may be configured as at least a part of the memory 830 or as a separate memory operated independently of the memory 830 .

The image signal processor 960 may perform one or more image processing on an image acquired through the image sensor 930 or an image stored in the memory 950 . The one or more image processes may include, for example, depth map generation, three-dimensional modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring), sharpening (sharpening), or softening (softening) Additionally or alternatively, the image signal processor 960 includes at least one of the components included in the camera module 880 (eg, an image sensor). 930), for example, exposure time control, readout timing control, etc. The image processed by the image signal processor 960 is stored back in the memory 950 for further processing. or may be provided as an external component of the camera module 880 (eg, the memory 830, the display module 860, the electronic device 802, the electronic device 804, or the server 808). According to an example, the image signal processor 960 may be configured as at least a part of the processor 820 or a separate processor operated independently of the processor 820. The image signal processor 960 may be configured as the processor 820 . and a separate processor, the at least one image processed by the image signal processor 960 may be displayed through the display module 860 as it is by the processor 820 or after additional image processing.

According to an embodiment, the electronic device 801 may include a plurality of camera modules 880 each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 880 may be a wide-angle camera, and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules 880 may be a front camera, and at least the other may be a rear camera.

The electronic device (eg, the electronic device 100 of FIG. 1 , the electronic device 801 of FIG. 8 ) according to various embodiments includes at least one sensor (eg, the sensor module 150 of FIG. 1 , the sensor of FIG. 8 ) module 876), a memory (eg, memory 120 of FIG. 1, memory 830 of FIG. 8), and at least one processor (eg, processor of FIG. 1) coupled to the at least one sensor and the memory 110) and the processor 820 of FIG. 8). When the memory is executed, the at least one processor detects external illuminance through the at least one sensor, determines a target frame number based on the external illuminance, and uses a camera according to the target frame number Instructions for acquiring a frame set, performing noise reduction (NR) on the frame set, and performing face recognition using the noise-removed frame set may be stored.

According to various embodiments, each frame in the frame set may be a frame including a face region among input frames acquired through the camera.

According to various embodiments, noise removal may be performed on a face region excluding a peripheral region in the frame set, and face recognition may be performed using the noise-removed face region.

According to various embodiments, the target frame number may be determined based on the external illuminance and brightness information of a frame in the frame set.

According to various embodiments, the brightness information of the frame may include at least one of brightness information of a face region of the frame, brightness information of an entire region of the frame, and information indicating a difference in brightness between a face region in the frame and a surrounding region. may include

According to various embodiments, as the external illuminance decreases, the number of target frames may increase.

According to various embodiments, whether a low illuminance state is determined based on the external illuminance may be determined, and if the low illuminance state, the target frame number may be determined in response to the external illuminance.

According to various embodiments, when a face region is detected in the input frame acquired through the camera, the input frame is included in the frame set, and when the face region is not detected in the input frame, the input frame is included in the frame three may be excluded.

According to various embodiments, face recognition may be performed using an input frame obtained through the camera in a normal illuminance state.

According to various embodiments, the noise reduction may be performed by a temporal noise reduction (NR) technique.

A face recognition method of an electronic device according to various embodiments of the present disclosure includes an operation of detecting external illuminance, determining the number of target frames based on the external illuminance, acquiring a frame set according to the number of target frames through a camera; It may include an operation of performing noise reduction (NR) on the frame set, and an operation of performing face recognition using the noise reduction frame set.

According to various embodiments, each frame in the frame set may be a frame including a face region among input frames acquired through the camera.

According to various embodiments, noise removal may be performed on a face region excluding a peripheral region in the frame set, and face recognition may be performed using the noise-removed face region.

According to various embodiments, the target frame number may be determined based on the external illuminance and brightness information of a frame in the frame set.

According to various embodiments, the brightness information of the frame may include brightness information of a face region of the frame, brightness information of an entire region of the frame, and information indicating a difference in brightness between the face region and surrounding regions in the frame. can

According to various embodiments, as the external illuminance decreases, the number of target frames may increase.

According to various embodiments, whether a low illuminance state is determined based on the external illuminance may be determined, and if the low illuminance state, the target frame number may be determined in response to the external illuminance.

According to various embodiments, the acquiring of the frame set includes: when a face region is detected in the input frame acquired through the camera, including the input frame in the frame set; and the face region in the input frame. and excluding the input frame from the frame set when this is not detected.

According to various embodiments, in a normal illuminance state, a face may be recognized using an input frame obtained through the camera.

According to various embodiments, the noise reduction may be performed by a temporal noise reduction (NR) technique.

Claims (20)

In an electronic device,
at least one sensor;
Memory; and
at least one processor coupled to the at least one sensor and the memory;
The memory, when executed, the at least one processor,
Detecting external illuminance through the at least one sensor,
determining the number of target frames based on the external illuminance,
Acquire a frame set according to the target number of frames through a camera,
performing noise reduction (NR) on the frame set,
An electronic device for storing instructions for performing face recognition using the noise-removed frame set. According to claim 1,
Each frame in the frame set is
An electronic device that is a frame including a face region among the input frames acquired through the camera. According to claim 1,
Noise removal is performed on the face area excluding the surrounding area in the frame set,
An electronic device for performing face recognition using the noise-removed face region. According to claim 1,
The electronic device for determining the number of target frames based on the external illuminance and brightness information of a frame in the frame set. 5. The method of claim 4,
The brightness information of the frame,
brightness information of a face region in the frame;
brightness information for the entire area of the frame; and
The electronic device comprising at least one of information indicating a difference in brightness between a face region and a surrounding region within the frame. According to claim 1,
The electronic device in which the number of the target frames increases as the external illuminance decreases. According to claim 1,
It is determined whether the state is in a low-illuminance state based on the external illuminance,
In the case of the low illuminance state, the electronic device determines the number of target frames in response to the external illuminance. According to claim 1,
When a face region is detected in the input frame acquired through the camera, the input frame is included in the frame set;
When a face region is not detected in the input frame, the input frame is excluded from the frame set. 8. The method of claim 7,
An electronic device in which face recognition is performed using an input frame obtained through the camera in a normal illumination state. According to claim 1,
An electronic device in which the noise reduction is performed by a temporal noise reduction (NR) technique. In the face recognition method of an electronic device,
Detecting external illuminance;
determining the number of target frames based on the external illuminance;
acquiring a frame set according to the target number of frames through a camera;
performing noise reduction (NR) on the frame set; and
and performing face recognition using the noise-removed frame set. 12. The method of claim 11,
Each frame in the frame set is
a frame including a face region among the input frames acquired through the camera. 12. The method of claim 11,
Noise removal is performed on the face area excluding the surrounding area in the frame set,
A method of performing face recognition using the noise-removed face region. 12. The method of claim 11,
The target number of frames is determined based on the external illuminance and brightness information of a frame in the frame set. 15. The method of claim 14,
The brightness information of the frame,
brightness information of a face region in the frame;
brightness information for the entire area of the frame; and
and information indicating a difference in brightness between a face region and a surrounding region within the frame. 12. The method of claim 11,
and the number of the target frames increases as the external illuminance decreases. 12. The method of claim 11,
It is determined whether the state is in a low-illuminance state based on the external illuminance,
In the case of the low illuminance state, the number of the target frames is determined in response to the external illuminance. 12. The method of claim 11,
The operation of obtaining the frame set includes:
when a face region is detected in the input frame acquired through the camera, including the input frame in the frame set; and
and excluding the input frame from the frame set when a face region is not detected in the input frame. 18. The method of claim 17,
A method in which face recognition is performed using an input frame obtained through the camera in the case of a normal illumination state. 12. The method of claim 11,
A method in which the noise reduction is performed by a temporal noise reduction (NR) technique.

Images