KR20210003916A - Multiple registrations of facial recognition - Google Patents

Abstract

Face recognition authentication on a device with a camera can operate with multiple registration profiles on the device. Multiple registration profiles may include separate profiles for different appearances of a user and/or separate profiles for different users authorized to use the device. Registration profiles may be created using a registration process in which the registration process is operated separately to create each of the different registration profiles. During the facial recognition authentication process, the user may unlock the device by having a matching score for the user's captured image(s) that exceeds the unlock threshold for at least one of the enrollment profiles. A user may have a matching score that exceeds the unlock threshold for multiple registration profiles. Each registration profile that unlocks the device for a user can be updated using a template update process that operates independently for each registration profile.

Description

Multiple registrations of facial recognition

Embodiments described herein relate to methods and systems for face detection and recognition in images captured by a camera on a device. More specifically, embodiments described herein relate to the use of multiple profiles for facial recognition and methods for updating profiles during use of a device.

Biometric authentication processes are being used more frequently to allow users to more easily access their devices without the need for passcode or password authentication. An example of a biometric authentication process is fingerprint authentication using a fingerprint sensor. Facial recognition is another biometric process that can be used to authenticate an authorized user of a device. Facial recognition processes are generally used to identify individuals in an image and/or compare the individuals in the images to a database of individuals to match the faces of the individuals.

In the case of authentication using facial recognition, a facial recognition system is generally used over time so that the user can still access the device using facial recognition even when the user's facial features change and create differences in the user's images It tries to adapt to changes in the facial features of the authorized user. For example, the user's facial features may change over time due to beard changes, haircuts, weight gain/loss, and/or aging.

There may also be a need for a device that allows multiple (two or more) subscription profiles on the device. In some cases, if the user has two or more distinct (or various) appearance features, multiple registration profiles may be required. Distinguishing appearance features include, for example, not wearing glasses versus wearing glasses, having a shaved face versus having a bearded face, and/or having a face with dark makeup versus a face without makeup. It may include appearances that are distinguished by a face. Additionally, if a user is required to have more than to access the device using facial recognition, multiple enrollment profiles may be utilized.

However, the facial recognition system needs to be kept secure with the use of multiple registration profiles while allowing the facial recognition system to adapt to changes in the user(s) appearance in each of the different registration profiles. Thus, the facial recognition system needs to adapt to the changes, while also ensuring that differences are recognized as changes to a particular registration profile.

Multiple registration profiles can be used for facial recognition authentication on a device with a camera. Registration profiles may include profiles for different appearances of different users and/or a user authorized to use the device. The registration process can be used to create templates for each registration profile on the device.

When a user attempts to gain access to a device using facial recognition authentication, features from the user's captured image(s) may be compared to each of the enrollment profiles on the device. The user may gain access to the device by having a matching score with at least one of the registration profiles exceeding the unlock threshold for the facial recognition authentication process. Templates for the registration profile used to unlock the device can be updated with features from the user's captured image(s) using a template update process. If the user has a matching score above the unlock threshold for multiple registration profiles, the templates in each of the registration profiles can be updated using separate template processes operating in parallel on the device. Thus, the templates for each registration profile can be updated by the device as the user's characteristics for the registration profile change over time.

The features and advantages of the methods and apparatus of the embodiments disclosed herein are preferred at the present time according to the embodiments disclosed herein when taken in conjunction with the accompanying drawings, but nonetheless the following of exemplary embodiments. It will be more fully understood with reference to the detailed description.
1 shows a representation of an embodiment of a device including a camera.
2 shows a representation of an embodiment of a camera.
3 shows a representation of an embodiment of a processor on a device.
4 shows a flowchart of an embodiment of an image registration process for an authorized user of a device.
5 shows a representation of an embodiment of a feature space with feature vectors after the registration process.
6 shows an embodiment representation of a template space for a registration profile in the device's memory.
7 shows a flowchart of an embodiment of registering multiple registration profiles on a device.
8 shows a representation of an embodiment of two template spaces of the device's memory.
9 shows a flowchart of an embodiment of a face recognition authentication process.
10 shows a flowchart of an embodiment of a template update process.
11 shows a representation of an embodiment of a template space represented as a feature space.
12 shows a flowchart of an embodiment of a template update subprocess.
13 shows a flowchart of an additional embodiment of the template update process.
14 shows a representation of an additional embodiment of a template space expressed as a feature space.
15 shows a flowchart of an embodiment of a facial recognition authentication process for use with multiple registration profiles.
16 shows a flowchart of an embodiment of a locking protocol.
17 shows a representation of an embodiment of a template space with two sets of dynamic templates.
18 shows a flowchart of an embodiment of a facial recognition authentication process for use with multiple registration profiles.
19 shows a flowchart of another embodiment of the locking protocol.
20 shows a flowchart of another embodiment of the locking protocol.
21 shows a block diagram of one embodiment of an exemplary computer system.
22 shows a block diagram of one embodiment of a computer-accessible storage medium.
While the embodiments described in this disclosure may allow various modifications and alternative forms, certain embodiments of the disclosure are shown by way of example in the drawings and will be described in detail herein. However, the drawings and detailed description thereof are not intended to limit the embodiments to the specific form disclosed, on the contrary, the intention is to all modifications, equivalents and alternatives falling within the spirit and scope of the appended claims. It is to be understood that it is intended to cover them. The headings used in this specification are for constitutional purposes only and are not intended to be used to limit the scope of the description. As used throughout this application, the word "may" is not an obligatory meaning (ie, meaning "must"), but rather a meaning of permissive (ie, "for It is used as "to have a possibility"). Similarly, the words "include, includes" and "including" are meant to include, but are not limited to.
Various units, circuits or other components may be described as being “configured” to perform a task or tasks. In such contexts, “configured to” is a broad description of a structure that generally means “having circuitry” to perform a task or tasks during operation. As such, the unit/circuit/component may be configured to perform a task even when the unit/circuit/component is not currently in an on state. In general, a circuit portion forming a structure corresponding to "configured to" may include hardware circuits and/or memory that store executable program instructions to implement an operation. The memory may include volatile memory such as static or dynamic random access memory and/or non-volatile memory such as optical or magnetic disk storage, flash memory, programmable read only memories, and the like. Hardware circuits include combinatorial logic circuitry, clocked storage devices such as flops, registers, latches, etc., finite state machines, static random access memory or embedded dynamic random Memory such as access memory, custom designed circuitry, programmable logic arrays, and the like. Similarly, various units/circuits/components may be described as performing a task or tasks for convenience of description. Such description is to be construed as including the phrase "configured to". Reference to a unit/circuit/component configured to perform one or more tasks is expressly intended not to apply the 35 USC §112(f) interpretation to that unit/circuit/component.
In an embodiment, hardware circuits according to the present disclosure may be implemented by coding the description of the circuit in a hardware description language (HDL) such as Verilog or VHDL. HDL descriptions can be synthesized against a library of cells designed for a given integrated circuit manufacturing technique and modified for timing, power and other reasons to create a final design database, which can be sent to the foundry and masked. And ultimately create an integrated circuit. Some hardware circuits, or portions thereof, can also be custom designed with a schematic editor and captured into an integrated circuit design along with the synthesized circuit portion. The integrated circuit may include transistors, and may further include other circuit elements (eg, passive elements such as capacitors, resistors, inductors, etc.) and interconnects between the transistors and the circuit elements. Some embodiments may implement multiple integrated circuits coupled together to implement hardware circuits and/or discrete elements may be used in some embodiments.
The scope of this disclosure is to (expressly or implicitly) any feature or combination of features disclosed herein, or any generalization thereof, whether or not alleviating any or all of the problems addressed herein. Includes. Accordingly, new claims may be drawn up during examination of this application (or an application claiming priority thereto) for any such combination of features. Specifically, with reference to the appended claims, features from dependent claims may be combined with features of independent claims, and features from respective independent claims are in specific combinations listed in the appended claims. Not only can they be combined, but they can be combined in any suitable way.

This specification includes references to “one embodiment” or “an embodiment”. Unless expressly disclaimed herein, although embodiments including any combination of features are generally contemplated, the appearances of the phrases “in one embodiment” or “in an embodiment” necessarily refer to the same embodiment. I don't. Certain features, structures, or characteristics can be combined in any suitable manner that is consistent with the present disclosure.

As described herein, one aspect of the technology is to collect and use data available from various sources to improve operation and access to devices. The present disclosure contemplates that in some cases, such collected data may include personal information data that may be used to uniquely identify, contact, or locate a particular individual. Such personal data may include image data (e.g., data from images of a user), demographic data, location-based data, phone numbers, email addresses, home addresses, or any other identifying information. have. In the case of image data, personal information data may include only data from images of a user, not images themselves.

The present disclosure recognizes that the use of such personal information data in the present technology may be used to benefit users. For example, personal information data can be used to control unlocking and/or authorizing devices using facial recognition. Thus, the use of such personal information data enables calculated control of access to devices. In addition, other uses for personal information data that benefit the user are also contemplated by the present disclosure.

The present disclosure further contemplates that entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and/or privacy practices. In particular, such entities should implement and continue to use privacy policies and practices that are generally perceived as meeting or exceeding industrial or administrative requirements for keeping personal data data private and secure. For example, in the case of using facial recognition to unlock and/or authorize devices, personal information from users should be collected for legitimate and reasonable uses of the entity, such uses being specific to the operation of the devices and Because they are not shared or sold beyond legitimate uses. In addition, such collection should occur only after receiving the users' prior consent, and personal information data should be kept secure on the devices for which personal information was collected. Additionally, such entities will take any necessary steps to secure and secure access to such personal information data and ensure that others with access to personal information data adhere to their privacy policies and procedures. Additionally, such entities may themselves be evaluated by third parties to demonstrate their adherence to widely recognized privacy policies and practices.

Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements may be provided to prevent or block access to such personal information data. For example, the technology allows users to choose to “opt in” or “opt out” of participation in the collection of personal information data during registration for services. Can be configured.

1 shows a representation of an embodiment of a device including a camera. In certain embodiments, device 100 includes camera 102, processor 104, memory 106, and display 108. Device 100 may, in some cases, be a small computing device, which in some cases may be small enough to be handheld (thus also commonly known as a handheld computer or simply handheld). In certain embodiments, device 100 is any of various types of computer system devices that are mobile or portable and perform wireless communications using WLAN communication. Examples of mobile devices include mobile phones or smart phones, and tablet computers. A variety of different types of devices include wearables such as smart watches, smart glasses, headphones, pendants, earpieces, as well as laptop computers, portable gaming devices, portable Internet devices, and other handheld devices. If it includes wireless or RF communication capabilities (eg, Wi-Fi, cellular, and/or Bluetooth), such as devices, it may fall into this category. In general, the term “mobile device” refers to any electronic, computing, and/or electronic device that is easily transported by a user and capable of wireless communication using, for example, WLAN, Wi-Fi, cellular, and/or Bluetooth. /Or can be broadly defined to include a communication device (or combination of devices). In certain embodiments, device 100 includes a processor 104, a memory 106, and any device used by a user having a display 108. Display 108 may be, for example, an LCD screen or a touch screen. In some embodiments, display 108 includes a user input interface for device 100 (eg, the display enables interactive input for a user).

Camera 102 may be used to capture images of the external environment of device 100. In certain embodiments, camera 102 is positioned to capture images in front of display 108. The camera 102 may be positioned to capture images of the user (eg, the user's face) while the user interacts with the display 108. 2 shows a representation of an embodiment of the camera 102. In certain embodiments, camera 102 includes one or more lenses and one or more image sensors 103 for capturing digital images. Digital images captured by camera 102 may include, for example, still images, video images, and/or frame-by-frame images.

In certain embodiments, camera 102 includes an image sensor 103. The image sensor 103 may be an array of sensors, for example. The sensors in the sensor array may include, but are not limited to, charge coupled device (CCD) and/or complementary metal oxide semiconductor (CMOS) sensor elements to capture infrared images (IR) or other non-visible electromagnetic radiation. Does not. In some embodiments, camera 102 includes more than one image sensor to capture multiple types of images. For example, camera 102 may include both IR sensors and RGB (red, green, and blue) sensors. In certain embodiments, camera 102 includes illuminators 105 for illuminating surfaces (or subjects) with different types of light detected by image sensor 103. For example, the camera 102 includes an illuminator for visible light (e.g., a flash illuminator), illuminators for RGB light, and/or illuminators for infrared light (e.g., a flood IR source and a speckle pattern projector). Can include. In some embodiments, the flood IR source and speckle pattern projector are different wavelengths of light (eg, not infrared). In certain embodiments, the illuminators 105 include an array of light sources, such as, but not limited to, vertical-cavity surface-emitting lasers (VCSELs). In some embodiments, image sensors 103 and illuminators 105 are included in a single chip package. In some embodiments, image sensors 103 and illuminators 105 are located on separate chip packages.

In certain embodiments, the image sensor 103 is an IR image sensor and the image sensor is used to capture infrared images used for face detection, face recognition authentication, and/or depth detection. Other embodiments of the image sensor 103 (eg, an RGB image sensor) may also be contemplated for use in face detection, face recognition authentication, and/or depth detection as described herein. For face detection, the illuminator 105A may provide a flood IR illumination to flood the subject with IR illumination (eg, IR flashlight), and the image sensor 103 captures images of the flooded IR illuminated subject. You can capture it. Flooded IR illumination images may be, for example, two-dimensional images of a subject illuminated by IR light. For depth detection or to generate a depth map image, the illuminator 105B may provide IR illumination with a speckle pattern. The speckle pattern can be a pattern of light spots (eg, a pattern of dots) having a known, controllable configuration and pattern projected onto the subject. The illuminator 105B may include a VCSEL array configured to form a speckle pattern or a light source configured to form a speckle pattern and patterned transparency. The configuration and pattern of the speckle pattern provided by the illuminator 105B may be selected, for example, based on a desired speckle pattern density (eg, dot density) in the subject. The image sensor 103 may capture images of a subject illuminated by the speckle pattern. The captured image of the speckle pattern on the subject is an imaging and processing system (e.g., an image signal processor as described herein) to generate or estimate a three-dimensional map of the subject (e.g., a depth map or depth map image of the subject). (ISP)) can be evaluated (e.g., analyzed and/or processed). Examples of depth map imaging are in US Pat. No. 8,150,142 to Freedman et al., US Pat. No. 8,749,796 to Pesach et al., and US Pat. No. 8,384,997 to Shpunt et al. It is described in U.S. Patent Application Publication No. 2016/0178915 to Mor et al., which is incorporated by reference as if fully described herein.

In certain embodiments, the images captured by camera 102 include images with the user's face (eg, the user's face is contained within the images). An image with a user's face may include any digital image in which at least a portion of the user's face is shown within a frame of the image. Such an image may only include the user's face, or may include the user's face in a smaller portion or portion of the image. The user's face may be captured with sufficient resolution in the image to enable image processing of one or more features of the user's face in the image.

Images captured by camera 102 may be processed by processor 104. 3 shows a representation of an embodiment of a processor 104 included in device 100. The processor 104 may include circuitry configured to execute instructions defined by an instruction set architecture implemented by the processor. The processor 104 may execute main control software of the device 100, such as an operating system. In general, software executed by processor 104 during use can control other components of device 100 to realize the desired functionality of the device. Processors can also run other software. These applications can provide user functionality and can rely on the operating system for low-level device control, scheduling, memory management, etc.

In certain embodiments, the processor 104 includes an image signal processor (ISP) 110. ISP 110 may include circuitry (eg, image signal processing circuitry) suitable for processing images received from camera 102. ISP 110 may include any hardware and/or software (eg, program instructions) capable of processing or analyzing images captured by camera 102.

In certain embodiments, the processor 104 includes a secure enclave processor (SEP) 112. In some embodiments, SEP 112 is involved in a facial recognition authentication process that involves images being captured by camera 102 and processed by ISP 110. The SEP 112 may be a security circuit configured to authenticate an active user (eg, a user currently using the device 100) as authorized to use the device 100. A "security circuit" may be a circuit that protects an isolated internal resource from being directly accessed by an external circuit. The internal resource may be a memory (eg, memory 106) that stores sensitive data such as personal information (eg, biometric information, credit card information, etc.), encryption keys, random number generator seeds, and the like. The internal resource may also be circuitry that performs services/operations associated with sensitive data. As described herein, SEP 112 may include any hardware and/or software (eg, program instructions) capable of authenticating a user using a facial recognition authentication process. The facial recognition authentication process may authenticate the user by capturing images of the user with the camera 102 and comparing the captured images to previously collected images of the authorized user for the device 100. In some embodiments, the functions of ISP 110 and SEP 112 may be performed by a single processor (e.g., either ISP 110 or SEP 112 may perform both functions and the other The processor can be omitted).

In certain embodiments, the processor 104 is a registration process (e.g., as shown in FIG. 4) to capture images (e.g., previously collected images) for an authorized user of the device 100 The image registration process 200, or registration process) is performed. During the registration process, the camera module 102 is configured to allow the SEP 112 (or other security process) to subsequently authenticate the user using the facial recognition authentication process, and images and/or images from the authorized user. Data can be captured (eg, collected). In some embodiments, images and/or image data from the registration process (eg, feature vector data from images) are used to create templates at device 100. The templates may be stored, for example, in a template space in the memory 106 of the device 100. In some embodiments, the template space may be updated by addition of templates and/or subtraction of templates from the template space. The template update process (e.g., the first template update process 300 and/or the second template update process 400 described herein) includes the processor 104 to add templates and/or remove templates from the template space. Can be done by For example, the template space may be updated with additional templates to adapt to changes in the appearance of the authorized user and/or changes in hardware performance over time. Templates can be removed from the template space to compensate for the addition of templates when the template space for storing the templates is full.

In some embodiments, the camera module 102 captures multiple image pairs for a facial recognition session. Each pair may include an image captured using a two-dimensional capture mode (eg, a flood IR image) and an image captured using a three-dimensional capture mode (eg, a depth map image). In certain embodiments, ISP 110 and/or SEP 112 processes flood IR images and depth map images independently of each other before a final authentication decision is made for the user. For example, ISP 110 may independently process the images to determine characteristics of each image individually. The SEP 112 then compares the individual image characteristics with the stored templates for each type of image, and the authentication score for each individual image (e.g., the match score between the user in the captured image and the user in the stored templates Or other matching rankings). Authentication scores for individual images (e.g., flood IR and depth map images) are combined to make a decision about the user's identity, and if authenticated, the user will use the device 100 (e.g., unlock the device) Can make it possible.

In some embodiments, ISP 110 and/or SEP 112 combines the images in each pair to provide a composite image used for facial recognition. In some embodiments, the ISP 110 may process the composite image to determine the characteristics of the image, and the SEP 112 compares it to the stored templates to determine the user's identity, and if authenticated, the user The device 100 may be allowed to be used.

In some embodiments, the combination of the flood IR image data and the depth map image data may cause the SEP 112 to compare faces in a three-dimensional space. In some embodiments, the camera module 102 communicates the image data to the SEP 112 over a secure channel. A secure channel is, for example, either a dedicated path for communicating data (i.e., a path shared only by the intended participants) or a dedicated path for communicating encrypted data using encryption keys known only to the intended participants. It can be one. In some embodiments, camera module 102 and/or ISP 110 perform various processing operations on image data prior to supplying image data to SEP 112 to facilitate comparisons performed by SEP. can do.

In certain embodiments, processor 104 runs one or more machine learning models. Machine learning models may be operated using any combination of hardware and/or software (eg, program instructions) located within processor 104 and/or on device 100. In some embodiments, one or more neural network modules 114 are used to run machine learning models on device 100. Neural network modules 114 may be located in ISP 110 and/or SEP 112.

The neural network module 114 may include any combination of hardware and/or software (eg, program instructions) located within the processor 104 and/or on the device 100. In some embodiments, the neural network module 114 is a multi-scale neural network or other neural network in which the scale of kernels used in the network may vary. In some embodiments, the neural network module 114 is a cyclic neural network (RNN), such as, but not limited to, a gated recurrent unit (GRU) circulatory neural network or a long short-term memory (LSTM) circulatory neural network.

The neural network module 114 may include a neural network circuit unit installed or configured with operating parameters learned by a neural network module or a similar neural network module (eg, a neural network module operating on a different processor or device). For example, the neural network module may be trained using training images (eg, reference images) and/or other training data to generate operating parameters for the neural network circuitry. The operating parameters generated from the training may then be provided to the neural network module 114 installed on the device 100. Providing the operating parameters generated from training to the neural network module 114 on the device 100 enables the neural network module to operate using training information programmed in the neural network module (e.g., generated by training ( training-generated) operating parameters can be used by the neural network module to evaluate and operate on images captured by the device).

4 shows a flowchart of an embodiment of an image registration process 200 for an authorized user of device 100. Process 200 may be used to create a registration profile for an authorized user of device 100 stored on the device (eg, in memory coupled to SEP 112 ). The enrollment profile may include one or more templates for authorized users created using process 200. The enrollment profile and templates associated with the enrollment profile may be used in the facial recognition process to allow (eg, authorize) the user to use the device and/or perform operations on the device (eg, unlock the device).

In some embodiments, an authorized user may create multiple registration profiles, with each registration profile created by the image registration process 200 associated with the user's specific images taken during the registration process. For example, the authorized user may create a first registration profile associated with the user's first distinct appearance (eg, the face of a user wearing glasses). The authorized user may also create a second registration profile associated with the user's second distinct appearance (eg, the face of a user without glasses). In some embodiments, multiple registration profiles may include one or more registration profiles for additional users of the device (e.g., a primary authorized user allows a secondary authorized user to register a profile on the device. can do). When the user is authenticated using any of the registration profiles created on the device, the user is authorized to use the device (e.g., unlocking the device), and multiple registration profiles can be used in parallel in the face recognition process. have.

In certain embodiments, process 200 is used when device 100 is first used by an authorized user and/or when the user chooses to create an enrollment profile in a facial recognition process. For example, process 200 may be initiated when device 100 is first acquired by an authorized user (e.g., purchased by an authorized user) and turned on for the first time by an authorized user. . In some embodiments, process 200 allows the user to enroll in a facial recognition process, update security settings for device 100, re-enroll, and/or add an enrollment profile on the device. When desired, it can be initiated by an authorized user.

In certain embodiments, process 200 begins with authenticating the user at 202. At 202, the user may be authenticated on device 100 using a non-face authentication process. For example, the user may be authenticated as an authorized user by entering a passcode, entering a password, or using a user authentication protocol other than face recognition. After the user is authenticated at 202, one or more registration (eg, reference or registration) images of the user are captured at 204. The registration images include images of the user illuminated by flood illuminator 105A (e.g., flood IR images) and/or images of the user illuminated by speckle illuminator 105B (e.g., depth map images). Can include. As described herein, flood IR images and depth map images may be used independently and/or in combination in facial recognition processes on device 100 (e.g., images are independent to provide an authentication decision). And these decisions can be combined to determine the final decision on user authentication).

Enrolled images may be captured using camera 102 when a user interacts with device 100. For example, registration images may be captured when the user follows prompts on the display 108 of the device 100. Prompts may include instructions for the user to take different motions and/or poses while registration images are being captured. During 204, camera 102 may capture multiple images for each motion and/or pose performed by the user. Capturing images of the user's different motions and/or different poses-these images still have a relatively clear depiction of the user-is in a limited or constrained position relative to the camera 102 on the device 100. It can be useful to provide a better variety of registration images that allow users to be authenticated without having to be present.

After multiple registration images have been captured at 204, selection of registration images for further image processing may be made at 206. The selection 206 of registration images, and further processing of the images may be performed by the ISP 110 and/or the SEP 112. Selection of registration images for further processing may include selecting images suitable for generating templates. For example, selecting images suitable for use in creating templates at 206 may include evaluating one or more selected criteria for the images and selecting images that meet the selected criteria. The selected images can be used to create templates for the user. The selected criteria are that the user's face is in the field of view of the camera, that the pose of the user's face is appropriate (e.g., that the user's face is not turned too far in any direction from the camera (i.e., face pitch, yaw , And/or the roll does not exceed certain levels)), the distance between the camera 102 and the user's face is within a selected distance range, the user's face having an occlusion less than a minimum value (e.g., The user's face is not occluded (blocked) by other objects by more than a minimum amount), the user pays attention to the camera (e.g., the user's eyes are looking at the camera), the user's eyes are not closed. Things, and appropriate lighting (lighting) in the image, but is not limited to these. In some embodiments, if more than one face is detected in the registration image, the registration image is rejected and not used for further processing (eg, not selected). The selection of images suitable for further processing may be a rule based on the images meeting a predetermined number of selected criteria or all of the selected criteria. In some embodiments, occlusion maps and/or landmark feature maps are used to identify the user's features in the images (e.g., facial features such as eyes, nose, and mouth) and to evaluate selected criteria in the images. Is used.

After the images are selected at 206, the user's characteristics in the selected (template) images may be encoded at 208. Encoding the selected images may include encoding the user's features (eg, facial features) to define features in the images as one or more feature vectors in the feature space. Feature vectors 210 may be the output of the encoding at 208. The feature space may be an n-dimensional feature space. The feature vector may be an n-dimensional vector of numerical values defining features from the image in the feature space (eg, the feature vector may be a vector of numerical values defining the user's facial features in the image).

5 shows a representation of an embodiment of a feature space 212 having feature vectors 210. Each feature vector 210 (black dot) may define facial features for the user from any one of a single image, a composite image (e.g., a composite of several images), or multiple images. I can. When the feature vectors 210 are generated from the facial features of a single user, the feature vectors can be similar to each other because the feature vectors are associated with the same person, as shown by the circle 211 in FIG. , Can have some "clustering". Feature vectors 256A and 256B (open diamonds) are feature vectors obtained from the facial recognition process 250, described below.

As shown in FIG. 4, process 200 may include, at 214, storing feature vectors 210 in memory of device 100 (e.g., memory protected by SEP 112). have. In certain embodiments, feature vectors 210 are stored as static templates 216 (e.g., registration templates or reference templates) in a template space (e.g., template space 220 described below) in memory. do. Static templates 216 may be used for a registration profile created by process 200. In some embodiments, static templates 216 (and other templates described herein) are for feature vectors obtained from registered flood IR images and for feature vectors obtained from registered depth map images. Includes individual templates for It should be understood that individual templates obtained from flood IR images and depth map images may be used independently and/or in combination during the additional processes described herein. For simplicity in this disclosure, static templates 216 are generically described, and static templates 216 (and use of templates) are obtained from flood IR images or depth map images. It should be understood that it may refer to any one of the templates obtained from them. In some embodiments, a combination of flood IR images and depth map images may be used to create templates. For example, pairs of feature vectors obtained from flood IR images and depth map images may be stored in static templates 216 for use in one or more facial recognition processes on device 100.

6 shows a representation of an embodiment of a template space 220 for a registration profile in memory 106 of device 100. In certain embodiments, template space 220 is located in a portion of memory 106 of device 100 protected by SEP 112. In some embodiments, the template space 220 includes a static portion 222 and a dynamic portion 224. Static templates 216 may be added to the static portion 222 of the template space 220, for example (e.g., these templates are permanently added to the memory and deleted or deleted unless the device is reset). Does not change). In some embodiments, static portion 222 includes a predetermined number of static templates 216. For example, in the case of the embodiment of the template space 220 shown in FIG. 6, six static templates 216 are allowed in the static portion 222. In some embodiments, nine static templates 216 may be allowed within the static portion 222. Other numbers of static templates 216 in static portion 222 may also be considered. After the registration process for the registration profile is completed and the static templates 216 are added to the static portion 222, additional dynamic templates 226 are added to the dynamic portion of the template space 220 for the registration profile ( 224) (eg, a portion where templates can be added and deleted without the need for a device reset).

Thus, the static templates 216 may be registration templates (or reference templates) created by the registration process 200 for a registration profile associated with the registration process. After the registration process 200 is complete, a selected number of static templates 216 are stored in the static portion 222 of the template space 220 for the registration profile. The number of static templates 216 stored in the static portion 222 after the registration process 200 is determined by the number of images selected as suitable for use as template images, for example, or the desired number of templates for the device. It may depend on the number of different feature vectors acquired during the registration process, which may be based on the number. After registration process 200, static templates 216 include feature vectors 210 (eg, registration or reference feature vectors) that can be used for facial recognition of an authorized user associated with the registration profile. Thus, the template space 22020 can be used in the face recognition authentication process to authorize a user associated with the registration profile.

In embodiments where there are multiple enrollment profiles on device 100, enrollment process 200 includes separate (distinguished) and independent template spaces (e.g., separate static templates) for each enrollment profile. It can be operated separately for each of the registration profiles to create a. 7 shows a flowchart of an embodiment of registering multiple registration profiles on a device. The registration process 200A may be operated to create a first registration profile on the device 100. The first registration profile may be associated with an authorized user of the device 100 (eg, a primary authorized user of the device). The registration process 200A may create a template space 220A for the first registration profile.

After the template space 220A is created, the authorized user may choose to generate a second registration profile at 230. Subsequently, the registration process 200B may be operated to create a second registration profile on the device 100. As described herein, a second registration profile may be created for a second distinct appearance of an authorized (primary) user or for an additional (eg, secondary) user of device 100. The registration process 200B may create a template space 220B for the second registration profile. The registration process 200A and the registration process 200B may operate according to the registration process 200 shown in FIG. 4.

8 shows a representation of an embodiment of two template spaces 220A, 220B in memory 106 of device 100. The template space 220A may be associated with a first registration profile on the device 100, and the template space 220B may be associated with a second registration profile on the device 100. Although two template spaces (for two registration profiles) are shown in Figures 7 and 8, any number of template spaces (and any corresponding number of registration profiles) is the device's available memory and processing power It should be understood that it can be utilized on device 100 as desired based on the desired operation of the device.

As shown in FIG. 8, separate template spaces 220A and 220B may be located in distinct, separate spaces within memory 106 of device 100. For example, the template spaces 220 may be stored in divided portions of a memory space used for templates in the memory 106. In certain embodiments, the template spaces 220A, 220B are within a portion of the memory 106 of the device 100 that is protected by the SEP 112. The template spaces 220A, 220B may include their own distinct and distinct static portions 222A, 224A, each having separate static templates 216A, 216B. The template spaces 220A, 220B may also include their own distinct and distinct dynamic portions 224A, 224B, each having separate dynamic templates 226A, 226B. Template spaces 220A, 220B may be accessed individually (eg, independently) and/or operated by processor 104 on device 100. For example, the template spaces 220A, 220B may be separately accessed by a face recognition authentication process, and may be individually operated by a template update process as described herein.

In multiple registration profiles stored on device 100 (e.g., multiple template spaces 220 on the device), all registration profiles (and all templates in template spaces) are from the device when the device is reset. Deleted or removed. When a request is made on the device to delete only at least one registration profile, all registration profiles may also be deleted from the device 100. All registration profiles may be deleted to prevent accidentally allowing a user associated with the registration profile for which any remaining registration profiles have been requested to be deleted to access device 100. For example, there may be overlap between registration profiles, which may allow a first user to access device 100 with a registration profile for a second user.

9 shows a flowchart of an embodiment of a face recognition authentication process 250. Process 250 may be used to authenticate a user as an authorized user of device 100 using the user's face recognition. In certain embodiments, process 250 is used to authenticate a user using a single registration profile (eg, single template space 220) on device 100. If multiple enrollment profiles are on device 100, one or more portions of process 250 may be operated in parallel (as described herein) to authenticate a user attempting to access and use the device. . Authentication of the authorized user allows the user to access and use the device 100 (e.g., unlock the device) and/or unlock selected functions of the device (e.g., the functionality of applications running on the device). , Payment systems (i.e., paying), access to personal data, an expanded view of notifications, etc.). In certain embodiments, process 250 is used as a primary biometric authentication process for device 100 (after registration of an authorized user). In some embodiments, process 250 is used as an authentication process in addition to other authentication processes (eg, fingerprint authentication, other biometric authentication, passcode input, password input, and/or pattern input). In some embodiments, another authentication process (e.g., passcode input, pattern input, other biometric authentication) may be used to access device 100 if the user fails to be authenticated using process 250. have.

At 252, camera 102 captures an image of the user's face attempting to authenticate for access to device 100 (eg, the camera captures an image of the user's “attempt to unlock”). The unlock attempt image may be a single image of the user's face (e.g., a single flood IR image or a single depth map image), or the unlock attempt image is taken over a short period of time (e.g., 1 second or less). It should be understood that there can be several images in a series of faces. In some embodiments, the series of several images of the user's face include pairs of flood IR images and depth map images (eg, pairs of successive flood IR images and depth map images). In some implementations, the unlock attempt image may be a composite of several images of a user illuminated by a flood illuminator and a speckle pattern illuminator.

Camera 102 may capture an unlock attempt image in response to a prompt by the user. For example, the unlock attempt image is obtained by moving the device to a selected position relative to the user's face by the user pressing a button (e.g., a home button or a virtual button) on the device 100 (e.g., the camera The device by the user moving the device or lifting the device from the table so that it is oriented towards the face), and/or by making a specific gesture or movement on the device (e.g., by tapping the display or holding the device) It can be captured when trying to access 100. It should be understood that, as described herein, the unlock attempt images may include either flood IR images or depth map images, or a combination thereof. In addition, the unlock attempt images can be processed independently or in combination as needed in association with their corresponding template (eg, flood IR images in association with a template for flood IR registration images).

At 254, the unlock attempt image is encoded to define the user's facial features as one or more feature vectors in the feature space. In some embodiments, one feature vector is defined for the unlock attempt image. In some embodiments, multiple feature vectors are defined for the unlock attempt image. The unlock feature vector(s) 256 may be the output of the encoding of the unlock attempt image at 254.

In certain embodiments, at 258, the unlock feature vector(s) 256 are compared to feature vectors in the templates of the template space 220 to obtain a matching score 260 for the unlock attempt image. . In certain embodiments, template space 220 is a template space for a single registration profile on device 100. The matching score 260 is the feature vector(s) 256 and the feature vectors in the template space 220 (e.g., static templates 216 added to the template space, as described herein, and/or It may be a score of differences between feature vectors in different dynamic templates 226). The closer the feature vector(s) 256 and the feature vectors in the template space 220 are (eg, the smaller the distance or the smaller the differences), the higher the matching score 260 may be. For example, as shown in Figure 5, the feature vector 256A (open diamond) is closer to the feature vectors 210 than the feature vector 256B (open diamond) (e.g., the feature vector 256B is a feature It is an outlier that is farther than vector 256A). Thus, feature vector 256A will have a higher matching score than feature vector 256B. Since feature vector 256B is farther away from feature vectors 210 than feature vector 256A, the lower matching score for feature vector 256B is in the unlock attempt image associated with feature vector 256B. It means that the reliability that the face is the face of the authorized user associated with the registration profile and template space 220 is lower.

In some embodiments, comparing the feature vector(s) 256 with the templates from the template space 220 to obtain a matching score 260 uses one or more classifiers or classification-enabled networks. Thus, it includes classifying and evaluating differences between the features vector(s) 256 and the templates from the template space 220. Examples of different classifiers that may be used include, but are not limited to, linear, segmented linear, nonlinear classifiers, support vector machines, and neural network classifiers. In some embodiments, the matching score 260 is evaluated using the feature vector(s) 256 and distance scores between the templates from the template space 220.

At 262, the matching score 260 is compared to an unlock threshold 264 for device 100. The unlocking threshold 264 is between the face of the authorized user requested by the device 100 to unlock the device (or unlock a feature on the device) and the face of the user in the unlock attempt image (feature It may represent a minimum difference (eg, distance in feature space) of features (as defined by vectors). For example, the unlocking threshold 264 may include templates associated with the face of the user to which the unlocking feature vectors (e.g., feature vectors 256) have been applied (e.g., static templates in the template space 220) 216)) and sufficiently similar (eg, close enough). As a further example, the unlock threshold 264 may be represented by a circle 265 in the feature space 212, shown in FIG. 5. As shown in FIG. 5, feature vector 256A is inside circle 265 and thus feature vector 256A will have a matching score 260 above the unlock threshold 264. However, the feature vector 256B is outside the circle 265, so the feature vector 256B will have a matching score 260 less than the unlock threshold 264. In certain embodiments, the unlock threshold 264 is set during manufacture and/or by the firmware of the device 100. In some embodiments, the unlock threshold 264 is updated (eg, adjusted) by the device 100 during operation of the device as described herein.

As shown in FIG. 9, at 262, if the matching score 260 exceeds the unlock threshold 264 (that is, when the user's face in the unlock attempt image substantially matches the face of the authorized user), The user in the unlock attempt image is authenticated as an authorized user for the registration profile on the device 100 and the device is unlocked at 266. In certain embodiments, after the device 100 is unlocked at 266, the unlocking feature vectors 256 and the matching score 260 may add or replace templates in the template space 220. It is provided to a first template update process 300, shown at 10. At 262, if the matching score 260 is less than the unlock threshold 264 (eg, not above the unlock threshold), then at 268 the device 100 is not unlocked (eg, the device remains locked). Note that if the matching score 260 is equal to the unlock threshold 264 then the device 100 can be locked or unlocked depending on the desired setting for the unlock threshold (e.g., tighter or looser restrictions). Should be. Additionally, either option for comparing the same match score can also be applied as desired for other embodiments described herein.

In certain embodiments, at 270 unlock attempts are compared to a threshold. The threshold may be, for example, the maximum number of unlock attempts allowed or the maximum allotted time for unlock attempts. In certain embodiments, the number of unlock attempts is counted (e.g., the number of attempts to unlock the device 100 using a different unlock attempt image captured at 252) and the maximum number of unlock attempts allowed. Compared to

In certain embodiments, when unlocking attempts reach a threshold (e.g., when the number of unlocking attempts reaches the maximum number of allowed attempts), the device 100 further attempts to use facial authentication at 272. Locked from the field. In some embodiments, when the device is locked at 272, an error message indicating that the facial recognition authentication process 250 has failed and/or is restricted or prevented from performing the desired operation of the device 100 is displayed (e.g., On the display 108). Device 100 may be locked at 272 from further attempts to use facial authentication for a specified period of time and/or until another authentication protocol is used to unlock the device. For example, unlock options 274 may be used to unlock device 100.

Unlock options 274 may include the user being presented with one or more options for proceeding with different types of authentication to unlock or access features on device 100 (e.g. You are presented with options to proceed with the protocol). Presenting the options is, for example, one of the displayed options to proceed with displaying one or more options on the display 108 of the device 100 and unlocking the device or accessing features on the device. It may include prompting the user through audible and/or visual communication to select one. The user may then proceed to unlock/access the device 100 using the selected option and according to additional audible and/or visual prompts as needed. After being successfully authenticated using the selected option, the user's initial request to unlock/access device 100 may be approved. Unlocking options 274 may include, but are not limited to, using a passcode, password, pattern input, different form of biometric authentication, or other authentication protocol to unlock device 100. . In some embodiments, the unlock options 274, when selected, are a "passcode/password/pattern use" affordance, or a passcode/password/pattern input user interface that, when selected, causes display of the passcode/password/pattern input user interface. A pattern input user interface, or, when displayed, a “fingerprint use” prompt that prompts the user to place a finger on the device's fingerprint sensor. In some embodiments, after the device 100 is unlocked using the unlock options at 274, the unlocking feature vectors 256 and the matching score 260 are compared to the second template, shown in FIG. It is provided to the update process 400.

If the unlock attempts at 270 are below the threshold (e.g., the number of unlock attempts is less than the maximum number of allowed attempts), the process 250 starts with the new unlock attempt image of the user being captured at 252 and runs again May be (restarted). In some implementations, the device 100 automatically captures a new unlock attempt image of the user's face without prompting the user (eg, capture of the new image is automatically implemented and/or hidden from the user). In some implementations, device 100 notifies (visually and/or audibly) the user that process 250 is being resumed. In some embodiments, device 100 may prompt the user to provide an input to resume process 250. For example, the user may be prompted to acknowledge (visually and/or audibly) an attempt to resume process 250 or otherwise confirm.

10 shows a flowchart of an embodiment of a first template update process 300. Process 300 may be used to update template space 220 (shown in FIG. 6) with one or more additional dynamic templates 226 based on feature vector(s) 256 from process 250. . In certain embodiments, the process 300 is used to update the template space 220 for a single registration profile on the device 100 (eg, the registration profile used in process 250 ). If multiple registration profiles are on device 100, process 300 may be operated separately for each registration profile to independently update different registration profiles on the device (as described herein. ). The process 300 may be used to update the template space 220 for incremental changes in the appearance of the authorized user associated with the registration profile. For example, the process 300 may include gradual changes in hair (e.g., hair color, hair length, and/or hairstyle), weight gain, weight loss, changes in glasses worn, or minor cosmetic damage. Template space 220 may be updated for changes (eg, black eyes, scars, etc.). Updating the template space 220 using the process 300 allows an authorized user to continue to successfully access the device 100 using the facial recognition authentication process 250, despite a gradual change in the user's appearance. Allow to do.

Process 300 may begin with evaluating 302 whether the matching score 260 is above a threshold 304. The threshold 304 indicates that the feature vector(s) 256 is sufficient with the feature vectors 210 (from the static templates 216) such that the feature vector(s) 256 can potentially be used as another template. It may be a threshold score to determine whether it is similar (eg, close) (eg, the threshold score may determine whether the feature vectors 256 are within a predetermined distance from the feature vectors 210 ). In certain embodiments, threshold 304 is greater than (exceeds) unlock threshold 264 (eg, threshold 304 requires a match score higher than unlock threshold 264). Thus, the threshold for the feature vector(s) 256 to become a template may be stricter than the threshold for unlocking the device. The threshold 304 may be set during manufacture and/or by the firmware of the device 100. Threshold 304 may be updated (eg, adjusted) by device 100 during operation of the device as described herein.

In some embodiments, if the match score 260 is below the threshold 304, the process 300 stops and the feature vector(s) 256 is deleted from the device 100. In some embodiments, if the matching score 260 is less than the threshold 304, the process 300 continues with the template update subprocess 301, described in FIG. 12. If the matching score 260 exceeds the threshold 304, the process 300 continues. In some embodiments, after evaluation 302, at 306 one or more qualities in the unlock attempt image are evaluated. For example, at 306 the pose (eg, pitch, yaw, and roll of the face), occlusion, attention, vision, and/or distance in the unlock attempt image may be evaluated. The pose and/or occlusion in the unlock attempt image can be evaluated using the landmark and/or occlusion maps described herein. At 308, if the appropriate qualifications are not met, the process 300 may stop. In certain embodiments, meeting suitable qualifications includes meeting selected criteria in the image for one or more of the assessed qualities described above. For example, the selected criteria are that the user's face is in the field of view of the camera, that the user's face pose is appropriate (e.g., that the user's face is not turned too far in any direction from the camera (i.e. The pitch, yaw, and/or roll of is not exceeding certain levels)), the distance to the user's face is within a certain distance, the user's face has an occlusion less than a minimum value (e.g., the user's The face is not occluded (blocked) by other objects by more than a minimum amount), the user pays attention to the camera (e.g., the user's eyes are looking at the camera), the user's eyes are not closed, And appropriate lighting (illumination) in the image, but is not limited to these. In some embodiments, evaluating qualities at 306 and 308 may occur at different locations within process 300. For example, evaluating the qualities at 306 and 308 can be done after comparing the matching score 324 to a threshold 326 or comparing the confidence score 332 to the confidence score 334 at 336, described below. It can happen later.

If the appropriate qualifications are met at 308, the process 300 continues at 310 with storing the feature vector(s) 256 in a backup space in the memory of the device 100. The backup space in the memory may be, for example, a second space or a temporary space in a memory including readable/writable memory and/or short-term memory. Feature vector(s) 256 may be stored in memory as temporary template 312.

In certain embodiments, after the temporary template 312 is stored in a backup space in memory, the process 300 applies the temporary template to additional unlock attempt images captured by the device 100 for an authorized user. It continues by comparing feature vectors. At 314, additional unlock attempt images of the user (or users if unauthorized access is attempted), the user(s) during additional (future) unlock attempts of the device 100 are captured. At 316, features of the user's face in the additional unlock attempt images are encoded to generate feature vectors 318. At 320, feature vectors 318 are compared to temporary template 312 to obtain a matching score 322.

At 324 the matching score 322 may then be compared to a threshold 326. In some embodiments, threshold 326 is an unlock threshold 264. In some embodiments, threshold 326 is threshold 304. If the matching score 322 at 324 exceeds the threshold 326, then at 328 a successful attempt is counted. If the matching score 322 at 324 is less than the threshold 326, then at 330 an unsuccessful attempt is counted. Countings 328 and 330 may continue until a desired number of unlock attempts (eg, matching score 322 and a desired number of comparisons of threshold 326) have been made. Once the desired number of attempts have been made, the number of successful attempts at 328 out of the total number of unlock attempts (e.g., sum of counts 328 and 330) gives the confidence score 332 for the temporary template 312. Can be used to evaluate. For example, of the 50 total unlock attempts, there may be 45 successful attempts, so the confidence score 332 is 45/50 or 90%. Confidence score 332 may be used to evaluate whether template 312 is added to template space 220, shown in FIG. 6, as dynamic template 226.

As described above, after initial registration, registration templates (eg, static templates 216, shown in FIG. 6) are added to the static portion 222 of the template space 220. After the registration process and static templates 216 have been added to the static portion 222, the process 300 shown in FIG. 10 may be used to add additional templates to the template space 220. Additional templates may be added as dynamic templates 226 to dynamic portion 224 (eg, a portion in which templates may be added and deleted without the need for a device reset). Dynamic templates 226 may be used in combination with static templates 216 in template space 220 for facial recognition authentication process 250, shown in FIG. 9.

In certain embodiments, when the confidence score 332 for the temporary template 312 is higher than the lowest confidence score of the static templates 216 in the static portion 222, the process 300, shown in FIG. Temporary templates 312 created by are added to the dynamic portion 224 as dynamic templates 226 shown in FIG. 6. The confidence score 334 is the lowest confidence score for the static templates 216 in the static portion 222 evaluated during the same unlock attempts used to evaluate the confidence score 332 for the temporary template 312. (E.g., the confidence score for the template with the lowest number of successful unlock attempts during the same unlock attempts using the temporary template 312). Confidence score 334 may be evaluated using the same threshold (eg, threshold 326) used for confidence score 332.

In certain embodiments, if, at 336, the confidence score 332 is greater than the confidence score 334, then at 338, the temporary template 312 is added to the dynamic portion 224 as the dynamic template 226. For example, temporary template 312 has 45 successful unlock attempts out of 50 total unlock attempts, while only one static template 216 makes 40 successful unlock attempts out of the same 50 total unlock attempts. If so, the temporary template 312 can be added to the dynamic portion 224 as one of the dynamic templates 226. At 336, if the confidence score 332 is less than the confidence score 334, the temporary template 312 is ignored or deleted at 340. Temporary templates 312 may be added until the maximum number of allowed dynamic templates 226 are stored in the dynamic portion 224.

Once the dynamic portion 224 reaches its maximum number of dynamic templates 226 in the dynamic portion 224, the temporary template 312 may replace one of the dynamic templates 226 at 338. For example, if the temporary template is less outlier than one of the dynamic templates 226, then the temporary template 312 may replace one of the dynamic templates 226. In certain embodiments, statistical analysis of the feature vectors representing the dynamic templates 226 and the temporary template 312 is used to assess whether the temporary template 312 is less outlier than one of the dynamic templates 226. do. Statistical analysis can include, for example, classification algorithms operated on feature vectors for templates.

11 shows a representation of an embodiment of a template space 220 expressed as a feature space. In the feature space description of template space 220, static templates 216, dynamic templates 226, and temporary template 312 are represented by feature vectors. For example, static templates 216 are represented by circles, dynamic templates 226 are represented by diamonds, and temporary template 312 is represented by stars. In certain embodiments, as described above, static templates 216 are not allowed to be replaced by temporary template 312. Thus, if the dynamic portion 224 has reached its maximum number of dynamic templates 226, the temporary template 312 is less outlier than one of the dynamic templates 226, the temporary template 312 is the dynamic templates ( 226) can be substituted.

Statistical analysis of feature vectors in the feature space correlated to the template space 220 may generate a circle (eg, a circle 342) that defines the maximum number of feature vectors closest. As shown in FIG. 11, circle 342 defines a feature vector for dynamic template 226' as an outlier of the circle. The feature vector for the dynamic template 226' is more outlier than the feature vector for the temporary template 312. Thus, the temporary template 312 may replace the dynamic template 226' in the template space 220. If the temporary template 312 was outlier than each of the dynamic templates 226, the temporary template may not have replaced any of the dynamic templates 226.

In certain embodiments, when temporary template 312 replaces dynamic template 226' in template space 220, one or more thresholds for device 100 may be recalculated. When the temporary template 312 is less outlier than the dynamic template 226', recalculation of the threshold(s) may further constrain the thresholds (e.g., the threshold for matching scores to require a closer match). Elevated). In some embodiments, the unlock threshold (e.g., unlock threshold 264, shown in Figure 9) is greater when the temporary template 312 replaces the dynamic template 226' in the template space 220. It becomes strict. In some embodiments, the template update threshold (e.g., threshold 304, shown in FIG. 10) is more stringent when the temporary template 312 replaces the dynamic template 226' in the template space 220. do.

12 shows a flowchart of an embodiment of the template update sub-process 301. As described above, if the matching score 260 is less than the threshold 304 but exceeds the unlock threshold 264, the sub-process 301 may proceed. Images with matching scores 260 within that range (above unlock threshold 264 and below threshold 304) will match more than images above threshold 304 (while still being able to unlock device 100). Can have more uncertainty in doing things Thus, these more ambiguous images can be processed using the subprocess 301.

In sub-process 301, at 350, one or more qualities in the unlock attempt image are evaluated. Evaluating the qualities of the unlock attempt image at 350 may be substantially similar to evaluating the qualities at 306 and 308, as shown in FIG. 10. As shown in Fig. 12, if the unlock attempt image passes the evaluation of the qualities at 350 (eg, meets the qualifications), another temporary template is stored in the backup space used for the temporary templates 312 at 352. A determination of whether there is space (eg, room) (eg, determination of whether the maximum number of temporary templates 312 are stored in the backup space) may be made.

If there is no place in the backup space ("N"), the unlock attempt image (and its corresponding feature vectors) may go through a policy deletion 354 as shown in FIG. 12. In policy deletion 354, feature vector(s) in a backup space (e.g., space for temporary template 312) with selected redundancy for existing features (e.g., the most redundant) is Can be replaced in space.

If there is room in the backup space ("Y"), at 356, feature vectors for the unlock attempt image are added to the backup space as a temporary template (eg, temporary template 312). Once the temporary template from the subprocess 301 is added to the backup space at 356, the temporary template is substantially temporary template 312 (e.g., compared to additional unlock attempt images as shown in FIG. 10). Can be processed as. In certain embodiments, a temporary template from subprocess 301 is used as a template (eg, temporary template 312 and/or dynamic template 226) for a selected amount of time. For example, because the temporary template from subprocess 301 is originally added with higher uncertainty than other templates, the amount of time allowed for use of the temporary template from subprocess 301 may be limited (e.g. , Temporary templates have a limited lifetime). In some embodiments, the amount of time selected is the maximum amount of successful unlock attempts using the temporary template from subprocess 301.

As described above, the first template update process 300 updates the registration profile (eg, templates in the template space) when the device 100 is unlocked or accessed using the face recognition authentication process 250 Can be used to The first template update process 300 may be used, for example, to update the registration profile in response to gradual changes in the user's appearance (eg, weight gain/loss).

However, in some embodiments, the facial features of the authorized user (e.g., the user's facial appearance) may change rapidly, or at least to a sufficiently large extent, so that the user can perform the facial recognition authentication process 250 shown in FIG. May have difficulty unlocking or accessing features (eg, operations) on device 100 using. Sudden or significant changes in the appearance of the user's face include, for example, shaving a beard or mustache, obtaining a large scar or other cosmetic damage to the face, making sudden makeup changes, giving sudden hair changes. can do. In some cases, if there was an error during the enrollment process and/or there are significant differences between the enrollment time and the user's environment during the unlock attempt, the user may also use the facial recognition authentication process 250 to lock the device 100. You may have difficulty unlocking/accessing. Having difficulty in unlocking the device 100 using the facial recognition authentication process 250 may be a frustrating experience for the user. When the difficulty in unlocking the device 100 using the face recognition authentication process 250 arises due to the changes/issues described above, the second template update process (e.g., the second Template update process 400), at least temporarily, after verification of the user's identity using the second authentication protocol, in spite of issues/changes, the user unlocks the device using the face recognition authentication process/ It can be used to make it accessible.

As shown in Fig. 9, the user is in the face recognition authentication process until the number of unsuccessful unlocking attempts at 270 reaches a selected value and the device 100 is locked from further attempts using the face recognition authentication process. (250) can be used to unsuccessfully attempt multiple unlock attempts. At such times, the user may be presented with one or more options for proceeding with different types of authentication to unlock or access features on device 100 in unlock options 274. After the user has been successfully authenticated using the selected option, the user will not be able to use the facial recognition authentication process to unlock/access the device, despite changes in the user's facial appearance that previously prevented the user from using the facial recognition authentication process. To enable unlocking/accessing the device in future unlock attempts using 250, the device 100 may be configured with the user (e.g., using the second template update process 400 described below). You can update your registration profile, at least temporarily. Thus, the user can automatically access the device 100 using the face recognition authentication process 250 in future unlock attempts for at least a short period of time, by successfully completing authentication using the selected option. .

13 shows a flowchart of an embodiment of a second template update process 400. 9, the facial recognition authentication process 250 cannot unlock the device 100, but the process 400 may be used when the device is unlocked using a passcode or other authentication protocol. In some embodiments, the device 100 may be unlocked using the passcode immediately after the unlock attempt fails or within a specified time frame after the unlock attempt fails (e.g., in time proximity to the unlock attempt) When the process 400 can be used. In certain embodiments, process 400 is used to update template space 220 for a single registration profile on device 100 (eg, a registration profile used in process 250 ). If multiple registration profiles are on device 100, process 400 will only operate on the registration profile that most closely matches feature vectors from the captured unlock attempt image (as described herein). I can.

In certain embodiments, feature vectors (e.g., feature vectors) generated from the unlock attempt image to enable device 100 to be unlocked using the facial recognition authentication process 250, shown in FIG. Enrollment profile when facial features of an authorized user have changed to the extent that prevents them from being sufficiently close (e.g., within an unlock threshold distance) to static templates 216 and/or dynamic templates 226 Process 400 is used to update the template space 220 for. For example, process 400 may be used for feature vector 256B shown outside circle 265 (unlock threshold circle) in FIG. 5. Possible causes for the user being able to unlock the device 100 using the facial recognition authentication process 250 are: an authorized user shaving a beard or mustache, or getting a large scar or other cosmetic damage to the face. Or, with large makeup changes, sudden hair changes, or other serious changes in facial features, these changes allow the first template update process 300 to gradually update the template space 220 over time. Including, but not limited to, instantaneous changes or "step changes" of the facial features of an authorized user.

The second template update process 400 may begin at 402 by evaluating whether the matching score 260 exceeds a threshold 404. Threshold 404 indicates that the feature vector(s) 256 is sufficient with the feature vectors 210 (from static templates 216) such that the feature vector(s) 256 can potentially be used as another template. It may be a threshold score to determine if it is similar (eg, close). In certain embodiments, the threshold 404 for the process 400 is less than the unlock threshold 264. Threshold 404 may be less than unlock threshold 264 because passcode (or other authentication) was entered before starting process 400 (e.g., more distances are allowed between feature vectors and templates). . Thus, the threshold for feature vector(s) 256 to become a template in process 400 may be less stringent than the threshold for unlocking the device and the threshold for process 300 shown in FIG. 10. However, the threshold 404 may be set to a value that sets a maximum allowable distance between the feature vectors 256 for the unlock attempt image and the feature vectors for the template space 220. Setting the maximum allowable distance can be used to prevent a user who is not an authorized user but has a passcode for the device 100 from being able to perform facial recognition authentication on the device. The threshold 404 may be set during manufacture and/or by the firmware of the device 100. Threshold 404 may be updated (eg, adjusted) by device 100 during operation of the device (eg, after templates are added or replaced in template space 220) as described herein.

Process 404 may be stopped and feature vector(s) 256 are deleted from device 100 if match score 260 is less than threshold 404. If the match score 260 exceeds the threshold 404, the process 400 continues. In some embodiments, after evaluation 404, at 406 one or more qualities in the unlock attempt image are evaluated. For example, at 406 a pose (eg, pitch, yaw, and roll of the face), occlusion, attention, field of view, and/or distance in the unlock attempt image may be evaluated. In some embodiments, the pose and/or occlusion in the unlock attempt image may be evaluated using the landmark and/or occlusion maps described herein. At 408, if the appropriate qualifications (as described above) are not met, the process 400 can be stopped.

If the appropriate qualifications are met at 408, the process 400 continues at 410 with storing the feature vector(s) 256 in a backup space in the memory of the device 100. The backup space in memory for process 400 may be a different backup space than that used for process 300. For example, the backup space in memory for process 400 may be a temporary space in memory including readable/writable memory partitioned from the backup space used for process 300. Feature vector(s) 256 may be stored in memory as temporary template 412.

In certain embodiments, after temporary template 412 is stored in the backup space, temporary template 412 is compared with feature vectors for additional images from failed facial recognition authentication unlock attempts of device 100 Can be. For example, in process 400, additional failed unlock attempt images at 414 may be captured. If the correct passcode is input at 416, feature vectors for images captured at 414 are encoded at 418 to generate feature vectors 420.

In certain embodiments, at 422, feature vectors 420 are compared to feature vector(s) for temporary template 412. Comparison of the feature vectors 420 and the feature vector(s) for the temporary template 412 may provide a matching score 424. At 426 a matching score 424 may be compared to a threshold 428. Threshold 428 is obtained, for example, from additional images from failed facial recognition authentication attempts followed by the entry of the feature vector(s) for the temporary template 412 and the passcode for the device 100 It may be a threshold or similarity threshold that defines at least a minimum matching level between the feature vectors 420 to be formed. Thus, the threshold 428 caused a failed unlock attempt and the change in the characteristics of the user who created the temporary template 412 is at least the minimum that still exists in images from additional failed unlock attempts using facial recognition authentication. It can be set to a value that guarantees a positive probability.

If the match score 424 at 426 exceeds the threshold 428, a successful match is counted at 430. If the match score 424 at 426 is less than the threshold 428, then the unsuccessful match at 432 is counted. Countings 430 and 432 may continue until a desired number of failed unlock attempts using facial recognition authentication (e.g., matching score 424 and desired number of comparisons of threshold 428) have been made. . Once the desired number of attempts have been made, the number of successful matches at 430 out of the total number of failed unlock attempts (e.g., the sum of counts 430 and 432) is the confidence score 434 for the temporary template 412. Can be used to evaluate For example, there may be 18 successful matches (eg, comparison) of the match score 424 and the threshold 428 out of a total of 20 failed unlock attempts. Confidence score 434 may be used to evaluate whether template 412 is added as dynamic template 226 to template space 220 for enrollment profile, shown in FIG. 6.

In some embodiments, it may be assumed that if a step change occurs in the facial features of an authorized user, the step change may remain for multiple successive unlock attempts using facial recognition authentication. For example, if the user has shaved his beard, the step change will remain for at least some length of time (eg, at least a week). In such embodiments, successful unlock attempts (or desired number of successful unlock attempts) using facial recognition authentication before reaching a selected number of successive unlock attempts (e.g., 10 or 15 unlock attempts). S) occurs, the temporary template 412 may be deleted from the backup space in memory. In some embodiments, the assumption that a step change may remain for multiple successive unlock attempts (eg, if the user's step change is due to temporary application of makeup).

In certain embodiments, at 436, the confidence score 434 is compared to the threshold 438 to assess whether the confidence score is greater than the threshold. Threshold 438 may be a threshold selected to ensure successful comparisons of the minimum number of times of matching score 424 and threshold 428 before allowing template 412 to be added to template space 220. At 436, if the confidence score 434 is greater than the threshold 438, then at 440, a temporary template 412 may be added to the template space 220, or a temporary template 412 may be added to the template space 220. It may replace a template (eg, replace one of the dynamic templates 226). If the confidence score 434 is less than the threshold 438, the temporary template 412 may be ignored or deleted at 442.

As described above, the temporary template 412 created by the process 400 is one of the dynamic templates 226 shown in FIG. 6 in the dynamic portion 224 of the template space 220 for the registration profile. Can be added. In the case of the process 400 shown in FIG. 13, a passcode (or other authentication) was used to verify that the temporary template 412 is for an authorized user. Thus, in certain embodiments, a temporary template 412 is added to the template space 220 at 440 without the need for comparison with dynamic templates 226 already in the dynamic portion 224. If the maximum number of allowed dynamic templates 226 in the dynamic portion 224 has not been reached, a temporary template 412 is added to the dynamic portion as one of the dynamic templates 226.

Once the maximum number of allowed dynamic templates 226 in the dynamic portion 224 has been reached, the temporary template 412 may replace one of the dynamic templates 226 in the dynamic portion. Since the passcode (or other authentication) was used to verify that the temporary template 412 is for an authorized user, the temporary template is not in the dynamic part 224 even though the temporary template is outlier than each of the dynamic templates 226. One of the dynamic templates 226 of may be substituted. In certain embodiments, temporary template 412 replaces the largest outlier of dynamic templates 226 regardless of the relative lie (eg, outlie) of the temporary template. In some embodiments, even if the temporary template is outlier than each of the dynamic templates, the temporary template 412 may replace the redundant (eg, most redundant) dynamic template for existing dynamic templates.

14 shows a representation of an embodiment of a template space 220 expressed as a feature space with feature vectors for the temporary template 412. In the feature space description of template space 220 in FIG. 14, static templates 216, dynamic templates 226, and temporary template 412 are represented by feature vectors. Static templates 216 are represented by circles, dynamic templates 226 are represented by diamonds, and temporary template 412 is represented by stars. As described above, static templates 216 may not be replaced by temporary template 412. Thus, if the dynamic portion 224 has reached its maximum number of dynamic templates 226, the temporary template 412 may replace one of the dynamic templates 226.

Statistical analysis of feature vectors in the feature space correlated to the template space 220 may generate a circle (eg, circle 444) that defines the maximum number of feature vectors closest. As shown in FIG. 14, the feature vector for dynamic template 226 ′ is the largest outlier of each of the feature vectors for dynamic templates 226. Accordingly, the temporary template 412 can replace the dynamic template 226' in the template space 220 regardless of the position of the feature vector for the temporary template. In the example shown in Fig. 14, the addition of a feature vector to the temporary template 412 shifts the circle 444 towards the feature vector for the temporary template 412, and the feature vector for the dynamic template 226' is It can be the biggest outlier of the circle. In some embodiments, when temporary template 412 replaces dynamic template 226' in template space 220, one or more thresholds for device 100 may be recalculated.

In some embodiments, a temporary template (e.g., a temporary Either template 312 or temporary template 412) may be used. The temporary template may be used to unlock the device 100 after the passcode (or other user authentication protocol) is used in combination with the temporary template. For example, in the case of the temporary template 412, a passcode is input to unlock the device 100 before the temporary template 412 is created and stored in a backup space of the device memory. Temporary template 412 may then be used to enable unlocking of device 100 using facial recognition authentication for a selected period of time (eg, a few days or a week). After the selected time period has expired, if the temporary template 412 has not been added to the template space 220, the user may be prompted for a passcode if the user's face recognition authentication fails.

As described above, in certain embodiments, multiple registration profiles are created on device 100. For example, multiple registration profiles may be created to register multiple users on device 100 and/or to register multiple features (eg, appearances) for a single user. Multiple users may be registered on device 100 by using the registration process 200 to create registration profiles for additional users of the device. For example, a primary user (e.g., a parent) may use the enrollment process 200 by a secondary user (e.g. It can be allowed to create a second registration profile on (100).

Multiple appearances for a single user are substantially different and may include appearances that cannot be recognized using a single registration profile. For example, a single user may consider appearances distinguished by wearing glasses versus wearing glasses, having a shaved face versus having a bearded face, and a face with dark makeup versus a face without makeup. Can have and/or have different drastic changes at different times of the week/day. For distinct appearances, a single user runs the registration process 200 for the first time to create a first registration profile for the first appearance (e.g., while wearing makeup), and runs the registration process a second time. It is possible to create a second registration profile for the second appearance (eg, while not wearing makeup). The user can access the device using facial recognition that authenticates the user using either the first registration profile or the second registration profile.

In embodiments with multiple registration profiles, the image registration process 200 is used to create each registration profile (e.g., first and second registration profiles) as a separate registration profile on the device 100. I can. For example, process 200 can be used to create separate template spaces (eg, template spaces 220A, 220B shown in FIG. 8) for each registration profile. In the facial recognition authentication process, separate template spaces may be independently compared with features from the unlock attempt image to authenticate the user to the device 100. If the template space for any of the registration profiles is determined to be a close match for features in the unlock attempt image (e.g., when the matching score exceeds the unlock threshold), the device 100 may be unlocked. .

In some embodiments, if multiple enrollment profiles are stored on device 100, the unlock threshold is increased (eg, the requirement for matching features in the unlock attempt image becomes more stringent). When a new registration profile is created, the amount of unlocking threshold that is increased is in the feature space between the feature vectors associated with the templates for the new registration profile and the feature vectors associated with the templates in the existing registration profile(s). It may be based on the distance (eg, the greater the distance between the feature vectors in the template for the new registration profile and the feature vectors in the existing registration profiles, the more the unlock threshold is increased). In some embodiments, the new unlock threshold may also be adjusted based on the match history of existing registration profiles (eg, the more matches in the history of existing profiles, the tighter the threshold may be).

15 shows a flowchart of an embodiment of a facial recognition authentication process 250' for use with multiple registration profiles. 15 shows a process 250' for use with two registration profiles. However, it should be understood that process 250 ′ may operate on any number of registration profiles stored on device 100. Process 250 ′ may begin at 252 with capturing an unlock attempt image of a user's face attempting to be authenticated for access to device 100 with camera 102. At 254, the unlock attempt image is encoded to define the user's facial features as one or more feature vectors in the feature space. In some embodiments, one feature vector is defined for the unlock attempt image. In some embodiments, multiple feature vectors are defined for the unlock attempt image. The unlock feature vector(s) 256 may be the output of the encoding of the unlock attempt image at 254.

At 258A, the unlock feature vector(s) 256 are the templates of the first template space 220A (template space for the first registration profile) to obtain a first matching score 260A for the unlock attempt image. Are compared with the feature vectors within. At 258B, the unlock feature vector(s) 256 are the templates of the second template space 220B (template space for the second registration profile) to obtain a second matching score 260A for the unlock attempt image Are compared with the feature vectors within. The matching scores 260A and 260B may be scores of differences between the feature vectors in the template spaces 220A and 220B and the feature vector(s) 256, respectively. After determining the first matching score 260A and the second matching score 260B for each of the first registration profile and the second registration profile, the first matching score may be compared to the unlock threshold 264 at 262A, The second match score may be compared to an unlock threshold at 262B.

In certain embodiments, as shown in FIG. 15, the unlock threshold 264 is a first matching score 260A (a matching score for a first registration profile associated with a primary authorized user) and a second matching score. (260B) (matching score for the second registration profile) may be the same unlock threshold for both. However, in some embodiments, the unlock threshold may be different for different registration profiles. For example, the unlocking threshold for the second registration profile may be higher than the unlocking threshold for the first registration profile associated with the first authorized user (e.g., matching to the second registration profile is Is more stringent than matching for).

In certain embodiments, the generation of the first matching score 260A at 258A and the generation of the second matching score 260B at 258B unlocks the first matching score 260A and the second matching score 260B. It may operate in parallel with a comparison to threshold 264 (eg, match scores are generated and compared independently and substantially simultaneously with the unlock threshold). In some embodiments, the generation of the first match score 260A and the second match score 260B and comparison with the unlock threshold 264 are operated continuously (e.g., the second match score 260B is generated and A first match score 260A is generated and compared to the unlock threshold before being compared to the unlock threshold).

As shown in FIG. 15, when the first matching score 260A or the second matching score 260B exceeds the unlocking threshold 264 (that is, the user's face in the unlocking attempt image is the first registration profile or If it substantially matches the face of the authorized user for any of the second registration profiles), the user in the unlock attempt image 252 is authenticated as an authorized user for the device 100, and the device is unlocked at 266. do. If both the first match score 260A and the second match score 260B are less than the unlock threshold 264 (e.g., not above the unlock threshold), the device 100 remains locked, and the process 250 ') continues with the locking protocol 450 (shown in FIG. 16).

After the device 100 is unlocked at 266, the unlocking feature vectors 256 and the first matching score 260A and/or the second matching score 260B are matched scores exceeding the unlocking threshold 264. Depending on the template update processes are provided. 15, when the first matching score 260A exceeds the unlocking threshold 264, the device 100 is unlocked at 266, and the first matching score together with the feature vectors 256 It is provided to process 300A. The process 300A may be a template update process for updating the first template space 220A according to the process 300 (shown in FIG. 10 ). Similarly, if the second match score 260B exceeds the unlock threshold 264, the device 100 is unlocked at 266, and the second match score is added to the process 300B along with the feature vectors 256. Is provided. The process 300B may be a template update process for updating the second template space 220B according to the process 300.

Process 300A and process 300B can only be executed if their corresponding match score (eg, first match score 200A or second match score 200B) exceeds unlock threshold 264. If both the first match score 200A and the second match score 200B exceed the unlock threshold 264, then both process 300A and process 300B are assigned corresponding template spaces 220A, 220B. Can be operated to update Processes 300A, 300B may operate in parallel with feature vectors from the same captured image(s) (e.g., feature vectors 256 for unlock attempt image 252) (e.g., processes It works simultaneously and independently).

16 shows a flowchart of an embodiment of a locking protocol 450. As described above, the locking protocol 450 recognizes faces when both the first matching score 260A and the second matching score 260B are less than the unlocking threshold 264 and the device 100 remains locked. It begins with the authentication process 250'. The locking protocol 450 may include some operations from the facial recognition authentication process 250 shown in FIG. 9. For example, as shown in FIG. 16, the locking protocol 450 may start at 268 in a state in which the device is not unlocked. At 270, the number of unlock attempts (eg, the number of attempts to unlock the device 100 using a different unlock attempt image captured at 252) is counted. If the number of unlock attempts at 270 is less than a selected value (e.g., threshold), the process 250' may be run again using another unlock attempt image (e.g., a new image of the user is captured and processed). .

If the number of unlock attempts exceeds the selected value, at 272 the device 100 is locked from further attempts to use facial authentication. In some embodiments, an error message indicating that the facial recognition authentication process 250 ′ has failed and/or is restricted or prevented from performing the desired operation of the device 100 is (e.g., on the display 108). Can be displayed. Device 100 may be locked at 272 from further attempts to use facial authentication for a specified period of time and/or until another authentication protocol is used to unlock the device. In some embodiments, unlock options 274 are used to unlock device 100.

In certain embodiments, after device 100 is unlocked using one of the unlock options at 274, the first matching score 260A and the second matching score 260B may be compared at 452. . When the first matching score 260A exceeds the second matching score 260B, the first registration profile (and the first template space 220A) is the most suitable for the feature vectors 256 of the unlock attempt image 252. It may have close (eg, minimum distance) feature vectors. If the first matching score 260A is less than the second matching score 260B, the second registration profile (and the second template space 220B) is closest to the feature vectors 256 of the unlock attempt image 252 (Eg, minimum distance) feature vectors.

As shown in FIG. 16, when the first matching score 260A exceeds the second matching score 260B, the unlocking feature vectors 256 and the first matching score 260A are in the first template space. For the second template update process 400A. Process 400A may be a template update process for updating the first template space 220A according to process 400. When the first matching score 260A is less than the second matching score 260B, the unlocking feature vectors 256 and the second matching score 260B may be provided to the process 400B. The process 400B may be a template update process for updating the second template space 220B according to the process 400. As shown in Fig. 16, when the locking protocol 450 is implemented, only the template space closest to the feature vectors 256 can be updated. Updating only the template space closest to the feature vectors 256 may prevent clarification between template spaces (and registration profiles) on the device, thereby preventing undesired access to the device between registration profiles.

In some embodiments, the template spaces described herein (e.g., template spaces 220, 220A, and/or 220B) can be applied to multiple orientations or angles of the device 100 relative to the user's face. Is updated to include dynamic templates for (eg, multiple poses of the user's face for the device). For example, a first set of dynamic templates may be created for a first pose of the user's face for the device 100, while a second set of dynamic templates may be generated for a second pose of the user's face for the device. Can be created for In some embodiments, the first pose of the user's face is a normal pose (e.g., the user's face is usually perpendicular to the device with the user's entire face (or most of the face) positioned in front of the camera). Is located). The second pose of the user's face may be, for example, an inverted pose (eg, the user's face is rotated about 180° from the normal pose).

In some embodiments, the user's face may have a low pitch angle relative to the camera because, in the second pose, the camera is flipped relative to the user's face. With a low pitch angle and a user's face upside down, the user's chin may occlude or block some features of the user's face. For example, the chin may block or occlude a portion of the user's eye. The user's nose may also block or occlude parts of the user's eyes in a low pitch angle and inverted pose.

Because the user can often lift or hold the device 100 in any orientation (e.g., the user may not be aware of the orientation of the device or may not have any intention to rotate the device to the proper orientation), the device may pose multiple Being able to unlock using facial authentication with angles (eg, a normal pose and a low pitch (inverted) pose) may provide the user with increased utility of the device. Increasing the usefulness of the device 100 may increase user satisfaction by the operation of face authentication on the device.

In certain embodiments, the template space for a registration profile (eg, template space 220) includes separate sets of dynamic templates for different pose angles. For example, the template space may include a first set of dynamic templates for a first (normal or front) pose and a second set of dynamic templates for a second (inverted or low pitch angle) pose. 17 shows a representation of an embodiment of a template space 220' having two dynamic portions 224 and 224'. The dynamic portion 224 may include a first set of dynamic templates 226 and the dynamic portion 224' may include a second set of dynamic templates 226'. In certain embodiments, dynamic templates 226 are for a first (normal or frontal) pose, and dynamic templates 226' are for a second (inverted or low pitch angle) pose. It will be appreciated that additional dynamic parts and dynamic templates may also be considered. For example, a third dynamic portion and a third set of dynamic templates may be used for the horizontal orientation of the user's face relative to the device 100.

In certain embodiments, dynamic templates 226' are created using template update process 300 and/or template update process 400, which is described herein for generating dynamic templates 226. do. As shown respectively in FIGS. 10 and 13 and described above, the dynamic templates 226 ′ are in “quality evaluation 306” during the template update process 300, or “quality evaluation” within the template update process 400. It may be created separately from the dynamic templates 226 based on the device's pose or orientation determined at 406". Thus, while the dynamic templates 226 may be generated by either the template update process 300 or the template update process 400 when the user's pose is determined to be the first (normal or frontal) pose, The dynamic template 226 ′ may be generated by the template update process 300 or the template update process 400 when the user's pose is determined to be a second (inverted, or low pitch angle) pose.

For example, in "quality evaluation 306", if the template update process 300 determines that the user's pose is the first (normal or frontal) pose, the template update process is a dynamic portion of the template space 220 ( 224), add a dynamic template 226. Alternatively, in "quality assessment 306", if the template update process 300 determines that the user's pose is the second (inverted, or low pitch angle) pose, the template update process Add dynamic template 226' to dynamic portion 224'. Dynamic templates 226, 226' may be added to dynamic portions 224, 224', respectively, until the maximum number of templates in each portion is reached. In certain embodiments, the dynamic portions 224, 224' have the same maximum number of dynamic templates 226, 226'. In some embodiments, the dynamic portions 224, 224' have a different maximum number of dynamic templates 226, 226'.

In implementations using multiple dynamic portions for multiple poses of the user, the facial recognition authentication process evaluates the user's pose (e.g., orientation relative to the device 100) and matches the unlock attempt image. Determine the set of dynamic templates (eg, dynamic portion) that will be used to determine the score. Figure 18 shows a flowchart of an embodiment of a facial recognition authentication process 250" for use with multiple dynamic portions. Figure 18 shows two dynamic portions (eg, dynamic portions 224, 224'). Shows a process 250" for use with. However, it should be understood that process 250" can operate on any number of dynamic portions stored on device 100.

Process 250" may begin with camera 102 capturing an unlock attempt image of a user's face attempting to be authenticated for access to device 100 at 252. At 254, an unlock attempt image Is encoded to define the user's facial features as one or more feature vectors in the feature space In some embodiments, one feature vector is defined for the unlock attempt image In some embodiments, the unlock attempt image A number of feature vectors are defined for the feature vector(s) 256 (eg, unlock feature vectors) may be the output of the encoding of the unlock attempt image at 254.

At 257, a pose of the user's face in the unlock attempt image may be evaluated. Evaluating the pose may include evaluating the orientation of the user's face relative to the device 100. For example, the user's face may be evaluated as in a normal orientation (eg, vertical) with respect to device 100 or in an abnormal orientation (eg, flipped) with respect to the device. In certain embodiments, the pose of the user's face is evaluated using feature vectors 256. For example, the neural network module may be trained to evaluate a pose of the user's face based on feature vectors 256. An example of evaluating feature vectors to determine a user's face pose is found in US Patent Application No. 15/910,551 to Gernoth et al., which is incorporated by reference as fully described herein. In some embodiments, data from one or more sensors on device 100 may be used to evaluate a pose of a user's face based on the orientation of the device. For example, accelerometers and/or gyroscopes on device 100 may be used to evaluate the orientation of the device with respect to the user's face or body and to determine a pose of the user's face in the captured image.

As shown in FIG. 18, the process 250 ″ is based on the pose evaluated at 257 (in addition to using the static portion 222 in the template space 220) or the dynamic portion 224 ') may still be used. In certain embodiments, as described herein, the dynamic portion 224 and the static portion 222 when the pose evaluates to be the first (normal or frontal) pose. The dynamic portion 224' may be used in combination with the static portion 222 when the pose is evaluated to be a second (inverted or low pitch angle) pose. Thus, the process 250" In addition to the static portion 222 that attempts to authenticate the user in the captured unlock attempt image, based on the evaluated pose, optionally continuing with either the dynamic portion 224 or the dynamic portion 224'. .

In certain embodiments, after the pose is evaluated at 257 and the dynamic portion to be used based on the evaluated pose is determined, the process 250" continues at 258 (or optionally 258') as shown in FIG. At 258 (or 258'), to obtain a match score 260 (for dynamic portion 224) or a match score 260' (for dynamic portion 224') for the unlock attempt image, the feature Vector(s) 256 are compared to feature vectors in the templates of template space 220 (using dynamic portion 224 or dynamic portion 224').

After the match score 260 (or match score 260' based on the evaluated pose) is determined, the match score is unlocked at 262 (for match score 260) or (match score 260') at 262. )) at 262'. In certain embodiments, unlock threshold 264 ′ has tighter limits than unlock threshold 264 (eg, more stringent for matching). Tighter constraints may be placed on the unlock threshold 264' due to features available for evaluation in images with a second pose. As described above, typically in a second (inverted or low pitch angle) pose, the user's chin may be a prominent feature, while some portions of the upper part of the user's face, including the eyes, are not evaluated in the image. It may be obscured or occluded (eg, fewer feature vectors may be evaluated for the eye or the area around the eye of the user's face). Evaluating the user's jaw to determine the user's identity may be less reliable than evaluating the features in and around the user's eye. Thus, tighter restrictions can be placed on the unlock threshold 264' in order to maintain security levels suitable for the facial recognition authentication process on the device 100.

As shown in Fig. 18, when the dynamic portion 224 is used based on the evaluated pose, if the matching score 260 exceeds the unlock threshold 264 (that is, the user's face in the unlock attempt image If it substantially matches the face of this authorized user), the user in the unlock attempt image 252 is authenticated as an authorized user for the device 100 and the device is unlocked at 266. Similarly, when the dynamic portion 224' is used based on the evaluated pose, if the matching score 260' exceeds the unlock threshold 264', the user in the unlock attempt image 252 is ), and the device is unlocked at 266'. After the device 100 is unlocked at 266 (or 266'), the unlocked feature vectors 256 and the processed match score (matching score 260 or match score 260') are updated in the template update process 300 ) (Shown in FIG. 10). The template update process 300 then operates to update the corresponding dynamic portion (eg, either dynamic portion 224 or dynamic portion 224') that operated process 250".

In certain embodiments, as shown in FIG. 18, when the dynamic portion 224 is used based on the evaluated pose, the matching score 260 is less than the unlock threshold 264 (e.g., the unlock threshold If not), the device 100 remains locked and the process 250" continues with the locking protocol 460 (shown in Figure 19). The dynamic portion 224' is based on the evaluated pose. When used, if the matching score 260' is less than the unlock threshold 264' (e.g., not above the unlock threshold), the device 100 remains locked, and the process 250' is (FIG. 20 Lock protocol 460').

19 shows a flowchart of an embodiment of a locking protocol 460. As described above, the locking protocol 460 starts at the face recognition authentication process 250" when the matching score 260 is less than the unlock threshold 264 and the device 100 remains locked. Protocol 460 may include some operations from face recognition authentication process 250 shown in Fig. 9. For example, as shown in Fig. 19, lock protocol 460 allows the device to lock at 268. At 270, the number of unlock attempts (eg, the number of attempts to unlock the device 100 using a different unlock attempt image captured at 252) is counted at 270. If the number of unlock attempts of is less than a selected value (eg, a threshold), the process 250" may be run again using another unlock attempt image (eg, a new image of the user is captured and processed).

If the number of unlock attempts exceeds the selected value, at 272 the device 100 is locked from further attempts to use facial authentication. In some embodiments, an error message indicating that the facial recognition authentication process 250 ″ has failed and/or is restricted or prevented from performing the desired operation of the device 100 (eg, on display 108) Device 100 may be locked from further attempts to use facial authentication at 272 for a specified period of time and/or until another authentication protocol is used to unlock the device. In examples, unlock options 274 are used to unlock device 100. In certain embodiments, device 100 is unlocked using one of the unlock options at 274. Afterwards, feature vectors 256 and matching score 260 may be provided to the second template update process 400 (shown in FIG. 13) to update the dynamic portion 224.

20 shows a flowchart of an embodiment of a locking protocol 460'. As described above, the locking protocol 460' begins with the face recognition authentication process 250" when the matching score 260' is below the unlock threshold 264' and the device 100 remains locked. The lock protocol 460' may include some operations from the face recognition authentication process 250 shown in Fig. 9. For example, as shown in Fig. 20, the lock protocol 460' is a device May start with the unlocked state at 268'. At 270', the number of unlock attempts is counted (e.g., an attempt to unlock the device 100 using a different unlock attempt image captured at 252). If the number of unlock attempts at 270' is less than a selected value (e.g., threshold), the process 250" may be run again using a different unlock attempt image (e.g., a new image of the user is Captured and processed).

If the number of unlock attempts exceeds the selected value, at 272' the device 100 is locked from further attempts to use face authentication. In some embodiments, an error message indicating that the facial recognition authentication process 250 ″ has failed and/or is restricted or prevented from performing the desired operation of the device 100 (eg, on display 108) The device may be locked from further attempts to use face authentication at 272' for a specified period of time and/or until another authentication protocol is used to unlock the device. At, unlock options 274' are used to unlock device 100. In certain embodiments, device 100 is unlocked using one of the unlock options at 274'. Afterwards, feature vectors 256 and matching score 260 ′ may be provided to the second template update process 400 (shown in FIG. 13) to update the dynamic portion 224 ′.

In certain embodiments, one or more process steps described herein may be performed by one or more processors (eg, computer processors) executing instructions stored on a non-transitory computer-readable medium. For example, the process 200, process 250, process 300, and process 400, shown in FIGS. 4, 9, 10, and 13, may be computer-readable storage media (e.g., non- Temporary computer readable storage medium).

21 shows a block diagram of one embodiment of an exemplary computer system 510. Exemplary computer system 510 may be used to implement one or more embodiments described herein. In some embodiments, the computer system 510 is a process 200, process 250, process 300, and process 400 shown in Figures 4, 9, 10, and 13 It is operable by a user to implement one or more of the embodiments described herein. In the embodiment of FIG. 21, computer system 510 includes a processor 512, a memory 514, and various peripheral devices 516. Processor 512 is coupled to memory 514 and peripheral devices 516. Processor 512 is configured to execute instructions, including instructions for process 200, process 250, process 300, and/or process 400, which may be in software. In various embodiments, processor 512 may be configured with any desired instruction set (e.g., Intel Architecture-32 (also known as IA-32, x86), IA-32, x86-64, PowerPC, Sparc with 64-bit extensions). , MIPS, ARM, IA-64, etc.) can be implemented. In some embodiments, computer system 510 may include more than one processor. Furthermore, processor 512 may include one or more processors or one or more processor cores.

Processor 512 can be coupled to memory 514 and peripheral devices 516 in any desired manner. For example, in some embodiments, processor 512 may be coupled to memory 514 and/or peripheral devices 516 through various interconnects. Alternatively or additionally, one or more bridge chips may be used to couple the processor 512, memory 514, and peripheral devices 516.

Memory 514 may include any type of memory system. For example, the memory 514 may include DRAM, and more particularly, double data rate (DDR) SDRAM, RDRAM, and the like. A memory controller may be included to interface to the memory 514 and/or the processor 512 may include a memory controller. The memory 514 may store instructions to be executed by the processor 512 during use, data to be operated by the processor during use, and the like.

Peripheral devices 516 are storage devices, optionally including any kind of hardware devices (e.g., computer-accessible storage media 600 shown in FIG. Other input/output (I/O) devices such as video hardware, audio hardware, user interface devices, networking hardware, etc.).

One or more data structures representing the device 100 (shown in Figure 1) included in the integrated circuit design and the process 200, process 250 (shown in Figures 4, 9, 10, and 13) Referring now to FIG. 22, which is a block diagram of one embodiment of a computer-accessible storage medium 600 comprising one or more code sequences representing process 300, and/or process 400. Each code sequence may include one or more instructions that, when executed by a processor in the computer, implement the described operations for the corresponding code sequence. Generally speaking, computer accessible storage media may include any storage media accessible by a computer during use to provide instructions and/or data to the computer. For example, a computer-accessible storage medium is a magnetic or optical medium, such as a disk (fixed or removable), tape, CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW , Or may include non-transitory storage media such as Blu-Ray. Storage media include RAM (e.g., synchronous dynamic RAM (SDRAM), Rambus DRAM (RDRAM), static RAM (SRAM), etc.), ROM, or volatile or nonvolatile memory media such as flash memory. Can be included as. Storage media may be physically contained within a computer in which the storage media provide instructions/data. Alternatively, storage media can be connected to a computer. For example, storage media may be connected to a computer via a network or wireless link, such as network attached storage. Storage media can be connected through a peripheral interface such as a universal serial bus (USB). In general, computer-accessible storage medium 600 may store data in a non-transitory manner, and non-transitory in this context may refer to not transmitting instructions/data on a signal. For example, non-transitory storage may be volatile (and may lose stored instructions/data in response to a power cut) or non-volatile.

Further modifications and alternative embodiments of the various aspects of the embodiments described herein will be apparent to those of ordinary skill in the art in view of this description. Therefore, the present description should be interpreted as illustrative only, and is for the purpose of teaching a general method of performing the embodiments to those of ordinary skill in the art. It should be understood that the forms of the embodiments shown and described herein are taken as presently preferred embodiments. Elements and materials may be substituted for those illustrated and described herein, parts and processes may be interchanged, and certain features of the embodiments may be utilized independently, all of which may be used after gaining the benefit of this description. As will be apparent to those of ordinary skill in the art to which the present invention belongs. Changes may be made in the elements described herein without departing from the spirit and scope of the following claims.

Claims (20)

As a method,
Obtaining a first image of a user's face using a camera located on the device, the device comprising a computer processor and a memory;
Encoding the first image to generate at least one first feature vector, the first feature vector representing one or more facial features of the user in the first image;
Comparing the first feature vector with one or more first reference templates stored in the memory of the device to obtain a first matching score-The first reference template includes reference templates for a first registration profile on the device Ham -;
Comparing the first feature vector with one or more second reference templates stored in the memory of the device to obtain a second matching score-The second reference template includes reference templates for a second registration profile on the device Ham -;
If at least one of the first match score or the second match score exceeds an unlock threshold, authorizing the user to perform at least one operation on the device requesting authentication;
If the first matching score exceeds a first threshold,
Storing the first feature vector as a first temporary template in the memory of the device; And
Operating a first template update process for the first reference template by using the first temporary template; And
If the second matching score exceeds a second threshold,
Storing the first feature vector as a second temporary template in the memory of the device; And
And operating a second template update process for the second reference template using the second temporary template. The method of claim 1, wherein the first threshold is substantially equal to the second threshold. The method of claim 1, wherein the first threshold and the second threshold are higher than the unlock threshold. The method of claim 1, wherein the first image is an infrared image of the face of the user. The method of claim 1, wherein the first registration profile includes a registration profile for the user of the device, and the second registration profile includes a registration profile for an additional user of the device. The method of claim 1, wherein the first registration profile includes a registration profile for a first appearance of the user of the device, and the second registration profile includes a registration profile for a second appearance of the user of the device. How to. The method of claim 1, wherein the first template update process,
Acquiring a second image of the face of the user using the camera;
Encoding the second image to generate at least one second feature vector, the second feature vector representing one or more facial features of the user in the second image;
Obtaining a third matching score by comparing the second feature vector with the first temporary template;
Evaluating a confidence score for the first temporary template by evaluating whether the third matching score exceeds a third threshold; And
And if the evaluated confidence score exceeds the selected confidence score, adding the first temporary template to the first reference template. The method of claim 1, wherein the first reference template in the first registration profile,
Acquiring a plurality of reference images of the face of the user using the camera;
Selecting reference images for encoding based on selected reference images that meet selected criteria for the reference images;
Generating the plurality of reference feature vectors by encoding a plurality of facial features of the user from the selected reference images; And
And storing the plurality of reference feature vectors as the first reference template in the memory of the device. As a method,
Obtaining a first image of a user's face using a camera located on the device, the device comprising a computer processor and a memory;
Encoding the first image to generate at least one first feature vector, the first feature vector representing one or more facial features of the user in the first image;
Comparing the first feature vector with one or more first reference templates stored in the memory of the device to obtain a first matching score-The first reference template includes reference templates for a first registration profile on the device Ham -;
Comparing the first feature vector with one or more second reference templates stored in the memory of the device to obtain a second matching score-The second reference template includes reference templates for a second registration profile on the device Ham -;
If at least one of the first match score or the second match score exceeds an unlock threshold, authorizing the user to perform at least one operation on the device requesting authentication; or
If both the first matching score and the second matching score are less than the unlocking threshold,
Providing the user with an option to proceed with at least one additional authentication protocol; And
In response to receiving the user's authorization via the at least one additional authentication protocol,
If the first matching score exceeds the second matching score, operating a template update process for the first reference template; or
And if the second match score exceeds the first match score, operating the template update process for the second reference template. The method of claim 9, wherein the at least one additional authentication protocol comprises entering a passcode for the device. 10. The method of claim 9, wherein the first registration profile comprises a registration profile for the user of the device, and the second registration profile comprises a registration profile for an additional user of the device. The method of claim 9, wherein the first registration profile includes a registration profile for a first appearance of the user of the device, and the second registration profile includes a registration profile for a second appearance of the user of the device. How to. The method of claim 9, wherein the template update process,
If the first match score or the second match score exceeds a first threshold, storing the first feature vector as a temporary template in the memory of the device;
Acquiring a second image of the face of the user using the camera;
Encoding the second image to generate at least one second feature vector, the second feature vector representing one or more facial features of the user in the second image;
Comparing the second feature vector with the stored temporary template to obtain a third matching score;
Evaluating a confidence score for the temporary template by comparing the third matching score with a second threshold, the second threshold exceeding the first threshold; And
If the evaluated confidence score exceeds the selected confidence score, adding the temporary template to the first reference template or the second reference template. 14. The method of claim 13, further comprising authenticating the user to use the device for a selected period of time using the temporary template. As a device,
camera;
At least one illuminator for providing infrared illumination;
And a circuit portion coupled to the camera and the illuminator, and the circuit portion,
Obtaining a first image of the user's face using the camera;
Encoding the first image to generate at least one first feature vector, the first feature vector representing one or more facial features of the user in the first image;
Comparing the first feature vector with one or more first reference templates stored in the memory of the device to obtain a first matching score-the first reference template includes reference templates for a first registration profile on the device -;
Comparing the first feature vector with one or more second reference templates stored in the memory of the device to obtain a second matching score-the second reference template includes reference templates for a second registration profile on the device -;
Authorizing the user to perform at least one operation on the device requesting authentication when at least one of the first matching score or the second matching score exceeds an unlocking threshold;
If the first matching score exceeds a first threshold,
Storing the first feature vector as a first temporary template in the memory of the device;
Operating a first template update process for the first reference template by using the first temporary template;
If the second matching score exceeds a second threshold,
Storing the first feature vector as a second temporary template in the memory of the device;
The device, configured to operate a second template update process for the second reference template using the second temporary template. The device of claim 15, wherein the circuitry is configured to perform a registration process to generate the first registration profile and the second registration profile. The method of claim 16, wherein the circuit unit, in performing the registration process,
Acquiring a plurality of reference images of the face of the user using the camera;
Selecting reference images for encoding based on selected reference images that satisfy selected criteria for the reference images;
Encoding a plurality of facial features of the user from the selected reference images to generate the plurality of reference feature vectors;
And storing the plurality of reference feature vectors as the reference templates in the memory of the device. The device of claim 15, wherein the first registration profile includes a registration profile for the user of the device, and the second registration profile includes a registration profile for an additional user of the device. The method of claim 15, wherein the first registration profile includes a registration profile for a first appearance of the user of the device, and the second registration profile includes a registration profile for a second appearance of the user of the device. That, the device. The method of claim 15, wherein the circuitry is configured to remove both the first registration profile and the second registration profile in response to a request to remove the first registration profile or a request to remove the second registration profile. device.

Images