CN112307451A

CN112307451A - Multi-mode identity recognition method and equipment

Info

Publication number: CN112307451A
Application number: CN201910703887.3A
Authority: CN
Inventors: 李伟; 王保辉; 李晓建; 胡伟湘
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-07-31
Filing date: 2019-07-31
Publication date: 2021-02-02
Also published as: WO2021017988A1

Abstract

The application provides a multi-mode identity recognition method and equipment, and relates to the technical field of artificial intelligence identity recognition; when a user is identified on the basis of one mode, actively acquiring registration information of other modes of the user to complete a multi-mode registration process; and the user does not need to separately register each mode, thereby simplifying the multi-mode registration process. The user registers in one mode on the electronic equipment, and the electronic equipment can actively acquire information in other modes, complete the registration process of multiple modes and support multi-mode identity recognition.

Description

Multi-mode identity recognition method and equipment

Technical Field

The embodiment of the application relates to the technical field of identity recognition, in particular to a multi-mode identity recognition method and equipment.

Background

With the continuous maturity of identification technologies such as fingerprint identification, face identification, voiceprint identification, and iris identification, electronic devices have been provided with the capability of identifying the identity of a user through fingerprints, faces, voiceprints, and irises. Identification modes such as fingerprint identification, face identification, voiceprint identification or iris identification are respectively called as a mode of identity identification. Identity recognition using one modality is called single modality identity recognition; identification using multiple modalities is known as multi-modal identification.

Each modality of identity recognition is generally divided into two processes, registration and authentication. For example, if the electronic device successfully performs fingerprint identification, a user needs to perform fingerprint registration on the electronic device first; that is, the user enters a fingerprint on the electronic device in advance, and the fingerprint entered in advance may be referred to as an enrolled fingerprint. When the electronic equipment performs identity authentication, the input fingerprint of the user performing identity authentication can be compared with the registered fingerprint, so that the identity of the user inputting the fingerprint is determined.

At present, in an electronic device supporting multiple modalities, registration processes of the modalities are independent, that is, a same user needs to register for each modality for multiple times, and the registration process is relatively complicated. For example, the user 1 performs fingerprint registration on the electronic device once; then, the user 1 performs a voiceprint registration on the electronic device; then, the user 1 performs face registration on the electronic equipment for one time; the user 1 also performs iris registration once on the electronic device; user 1 has performed four registration processes on the electronic device.

Moreover, after the same user registers for different modalities, the registration information of each modality cannot be automatically associated with the same user on the electronic device. For example, the user 1 performs fingerprint registration once on the electronic device, and the registered fingerprint 1 is stored on the electronic device; the user 1 performs voiceprint registration on the electronic equipment once, and the electronic equipment stores the registered voiceprint 1; the user 2 performs fingerprint registration on the electronic equipment once, and the electronic equipment stores the registered fingerprint 2; the electronic device cannot automatically associate the enrolled fingerprint 1 and the enrolled voiceprint 1 to the same user.

For an electronic device supporting multi-modal, how to simplify the multi-modal registration process is a problem to be solved.

Disclosure of Invention

The embodiment of the application provides a multi-mode identity recognition method and device, which can actively acquire registration information of other modes of a user in the process of identity recognition of the user based on one mode, simplify a multi-mode registration process and improve the convenience of identity recognition of the user.

In a first aspect, an embodiment of the application provides a multi-modal identity recognition method, a multi-modal identity recognition device and electronic equipment.

In one possible design, the method may include: the method comprises the steps that the electronic equipment receives a registration operation of a user in a first mode, and the electronic equipment also receives an authentication operation of the user in the first mode; the first mode is any one of a plurality of identity recognition modes supported by the electronic equipment; if the electronic equipment successfully authenticates the user based on the first mode, the electronic equipment collects registration information of one or more second modes of the user in response to the authentication operation of the user in the first mode; and completing the registration process of the user in the one or more second modalities according to the registration information of the one or more second modalities of the user.

According to the method, a user registers in one mode on the electronic equipment, the electronic equipment can actively acquire registration information in other modes, the registration process of the second mode is completed, multi-mode identity recognition is supported, and a multi-mode registration process is simplified. Moreover, after the user registers in one mode, the electronic device actively acquires the registration information of other modes of the user, the registration information of multiple modes acquired by the electronic device is automatically associated with the same user, and the electronic device can authenticate the user based on one or more of the multiple modes.

In one possible design, the electronic device collects registration information of one or more second modalities of the user for a preset time period. That is, the time period for collecting the registration information may be set.

In one possible design, the electronic device receives a confirmation operation of the user to acquire the registration information of the one or more second modalities of the user before the electronic device acquires the registration information of the one or more second modalities of the user. That is to say, the electronic device determines that the user agrees to the automatic acquisition of the registration information by the electronic device, and the electronic device executes the scheme, so that the user information is prevented from being leaked.

In one possible design, after the electronic device receives the registration operation of the user in the first modality, if it is determined that the registration of the user in the first modality is successful, the electronic device collects registration information of one or more second modalities of the user in response to the registration operation of the user in the first modality. In the method, the registration information of the second modality can be collected once after the user successfully registers actively.

In one possible design, before the electronic device completes the registration process of the user in the one or more second modalities according to the registration information of the one or more second modalities of the user, the electronic device determines that the registration information meets the preset first criteria. Wherein the first criterion may comprise the number of samples taken being greater than a first threshold.

In a possible design, after the electronic device collects the registration information of the second modality, the collected information may be further filtered.

In one possible design, the modalities supported by the electronic device may include: fingerprint recognition, face recognition, voiceprint recognition or iris recognition.

In one possible design, the first modality is fingerprint identification, and the electronic device receives an operation of fingerprint authentication performed by a user on the electronic device; and if the fingerprint authentication of the user is successful, responding to the operation of fingerprint authentication of the user on the electronic equipment, and acquiring the face image of the user by the electronic equipment. Therefore, the electronic equipment can use the collected face image to complete face registration; the electronic equipment can perform identity authentication on the user by adopting at least one identity recognition mode of fingerprint recognition and face recognition. In one possible design, the electronic device also captures a voice of the user in response to the user performing a fingerprint authentication operation on the electronic device. Therefore, the electronic equipment can use the collected voice to complete the voiceprint registration of the user; the electronic equipment can perform identity authentication on the user by adopting one or more identity recognition modes of fingerprint recognition, face recognition and voiceprint recognition.

In one possible design, the first modality is voiceprint recognition, and the electronic device receives input voice of a user and performs voiceprint authentication on the user according to the input voice of the user; and if the voiceprint authentication of the user is successful, the electronic equipment acquires a face image of the user. Therefore, the electronic equipment can use the collected face image to complete face registration; the electronic equipment can perform identity authentication on the user by adopting at least one identity recognition mode of voiceprint recognition and face recognition.

In one possible design, the electronic device may filter the acquired registration information of the second modality. For example, the face image is collected, and the face image can be screened to remove an incomplete face image and an unclear face image. In one possible design, the electronic device may filter the collected registration information of the second modality according to the stored registration information. For example, if the user is determined to be a male according to the stored voiceprint information, the female face image in the acquired face image can be removed.

In a possible design, after the electronic device completes a registration process of the second modality, a prompt message may be displayed for confirming to the user that the identification mode of the second modality is enabled.

Correspondingly, the application also provides a multi-mode identity recognition device, and the device can realize the multi-mode identity recognition method of the first aspect. The device can realize the method through software, hardware or corresponding software executed by hardware.

In one possible design, the apparatus may include: the device comprises a registration module, an authentication module and an input module. The registration module is used for receiving registration operation of a user in a first mode; the first mode is any one of a plurality of identity recognition modes supported by the multi-mode identity recognition device; the authentication module is used for receiving authentication operation of the user in a first mode; the input module is used for responding to the authentication operation of the user in the first mode and collecting the registration information of one or more second modes of the user if the authentication module successfully authenticates the user based on the first mode; and the registration module is also used for completing the registration process of the user in one or more second modalities according to the registration information of the one or more second modalities of the user.

The modalities supported by the multi-modality identity recognition device can include: fingerprint recognition, face recognition, voiceprint recognition or iris recognition.

In one possible embodiment, the input module is specifically configured to collect registration information of one or more second modalities of the user within a preset time period.

In one possible design, the apparatus further includes: and the human-computer interaction module is used for receiving confirmation operation of the user on acquisition of the registration information of one or more second modalities of the user.

In one possible design, the input module is further configured to collect registration information of one or more second modalities of the user in response to a registration operation of the user in the first modality if the registration module determines that the user is successfully registered in the first modality.

In a possible design, the human-computer interaction module is further configured to confirm to the user that the identification mode of the second modality is enabled after the device completes a registration process of the second modality.

Accordingly, the present application also provides an electronic device, which may include a processor, an input unit, and a memory. The memory is for coupling with the processor and holds the necessary program instructions and data for the electronic device.

In one possible design, the processor is configured to receive a registration operation of a user in a first modality, where the first modality is any one of a plurality of identification manners supported by the electronic device; the authentication server is also used for receiving an authentication operation of a user in a first modality; the input unit is used for responding to the authentication operation of the user in the first mode and collecting the registration information of one or more second modes of the user if the processor determines that the electronic equipment successfully authenticates the user based on the first mode; and the processor is further used for completing the registration process of the user in the one or more second modalities according to the registration information of the one or more second modalities of the user.

Among them, the supported modalities of the electronic device may include: fingerprint recognition, face recognition, voiceprint recognition or iris recognition.

In one possible embodiment, the input unit is specifically configured to collect registration information of one or more second modalities of the user within a preset time period.

In one possible design, the processor is further configured to receive a confirmation operation of the user to acquire the registration information of the one or more second modalities of the user.

In one possible design, the input unit is further configured to collect registration information of one or more second modalities of the user in response to a registration operation of the user in the first modality if the processor determines that the user is successfully registered in the first modality.

In a possible design, the processor is further configured to confirm to the user that the second modality is enabled after the device completes a registration process of the second modality.

In a second aspect, an embodiment of the application provides a multi-modal identity recognition method, a multi-modal identity recognition device and an electronic device.

In one possible design, the method may include: the electronic equipment receives an authentication operation of a user in a first mode; the first mode is any one of a plurality of identity recognition modes supported by the electronic equipment; if the electronic equipment successfully authenticates the user based on the first mode, the electronic equipment collects registration information of the first mode and one or more second modes of the user in response to the authentication operation of the user in the first mode; the electronic equipment updates the algorithm model of the first mode according to the registration information of the first mode of the user; and updating the algorithm model of each second mode according to the registration information of each second mode.

In the method, the electronic equipment can acquire the registration information of multiple modalities in the process of carrying out user identity recognition based on one modality, and respectively updates the algorithm model of each modality according to the acquired registration information, so that the better the algorithm model is used, and the accuracy of the user identity recognition is improved.

In one possible design, the electronic device collects registration information of a first modality of the user and one or more second modalities within a preset time period. That is, the time period for collecting the registration information may be set.

In one possible design, before the electronic device collects the registration information of the first modality and the one or more second modalities of the user, the electronic device receives a confirmation operation of the user to collect the registration information of the first modality and the one or more second modalities of the user. That is to say, the electronic device determines that the user agrees to the automatic acquisition of the registration information by the electronic device, and the electronic device executes the scheme, so that the user information is prevented from being leaked.

In one possible design, after the electronic device collects the registration information of the first modality and the one or more second modalities, the collected information may be further filtered.

In one possible design, the modalities supported by the electronic device include: fingerprint recognition, face recognition, voiceprint recognition or iris recognition.

In one possible design, the first modality is fingerprint identification, and the electronic device receives an operation of fingerprint authentication performed by a user on the electronic device; and if the fingerprint authentication of the user is successful, responding to the operation of fingerprint authentication of the user on the electronic equipment, and acquiring the face image of the user by the electronic equipment. In this way, the electronic device can update the algorithm model of fingerprint identification using the acquired fingerprint image; and updating the algorithm model of the face recognition by using the acquired face image. In one possible design, the electronic device also captures a voice of the user in response to the user performing a fingerprint authentication operation on the electronic device. In this way, the electronic device may also update the algorithmic model of voiceprint recognition using the collected user speech.

In one possible design, the first modality is voiceprint recognition, and the electronic device receives input voice of a user and performs voiceprint authentication on the user according to the input voice of the user; and if the voiceprint authentication of the user is successful, the electronic equipment acquires a face image of the user. In this way, the electronic device may update the algorithmic model of voiceprint recognition using the collected input speech of the user; and updating the algorithm model of the face recognition by using the acquired face image.

In one possible design, the electronic device may filter the collected registration information of the first modality and the one or more second modalities. For example, the face image is collected, and the face image can be screened to remove an incomplete face image and an unclear face image. In one possible design, the electronic device may filter the collected registration information according to the stored registration information. For example, if the user is determined to be a male according to the stored voiceprint information, the female face image in the acquired face image can be removed.

Correspondingly, the application also provides a multi-mode identity recognition device, and the device can realize the multi-mode identity recognition method of the second aspect. The device can realize the method through software, hardware or corresponding software executed by hardware.

In one possible design, the apparatus may include: the device comprises an authentication module, an input module and an algorithm model module. The authentication module is used for receiving authentication operation of a user in a first mode; the first mode is any one of a plurality of identity recognition modes supported by the multi-mode identity recognition device; the input module is used for responding to the authentication operation of the user in the first mode and collecting the registration information of the first mode and one or more second modes of the user if the authentication module successfully authenticates the user based on the first mode; the algorithm model module is used for updating the algorithm model of the first mode according to the registration information of the first mode of the user; and updating the algorithm model of each second mode according to the registration information of each second mode.

In one possible design, the input module is specifically configured to collect registration information of a first modality and one or more second modalities of a user within a preset time period.

In one possible design, the apparatus further includes: and the human-computer interaction module is used for receiving confirmation operation of the user on acquisition of the registration information of the first modality and one or more second modalities of the user.

In one possible design, the processor is configured to receive an authentication operation of a user in a first modality; the first mode is any one of a plurality of identity recognition modes supported by the electronic equipment; the input unit is used for responding to the authentication operation of the user in the first mode and acquiring the registration information of the first mode and one or more second modes of the user if the processor successfully authenticates the user based on the first mode; the processor is further used for updating the algorithm model of the first modality according to the registration information of the first modality of the user; and updating the algorithm model of each second mode according to the registration information of each second mode.

In one possible embodiment, the input unit is specifically configured to collect registration information of a first modality and one or more second modalities of a user within a preset time period.

In one possible design, the processor is further configured to receive a confirmation operation of the user to acquire the registration information of the first modality of the user and the one or more second modalities.

In a third aspect, the present application further provides a graphical user interface, GUI, stored in an electronic device.

In one possible design, in response to the electronic device successfully authenticating the user based on the first modality, the electronic device displays a first GUI at a first interface; the first GUI comprises first information, the first information is used for prompting to collect registration information of a user in at least one mode, and the mode comprises the following steps: fingerprint identification, face identification, voiceprint identification or iris identification; the first GUI further comprises a first button and a second button, wherein the first button is used for triggering the electronic equipment to confirm that the user agrees to acquire registration information of the user in at least one modality; the second button is used for triggering the electronic equipment to confirm that the user does not agree to acquire the registration information of the user in at least one mode.

Wherein the first GUI may be a prompt box at which the user may confirm or disagree with acquiring registration information for each modality. The first button may be a confirm button and the second button may be a cancel button.

The first interface may be a display interface on the electronic device after the user is successfully authenticated.

In one possible design, the graphical user interface further includes: responding to the clicking operation of the user on the first button, and displaying a second GUI on the first interface by the electronic equipment; the second GUI includes a plurality of options, wherein each option corresponds to a modality.

Wherein the second GUI may be a second prompt box; the user may select the type of personal information authorized to be collected by selecting various options in the second prompt box.

In one possible design, the first GUI further includes a plurality of options, wherein each option corresponds to a modality. In this implementation, options for each modality are provided in the first GUI. The user may select the category of personal information authorized to be collected directly in the first GUI.

In a fourth aspect, the present application further provides a graphical user interface, GUI, stored in an electronic device.

In one possible design, in response to a user successfully registering in a first modality on the electronic device, the electronic device displays a first GUI in a first interface; the first GUI comprises first information, and the first information is used for prompting the acquisition of registration information of a user in at least one modality, wherein the modality comprises: fingerprint identification, face identification, voiceprint identification or iris identification; the first GUI further comprises a first button and a second button, wherein the first button is used for triggering the electronic equipment to confirm that the user agrees to acquire registration information of the user in at least one modality; the second button is used for triggering the electronic equipment to confirm that the user does not agree to acquire the registration information of the user in at least one mode.

The first interface may be an interface on the electronic device where the user performs identity registration.

In a fifth aspect, the present application further provides a graphical user interface, GUI, stored in an electronic device.

In one possible design, in response to the registration process of the second modality being completed on the electronic device, the electronic device displays a third GUI at the second interface; the third GUI includes second information, and the second information is used to prompt the electronic device to register successfully in a second modality, where the second modality includes: fingerprint recognition, face recognition, voiceprint recognition or iris recognition.

Wherein the third GUI may be a prompt box.

The second interface may be a desktop of the electronic device; the interface may also be displayed currently when the electronic device completes the registration process of the second modality.

In one possible design, the third GUI further includes a third button and a fourth button, and the third button is used to trigger the identification mode for enabling the second modality; the fourth button is used to trigger an identification mode that does not enable the second modality.

The third button may be a confirm button and the fourth button may be a cancel button.

The present application also provides a computer storage medium having stored therein instructions that, when run on an electronic device, cause the electronic device to perform the method of any of the above aspects.

The present application also provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any of the above aspects.

Any one of the above-mentioned apparatuses, electronic devices, computer storage media, or computer program products is configured to execute the above-mentioned corresponding method, so that the beneficial effects achieved by the apparatuses, the electronic devices, the computer storage media, or the computer program products can refer to the beneficial effects of the corresponding schemes in the above-mentioned corresponding methods, and are not described herein again.

Drawings

FIG. 1 is a schematic diagram of a fingerprint registration interface;

FIG. 2 is a schematic diagram of a face registration interface;

FIG. 3 is a schematic diagram of a voice registration interface;

FIG. 4 is a schematic view of an iris enrollment interface;

fig. 5 is a schematic composition diagram of a hardware structure of an electronic device according to an embodiment of the present disclosure;

fig. 6 is a first flowchart of a multi-modal identity recognition method according to an embodiment of the present application;

fig. 7A is a first schematic view of an example display interface of an electronic device according to an embodiment of the present disclosure;

fig. 7B is a schematic view illustrating an example display interface of an electronic device according to an embodiment of the present application;

fig. 7C is a third schematic view of an example display interface of an electronic device according to an embodiment of the present application;

fig. 7D is a fourth schematic view of an example display interface of an electronic device according to an embodiment of the present application;

fig. 8 is a schematic view of an example display interface of an electronic device according to an embodiment of the present disclosure;

fig. 9 is a flowchart of a multi-modal identity recognition method according to an embodiment of the present application;

fig. 10 is a schematic flowchart of a multi-modal identity recognition method executed in an electronic device according to an embodiment of the present application;

fig. 11 is a schematic structural component diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Each mode of identity recognition is generally divided into two processes of registration and authentication.

In one example, fingerprint identification technology is divided into two processes, fingerprint enrollment and fingerprint authentication.

The user firstly registers the fingerprint on the electronic equipment. A user inputs a registration fingerprint; the electronic equipment collects fingerprint images of the registered fingerprints. Optionally, the electronic device may further perform preprocessing on the acquired fingerprint image, for example, the preprocessing on the fingerprint image may include: image normalization processing, image enhancement processing, image binarization processing and the like. And then, the electronic equipment can directly extract the fingerprint characteristics of the acquired fingerprint image to obtain the fingerprint characteristics of the registered fingerprint. Optionally, the electronic device may also perform preprocessing on the acquired fingerprint image, and then perform fingerprint feature extraction on the preprocessed fingerprint image to obtain the fingerprint feature of the registered fingerprint. Further, the electronic device may store fingerprint characteristics of the enrolled fingerprints.

In the fingerprint authentication process, the electronic equipment collects a fingerprint image of the user authentication fingerprint, and performs fingerprint feature extraction on the fingerprint image of the authentication fingerprint to obtain the fingerprint feature of the authentication fingerprint. The electronic equipment compares the fingerprint characteristics of the authentication fingerprint with the fingerprint characteristics of the registered fingerprint, and if the matching degree of the fingerprint characteristics of the authentication fingerprint and the fingerprint characteristics of the registered fingerprint is larger than a preset value, the user performing fingerprint authentication is determined to be the user entering the registered fingerprint. In one implementation, the degree of match of two features may be characterized with a confidence level. The confidence may be a score, with a higher score indicating a higher match between the two features. It is to be understood that the expression form of the confidence level is not limited in the embodiments of the present application, and the confidence level may also be described as a confidence probability, a confidence score, and the like.

The specific implementation of the processes of collecting the fingerprint image, preprocessing the fingerprint image, extracting the fingerprint features, and the like, of the electronic device can refer to the description in the conventional technology, and is not repeated here.

In one example, the face recognition technology is divided into two processes of face registration and face authentication.

The user firstly carries out face registration on the electronic equipment. A user inputs a registered face; the electronic equipment collects the image input by the user, and detects and extracts the face image from the image input by the user. And then, the electronic equipment can extract the face features of the registered face image to obtain the face features of the registered face and store the face features of the registered face image.

In the face authentication process, the electronic equipment collects a face image of a user, and carries out face feature extraction on the collected face image to obtain the face feature of the collected face image. The electronic equipment compares the face characteristics of the face image acquired in the face authentication process with the face characteristics of the registered face image, and if the matching degree of the face characteristics of the face image acquired in the face authentication process and the face characteristics of the registered face image is larger than a preset value, the user performing face authentication is determined to be the user of the registered face image.

The specific implementation of the processes of extracting the face image and extracting the face features by the electronic device may refer to the description in the conventional technology, and is not described herein again.

In one example, voiceprint recognition techniques are divided into two processes, voiceprint enrollment and voiceprint authentication.

The user first performs voiceprint registration on the electronic device. A user inputs registration voice; the electronic equipment collects the registration voice through the sound pickup equipment. Optionally, the electronic device may perform pre-processing on the collected enrollment speech, for example, the pre-processing on the speech may include noise reduction processing, speech enhancement, and the like. And then, the electronic equipment can directly extract the voiceprint features of the collected registration voice to obtain the voiceprint features of the registration voice. Optionally, the electronic device may also perform preprocessing on the collected registration voice, and then perform voiceprint feature extraction on the preprocessed registration voice to obtain a voiceprint feature of the registration voice. Further, the electronic device may store voiceprint characteristics of the registered voice.

In the voiceprint authentication process, the electronic device collects authentication voice of the user. The electronic equipment can directly extract the voiceprint characteristics of the collected authentication voice to obtain the voiceprint characteristics of the authentication voice. Optionally, the electronic device may also perform preprocessing on the acquired authentication voice, and then perform voiceprint feature extraction on the preprocessed authentication voice to obtain a voiceprint feature of the authentication voice. The electronic equipment compares the voiceprint characteristics of the authentication voice with the voiceprint characteristics of the registration voice, and if the matching degree of the voiceprint characteristics of the authentication voice and the voiceprint characteristics of the registration voice is larger than a preset value, the user performing voiceprint authentication is determined to be the user inputting the registration voice.

The specific implementation of the processes of acquiring the voice, preprocessing the voice, extracting the voiceprint features, and the like of the electronic device can refer to the description in the conventional technology, and is not described herein again.

In one example, iris recognition techniques are divided into two processes, iris enrollment and iris authentication.

The user first performs iris registration on the electronic device. The electronic equipment collects the images of the eyes of the user and extracts the iris images from the images of the eyes of the user. Optionally, the electronic device may perform preprocessing on the iris image, for example, the preprocessing on the iris image may include: iris localization, image normalization, image enhancement, and the like. And then, the electronic equipment can directly extract the iris features of the extracted iris image to obtain the iris features of the iris image. Optionally, the electronic device may also perform preprocessing on the extracted iris image, and then perform iris feature extraction on the preprocessed iris image to obtain iris features of the iris image. Further, the electronic device may save iris features of the iris image.

In the iris authentication process, the electronic device collects an iris image of the user. Optionally, the electronic device may pre-process the iris image. Then, the electronic equipment can directly extract the iris features of the acquired iris image to obtain the iris features of the acquired iris image; optionally, the electronic device may also perform preprocessing on the acquired iris image, and then perform iris feature extraction on the preprocessed iris image to obtain iris features of the iris image. The electronic equipment compares the iris characteristics of the iris image acquired in the iris authentication process with the iris characteristics of the iris image acquired in the iris registration process, and if the matching degree of the iris characteristics of the iris image acquired in the iris authentication process and the iris characteristics of the iris image acquired in the iris registration process is larger than a preset value, the user performing the iris authentication is determined to be the user performing the iris registration.

The specific implementation of the processes of extracting the iris image, preprocessing the iris image, extracting the iris features, and the like by the electronic device may refer to the description in the conventional technology, and is not described herein again.

For an electronic device supporting multiple modalities, a user needs to register multiple modalities respectively. For example, for an electronic device supporting fingerprint recognition, face recognition, voiceprint recognition and iris recognition, fingerprint recognition, face recognition, voiceprint recognition and iris recognition are required to be performed on a user, and the user needs to perform fingerprint registration, face registration, voiceprint registration and iris registration. The user registers for many times, and the process is complicated.

Exemplarily, the electronic device is taken as a mobile phone, and the processes of fingerprint registration, face registration, voiceprint registration and iris registration of the user are introduced respectively.

As shown in fig. 1, a user can turn on the setting function of the mobile phone and perform fingerprint registration, face registration, voiceprint registration or iris registration on a "security and privacy" interface, respectively. Included in the "security and privacy" interface may be a "fingerprint" option 101, a "face recognition" option 102, a "voice" option 103, and an "iris recognition" option 104, among others. The user may perform a fingerprint registration, a face registration, a voiceprint registration, or an iris registration by clicking on "fingerprint" option 101, "face recognition" option 102, "voice" option 103, or "iris recognition" option 104, respectively.

With continued reference to fig. 1, the handset may receive a user click operation (e.g., a single click operation) on the "fingerprint" option 101. In response to a user clicking on the "fingerprint" option 101, the handset may display a "fingerprint" interface 105. The "fingerprint" interface 105 may include a "create an on-screen fingerprint" option 106, where the "create an on-screen fingerprint" option 106 is used to trigger the handset to create a new registered fingerprint. The mobile phone may receive a user click operation on the "new intra-screen fingerprint" option 106, and in response to the user click operation on the "new intra-screen fingerprint" option 106, the mobile phone may display a "new fingerprint" interface 107. The "newly created fingerprint" interface 107 may include entering a fingerprint prompt message "a finger is placed in the on-screen fingerprint sensing area and pressed slightly harder. The "new fingerprint" interface 107 may also include a "start entry" button 108 for triggering the entry fingerprint process. The handset may receive a user click operation on the "start entry" button 108, and the "new fingerprint" interface 107 of the handset refreshes in response to the user click operation on the "start entry" button 108. The 'newly built fingerprint' interface 107 of the mobile phone displays a fingerprint sensing area 109, and the fingerprint sensing area 109 is used for collecting fingerprints input by a user. The "create new fingerprint" interface 107 of the mobile phone may further include a fingerprint acquisition progress display area 10a for displaying the progress of fingerprint acquisition. When the mobile phone acquires a complete registered fingerprint, the interface 107 for creating the new fingerprint is refreshed. The "newly created fingerprint" interface 107 may display a fingerprint entry success prompt message "entry success". The "newly-built fingerprint" interface 107 of the mobile phone can also comprise an "ok" button 10b, when the mobile phone receives the click operation of the "ok" button 10b by the user, the fingerprint registration is determined to be completed once, and the mobile phone displays a "security and privacy" interface.

Referring to fig. 2, the mobile phone may receive a click operation (e.g., a single click operation) of the "face recognition" option 102 from the user. In response to a user clicking on the "face recognition" option 102, the handset may display a "face recognition" interface 201. The 'face recognition' interface 201 can include a 'set face' option 202, and the 'set face' option 202 is used for triggering the mobile phone to create a new registered face. The mobile phone can receive the click operation of the user on the 'set face content' option 202, and in response to the click operation of the user on the 'set face content' option 202, the mobile phone can display an 'enter face content' interface 203. The face-entry interface 203 may include face-entry prompt information for prompting a user about notes in a face entry process, and the face-entry interface 203 may further include a "start entry" button 204 for triggering the face entry process. The handset may receive a user click operation on the "start entry" button 204, and in response to the user click operation on the "start entry" button 204, the handset displays an "entering" interface 205. The "entering" interface 205 of the handset may include a face acquisition area 206 for acquiring a face image entered by the user. After the mobile phone acquires a complete registered face image, an 'entry success' interface 207 can be displayed, so that one face registration is completed. Further, the handset may return to displaying the "security and privacy" interface.

Referring to fig. 3, the handset may receive a click operation (e.g., a single click operation) of the "voice" option 103 from the user. In response to a user clicking on the "voice" option 103, the handset may display a "voice" interface 301. The "voice" interface 301 may include a "new voice" option 302, where the "new voice" option 302 is used to trigger the handset to create a new registered voiceprint. The mobile phone can receive the click operation of the user on the 'new sound' option 302, and in response to the click operation of the user on the 'new sound' option 302, the mobile phone can display an 'input sound' interface 303. The "enter sound" interface 303 may include a "microphone" option 304 and a recording progress bar 305. Where the "microphone" option 304 is used to trigger the handset to start recording sound. The recording progress bar 305 is used for displaying the progress of the sound recorded by the mobile phone. Optionally, the "record sound" interface 303 may further include a recording prompt message "please record sound about 20 cm away from the mobile phone in a quiet environment. The "enter sound" interface 303 also includes a "cancel" button 306 and an "ok" button 307. The "ok" button 307 is used to trigger the handset to save the recorded sound. A "cancel" button 306 is used to trigger the handset to cancel the voiceprint registration. In response to a user's click operation of the "ok" button 307, the cellular phone may display an entry sound completion interface 308. Further, the handset may return to displaying the "security and privacy" interface.

Referring to fig. 4, the mobile phone may receive a user click operation (e.g., a single click operation) on the "iris recognition" option 104. In response to a user clicking on the "iris recognition" option 104, the handset may display an "iris recognition" interface 401. The "iris recognition" interface 401 may include an "enter iris" option 402, where the "enter iris" option 402 is used to trigger the mobile phone to create a new registered iris. The cell phone may receive a user click operation on the "enter iris" option 402, and in response to the user click operation on the "enter iris" option 402, the cell phone may display an "enter iris" interface 403. The "enter iris" interface 403 may include iris image entry prompt information for prompting the user for notes in entering the iris image, and the "enter iris" interface 403 may also include a "start entry" button 404 for triggering the enter iris process. The cell phone may receive a user click operation on the "start entry" button 404, and in response to the user click operation on the "start entry" button 404, the cell phone displays an "entering" interface 405. The "entering" interface 405 of the handset may include an iris acquisition area 406 for acquiring iris images entered by the user. When the mobile phone acquires a complete registered iris image, an 'entry success' interface 407 can be displayed, so that one iris registration is completed. Further, the handset may return to displaying the "security and privacy" interface.

To complete fingerprint registration, face registration, voiceprint registration and iris registration on a mobile phone, a user needs to set the mobile phone four times respectively. For example, fingerprint registration is completed by the registration method shown in fig. 1, face registration is completed by the registration method shown in fig. 2, voiceprint registration is completed by the registration method shown in fig. 3, and iris registration is completed by the registration method shown in fig. 4; the registration process is cumbersome.

The embodiment of the application provides a multi-mode identity recognition method, and after a user registers in one mode, electronic equipment can actively acquire user registration information in other modes. Therefore, the user can register in one mode only, and the electronic equipment can acquire user registration information in multiple modes, so that the user identity can be identified based on any one or more modes in the multiple modes; the multi-mode registration process is simplified, and the convenience of user identity recognition is improved.

The electronic equipment actively acquires the user registration information of other modalities after the user registers in one modality, and the user registration information of multiple modalities acquired by the electronic equipment is automatically associated with the same user. When the electronic equipment authenticates the identity of the user, a multi-mode combined authentication mode can be adopted, so that the accuracy of user identity identification is improved.

The multi-modal identity recognition method provided in the embodiment of the present application may be applied to electronic devices such as a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a Personal Digital Assistant (PDA), a wearable device (such as a smart watch, smart glasses, or a smart helmet), a virtual reality device, a smart home device, and a vehicle-mounted computer, and the embodiment of the present application does not limit the electronic devices.

Referring to fig. 5, a schematic structural diagram of an electronic device 100 is shown.

As shown in fig. 5, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a Subscriber Identity Module (SIM) card interface 195, and the like.

It is to be understood that the illustrated structure of the present embodiment does not constitute a specific limitation to the electronic apparatus 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

The controller may be a neural center and a command center of the electronic device 100, and is a decision maker instructing each component of the electronic device 100 to work in coordination according to instructions. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution.

A memory may also be provided in processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Avoiding repeated accesses reduces the latency of the processor 110, thereby increasing the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (I2C) interface, an integrated circuit built-in audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose input/output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and/or a Universal Serial Bus (USB) interface, etc.

The I2C interface is a bi-directional synchronous serial bus that includes a serial data line (SDA) and a Serial Clock Line (SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 may be coupled to the touch sensor, charger, flash, camera 193, etc. through different I2C bus interfaces, respectively. For example: the processor 110 may be coupled to the touch sensor via an I2C interface, such that the processor 110 and the touch sensor communicate via an I2C bus interface to implement touch functionality of the electronic device 100.

The I2S interface may be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 via an I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may communicate audio signals to the wireless communication module 160 via the I2S interface, enabling answering of calls via a bluetooth headset.

The PCM interface may also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled by a PCM bus interface. In some embodiments, the audio module 170 may also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to implement a function of answering a call through a bluetooth headset. Both the I2S interface and the PCM interface may be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communications. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 with the wireless communication module 160. For example: the processor 110 communicates with a bluetooth module in the wireless communication module 160 through a UART interface to implement a bluetooth function. In some embodiments, the audio module 170 may transmit the audio signal to the wireless communication module 160 through a UART interface, so as to realize the function of playing music through a bluetooth headset.

MIPI interfaces may be used to connect processor 110 with peripheral devices such as display screen 194, camera 193, and the like. The MIPI interface includes a Camera Serial Interface (CSI), a Display Serial Interface (DSI), and the like. In some embodiments, processor 110 and camera 193 communicate through a CSI interface to implement the capture functionality of electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal and may also be configured as a data signal. In some embodiments, a GPIO interface may be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface may also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transmit data between the electronic device 100 and a peripheral device. And the earphone can also be used for connecting an earphone and playing audio through the earphone.

It should be understood that the connection relationship between the modules according to the embodiment of the present invention is only illustrative, and is not limited to the structure of the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners or a combination of multiple interface connection manners in the above embodiments.

The charging management module 140 is configured to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 may also supply power to the electronic device through the power management module 141 while charging the battery 142.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may also be used to monitor parameters such as battery capacity, battery cycle count, battery state of health (leakage, impedance), etc. In some other embodiments, the power management module 141 may also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be disposed in the same device.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution including 2G/3G/4G/5G wireless communication applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a Low Noise Amplifier (LNA), and the like. The mobile communication module 150 may receive the electromagnetic wave from the antenna 1, filter, amplify, etc. the received electromagnetic wave, and transmit the electromagnetic wave to the modem processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modem processor, and convert the signal into electromagnetic wave through the antenna 1 to radiate the electromagnetic wave. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then passes the demodulated low frequency baseband signal to a baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 150 or other functional modules, independent of the processor 110.

The wireless communication module 160 may provide a solution for wireless communication applied to the electronic device 100, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (bluetooth, BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), and the like. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification on the signal, and convert the signal into electromagnetic waves through the antenna 2 to radiate the electromagnetic waves.

In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 and antenna 2 is coupled to wireless communication module 160 so that electronic device 100 can communicate with networks and other devices through wireless communication techniques. The wireless communication technology may include global system for mobile communications (GSM), General Packet Radio Service (GPRS), code division multiple access (code division multiple access, CDMA), Wideband Code Division Multiple Access (WCDMA), time-division code division multiple access (time-division code division multiple access, TD-SCDMA), Long Term Evolution (LTE), LTE, BT, GNSS, WLAN, NFC, FM, and/or IR technologies, etc. The GNSS may include a Global Positioning System (GPS), a global navigation satellite system (GLONASS), a beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and/or a Satellite Based Augmentation System (SBAS).

The electronic device 100 implements display functions via the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, video, and the like. The display screen 194 includes a display panel. The display panel may adopt a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-oeld, a quantum dot light-emitting diode (QLED), and the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, with N being a positive integer greater than 1.

The electronic device 100 may implement a shooting function through the ISP, the camera 193, the video codec, the GPU, the display 194, the application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when a photo is taken, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converting into an image visible to naked eyes. The ISP can also carry out algorithm optimization on the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, the electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

In the embodiment of the present application, the camera 193 may be used in a face recognition technology to collect a face image of a user. The camera 193 may also be used in iris recognition technology to capture an iris image of a user. The electronic device 100 may include one or more cameras 193; for example, the camera for collecting the face image and the camera for collecting the iris image may be the same camera or different cameras.

After the camera 193 collects a face image or an iris image, the ISP converts the collected image into a digital image signal and outputs the digital image signal to the DSP for processing. Thereafter, the processor 110 may perform face recognition or iris recognition based on the digital image signal.

The digital signal processor is used for processing digital signals, and can process digital image signals and other digital signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to perform fourier transform or the like on the frequency bin energy.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, and the like.

The NPU is a neural-network (NN) computing processor that processes input information quickly by using a biological neural network structure, for example, by using a transfer mode between neurons of a human brain, and can also learn by itself continuously. Applications such as intelligent recognition of the electronic device 100 can be realized through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to extend the memory capability of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music, video, etc. are saved in an external memory card.

The internal memory 121 may be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The storage data area may store data (such as audio data, phone book, etc.) created during use of the electronic device 100, and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like.

The electronic device 100 may implement audio functions via the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playing, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be disposed in the processor 110, or some functional modules of the audio module 170 may be disposed in the processor 110.

The speaker 170A, also called a "horn", is used to convert the audio electrical signal into an acoustic signal. The electronic apparatus 100 can listen to music through the speaker 170A or listen to a handsfree call.

The receiver 170B, also called "earpiece", is used to convert the electrical audio signal into an acoustic signal. When the electronic apparatus 100 receives a call or voice information, it can receive voice by placing the receiver 170B close to the ear of the person.

The microphone 170C, also referred to as a "microphone," is used to convert sound signals into electrical signals. When making a call or transmitting voice information, the user can input a voice signal to the microphone 170C by speaking the user's mouth near the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C to achieve a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may further include three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, perform directional recording, and so on.

In this embodiment, the microphone 170C may be configured to collect a registration voice and an authentication voice of a user, and convert a sound signal into an electrical signal. The microphone 170C transmits the voice signal to the processor 110 and the processor 110 performs voiceprint recognition.

The headphone interface 170D is used to connect a wired headphone. The headset interface 170D may be the USB interface 130, or may be a 3.5mm open mobile electronic device platform (OMTP) standard interface, a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The sensor module 180 may include a fingerprint sensor. The fingerprint sensor is used for collecting fingerprints. The electronic device 100 may perform fingerprint feature extraction on a fingerprint image collected by a fingerprint sensor. The electronic device 100 can perform fingerprint identification by using the collected fingerprint characteristics, so as to unlock the fingerprint, access the application lock, photograph the fingerprint, answer an incoming call by using the fingerprint, and the like.

The sensor module 180 may further include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

The keys 190 include a power-on key, a volume key, and the like. The keys 190 may be mechanical keys. Or may be touch keys. The electronic apparatus 100 may receive a key input, and generate a key signal input related to user setting and function control of the electronic apparatus 100.

The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration cues, as well as for touch vibration feedback. For example, touch operations applied to different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects. The motor 191 may also respond to different vibration feedback effects for touch operations applied to different areas of the display screen 194. Different application scenes (such as time reminding, receiving information, alarm clock, game and the like) can also correspond to different vibration feedback effects. The touch vibration feedback effect may also support customization.

Indicator 192 may be an indicator light that may be used to indicate a state of charge, a change in charge, or a message, missed call, notification, etc.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be brought into and out of contact with the electronic apparatus 100 by being inserted into the SIM card interface 195 or being pulled out of the SIM card interface 195. The electronic device 100 may support 1 or N SIM card interfaces, N being a positive integer greater than 1. The SIM card interface 195 may support a Nano SIM card, a Micro SIM card, a SIM card, etc. The same SIM card interface 195 can be inserted with multiple cards at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 may also be compatible with different types of SIM cards. The SIM card interface 195 may also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as communication and data communication. In some embodiments, the electronic device 100 employs esims, namely: an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

The embodiment of the present application provides a multi-modal identity recognition method, which may be applied to the electronic device 100 shown in fig. 5, and as shown in fig. 6, the method may include:

s601, the electronic device receives a registration operation of a first user in a first mode.

The first modality may be any one of a plurality of modalities supported by the electronic device. One mode of identification of electronic devices is referred to as a modality of identification. For example, the modalities of identification supported by the electronic device may include: fingerprint identification, face identification, voiceprint identification, iris identification and the like.

Illustratively, the electronic device supports four modalities, namely fingerprint recognition, face recognition, voiceprint recognition and iris recognition. The user may perform fingerprint registration, face registration, voice print registration, or iris registration on the electronic device. In the embodiment of the application, the first modality is used as fingerprint identification, and the user 1 performs fingerprint registration on a mobile phone as an example.

The method comprises the steps that the electronic equipment receives fingerprint registration operation of a first user and stores a registered fingerprint of the first user; the first user may be any one of the users. For example, the electronic device stores the registered fingerprint of the first user, may store an image of the registered fingerprint, or may store the fingerprint feature of the registered fingerprint, and the like. Illustratively, the user 1 registers a fingerprint on the mobile phone by using the method shown in fig. 1 to generate a registered fingerprint 1, and the mobile phone stores the registered fingerprint 1 of the user 1.

S602, the electronic device authenticates the first user based on the first mode.

For example, the first user performs fingerprint registration on the electronic device, and the electronic device may authenticate the first user based on a fingerprint identification manner.

Exemplarily, a user 1 inputs an authentication fingerprint on a mobile phone, the mobile phone acquires the authentication fingerprint of the user 1, and fingerprint feature extraction is performed on the authentication fingerprint; comparing the fingerprint characteristics of the authentication fingerprint with the stored fingerprint characteristics of the registered fingerprint; the handset determines that the authentication fingerprint is a fingerprint belonging to user 1.

S603, after the electronic equipment successfully authenticates the first user based on the first modality, the electronic equipment collects registration information of one or more second modalities of the first user.

In some embodiments, the electronic device completes an authentication process for the first user based on the first modality and determines that the authentication process is successful; registration information for one or more second modalities of the first user is collected. The second modality can be any one of multiple modalities supported by the electronic equipment, different from the first modality; the registration information of the second modality is information used for performing a registration process in the second modality, for example, the registration information of fingerprint recognition may be a fingerprint image, the registration information of face recognition may be a face image, the registration information of voiceprint recognition may be voice, and the registration information of iris recognition may be an iris image.

For example, the mobile phone completes a fingerprint authentication process; if the mobile phone confirms that the fingerprint authentication process is finished, opening a camera to collect a face image of a user performing fingerprint authentication; in some examples, the handset also turns on a microphone to capture the user's voice.

In some embodiments, the electronic device may also collect registration information of one or more second modalities of the user after the first user completes a registration process of the first modality. For example, if the mobile phone determines that the user 1 completes the fingerprint registration process on the mobile phone, the mobile phone opens the camera to collect the face information of the user performing fingerprint authentication; in some examples, the handset also turns on a microphone to capture the user's voice.

In some embodiments, the electronic device may preset a duration of time for which the second modality registration information is acquired once. For example, after the mobile phone completes a fingerprint authentication process, face information of the user can be collected within a first time period; collecting the voice of the user in a second time length; the first time period and the second time period may be the same or different.

In some embodiments, prior to the electronic device collecting registration information for one or more second modalities of the user, the user may be confirmed whether they agree to collect registration information for the respective modalities. If it is determined that the user agrees to acquire registration information for one or more second modalities, registration information for the user's second modality is acquired. In one example, the electronic device may confirm with the first user whether to agree to acquire registration information for the respective modalities prior to acquiring the second modality registration information for the first user for the first time. For example, after completing a registration process of a first user in a first modality, the electronic device may confirm whether the first user agrees to collect registration information of each modality; for another example, the electronic device may confirm whether the user agrees to collect registration information of each modality after completing a first authentication process of the first user based on the first modality.

Illustratively, after the handset completes the fingerprint registration process of the user 1, it is confirmed whether the user agrees to collect registration information of each modality.

As shown in fig. 7A, when the mobile phone receives a user's click operation on the "confirm" button 10b in the "newly create fingerprint" interface 107, it is determined that fingerprint registration is completed once, and the mobile phone displays a "security and privacy" interface. At the "security and privacy" interface, a prompt 701 may be displayed, and the user may confirm or disapprove the acquisition of registration information for each modality at prompt 701. Prompt box 701 may include prompt information 702 for displaying prompt information to collect registration information for each modality. The prompt box 701 may further include an "ok" button 703, where the "ok" button 703 is used to trigger the handset to confirm that the user agrees to acquire registration information of each modality; prompt box 701 may also include a "cancel" button 704, where "cancel" button 704 is used to trigger the handset to confirm that the user disagrees with collecting registration information for each modality. The mobile phone may receive a click operation of the "ok" button 703 by the user, and confirm that the user agrees to acquire registration information of each modality in response to the click operation of the "ok" button 703 by the user. The mobile phone can receive the click operation of the user on the cancel button 704, and in response to the click operation of the user on the cancel button 704, the mobile phone confirms that the user does not agree to acquire the registration information of each modality.

In some examples, the handset may enable the user to select the kind of personal information that the user is authorized to collect. For example, a user may authorize a cell phone to collect one type of personal information, or a user may authorize a cell phone to collect multiple types of personal information. Illustratively, with continued reference to fig. 7A, in one implementation, in response to the user clicking on the "ok" button 703, the handset displays a prompt box 705 on the "security and privacy" interface, where the user may select the type of personal information authorized for collection at the prompt box 705. For example, the prompt box 705 includes prompt information 706, and the prompt information 706 is used for displaying the category of personal information for the user to select. Referring to fig. 7A, the prompt 706 includes four options, i.e., "fingerprint", "face", "voice", and "iris". The user may select whether to authorize the collection of personal information indicated by each option by clicking on a checkbox for that option. The prompt box 705 may also include an "ok" button 707, the "ok" button 707 being used to trigger the handset to confirm the kind of personal information selected by the user. The cellular phone may receive a click operation of the "ok" button 707 by the user, and in response to the click operation of the "ok" button 707 by the user, the cellular phone confirms the kind of personal information selected by the user. Further, the handset may return to displaying the "security and privacy" interface.

In another example, as shown in fig. 7B, when the mobile phone receives a user click operation on the "determine" button 10B in the "newly create fingerprint" interface 107, it is determined that fingerprint registration is completed, and the mobile phone displays a "security and privacy" interface. At the "security and privacy" interface, a prompt box 708 may be displayed. The user may select the type of personal information authorized to be collected at prompt box 708; the mobile phone can acquire the type of the personal information authorized to be acquired and selected by the user; and according to the type of the personal information authorized to be acquired selected by the user, confirming that the user agrees or disagrees to acquire the registration information of each modality. The prompt box 708 may include prompt information 709 for displaying the categories of personal information for selection by the user. The hint information 709 includes four options, fingerprint, face, voice, and iris. The user may select whether to authorize the collection of personal information indicated by each option by clicking on a checkbox for that option. The prompt box 708 may also include an "ok" button 70a, the "ok" button 70a being used to trigger the handset to confirm the category of personal information selected by the user. The cellular phone may receive a click operation of the "ok" button 70a by the user, and in response to the click operation of the "ok" button 70a by the user, the cellular phone confirms the kind of personal information selected by the user. If one or more of the check boxes of the four options of the prompt information 709 is checked, the mobile phone confirms that the user agrees to collect the registration information of each modality and confirms the kind of personal information selected by the user in response to the click operation of the "ok" button 70a by the user; if the check boxes of the four options of the prompt information 709 are not checked, the user is confirmed not to agree to collect the registration information of each modality. Further, the handset may return to displaying the "security and privacy" interface.

Illustratively, after the handset completes the first fingerprint authentication process of the user 1, it is confirmed whether the user agrees to collect registration information of each modality.

As shown in fig. 7C, when the mobile phone is in a blank screen state, and receives an operation of performing fingerprint authentication by the user, and the mobile phone determines that the fingerprint authentication process is successful, the desktop of the mobile phone is displayed. On the desktop of the mobile phone, a prompt box 701 can be displayed, and the user can confirm or disagree with collecting registration information of each modality in the prompt box 701. In one implementation, in response to the user clicking the "ok" button 703 in the prompt box 701, the mobile phone may further display a prompt box 705 on the desktop, and the user may select a category of personal information authorized to be collected in the prompt box 705. Further, after confirming the type of the personal information selected by the user, the mobile phone may return to the display desktop.

In another example, as shown in fig. 7D, when the mobile phone is in a blank screen state and receives an operation of performing fingerprint authentication by a user, the mobile phone displays a desktop of the mobile phone if it determines that the fingerprint authentication process is successful. A prompt box 708 can be displayed on the desktop of the mobile phone, and the user can select the type of personal information authorized to be collected in the prompt box 708; the mobile phone can acquire the type of the personal information authorized to be acquired and selected by the user; and according to the type of the personal information authorized to be acquired selected by the user, confirming that the user agrees or disagrees to acquire the registration information of each modality. Further, after confirming that the user agrees or disagrees with collecting the registration information of each modality, the mobile phone may return to the display desktop.

The electronic device acquires registration information of the second modality of the first user if the user agrees to acquire registration information of one or more second modalities. Illustratively, the handset determines that the user is authorized to capture fingerprint, face and voice information. After finishing the fingerprint registration process of the user 1, the mobile phone opens a camera to acquire a face image of the user 1; the microphone is also turned on to pick up the voice of the user 1.

For example, the mobile phone may acquire a face image of the user 1 through a camera of the mobile phone within a preset first time period; optionally, the mobile phone may further perform preprocessing (e.g., image enhancement) on the acquired face image.

Within the first duration, the mobile phone may acquire the face image for multiple times to acquire multiple samples of the face image of the user 1. Furthermore, the mobile phone can screen a plurality of samples of the face image. Optionally, the multiple samples may be face images directly acquired by a mobile phone; optionally, the plurality of samples may also be preprocessed face images. For example, screening multiple samples of a face image may include: and deleting the incomplete face image in the sample, deleting the side face image in the sample and the like.

In an implementation manner, the mobile phone may further screen a plurality of samples of the face image of the user 1 according to the stored information of the user 1. For example, if the mobile phone stores the voiceprint information of the user 1 and determines that the user 1 is a male according to the voiceprint information of the user 1, the face image of the female in the sample may be deleted.

For example, the mobile phone may collect the voice of the user 1 through the microphone of the mobile phone within the preset second time period; optionally, the mobile phone may further perform preprocessing (e.g., noise reduction processing, speech enhancement, etc.) on the collected speech.

In the second duration, the mobile phone can collect voice for multiple times to obtain multiple samples of the voice of the user 1. Further, the mobile phone can screen a plurality of samples of the voice. Optionally, the multiple samples may be voices directly collected by the mobile phone; optionally, the plurality of samples may also be preprocessed speech. For example, screening multiple samples of speech may include: and deleting the voice with the signal-to-noise ratio lower than a preset value in the sample, deleting the voice with the user gender inconsistent in the sample, and the like.

In some embodiments, registration information for one or more second modalities of the first user may be collected after each successful authentication of the first user based on the first modality before successful registration of the second modality of the first user on the electronic device.

For example, the first modality is fingerprint identification, and the user 1 successfully completes the fingerprint registration process on the mobile phone. The mobile phone determines that the user authorizes to collect the fingerprint, the face and the voice information, and then the mobile phone can open the camera to collect the face image of the user 1 after the user 1 is successfully authenticated by adopting fingerprint identification each time; the microphone is also turned on to pick up the voice of the user 1. In this way, the mobile phone can acquire multiple samples of the face image of the user 1 and multiple samples of the voice of the user 1. Exemplarily, after the fingerprint authentication of the user 1 is successful for the first time, the mobile phone acquires 10 face image samples of the user 1; after the fingerprint authentication of the user 1 is successful for the second time, acquiring 20 face image samples of the user 1; after the fingerprint authentication of the user 1 is successful for the third time, acquiring 50 face image samples of the user 1; after the fingerprint authentication of the user 1 is successful for the fourth time, acquiring face image samples of 40 users 1; thus, after 4 times of acquisition, the mobile phone acquires 120 face image samples of the user 1.

S604, the electronic device completes the registration process of the first user in the second mode according to the acquired registration information of the first user in the second mode.

In some embodiments, the electronic device obtains registration information of the second modality of the first user, and if the registration information is determined to reach a preset first standard, the registration process of the first user in the second modality is completed according to the registration information.

For example, the first criterion may include that the number of samples taken is greater than a first threshold (e.g., 100); for example, the first criterion may include that a resolution of at least one of the acquired samples reaches a second threshold.

For example, the mobile phone acquires registration information (120 samples of the face image) for face recognition of the user 1, and determines that the number of the acquired samples of the face image of the user 1 is greater than the first threshold (100), and then completes the face registration process of the user 1 according to the acquired registration information for face recognition.

In an implementation manner, the electronic device may train a preset algorithm model of the second modality by using the obtained multiple samples of the second modality of the first user, so as to obtain the trained algorithm model. And the electronic equipment extracts the features of the second mode of the first user according to the trained algorithm model of the second mode, stores the extracted features and completes the registration process of the first user in the second mode. The specific method for training the preset algorithm model by the electronic device through the multiple samples and the specific method for extracting the features according to the preset algorithm model can refer to the description of the conventional technology, and the embodiment of the application is not repeated.

For example, the mobile phone trains a preset algorithm model for face recognition by using 120 samples of the acquired face image of the user 1 to obtain a trained face recognition algorithm model; the mobile phone extracts the face features of the user 1 according to the trained face recognition algorithm model, and stores the face features of the user 1, namely, the face registration process of the user 1 is completed.

In some embodiments, after the electronic device completes the registration process of the first user in the second modality, a registration success prompt message may be further displayed on the display interface; in some embodiments, the electronic device may also determine that the user agrees to enable the manner of identification of the second modality.

Illustratively, as shown in fig. 8, the mobile phone determines that the face registration process of the user 1 is completed, and displays a registration success prompt box 801 on the desktop. The registration success prompt box 801 may include registration success prompt information 802 "is the mobile phone successfully registered your face information, enabled? The "registration success prompt box 801 may further include a" yes "button 803 and a" no "button 804; the "yes" button 803 is used to trigger enabling of face recognition by user 1, and the "no" button 804 is used to trigger disabling of face recognition by user 1. The mobile phone may receive a user click operation on the "yes" button 803, and in response to the user click operation on the "yes" button 803, the mobile phone enables face recognition of the user 1. The mobile phone may receive a user's clicking operation on the "no" button 804, and in response to the user's clicking operation on the "no" button 804, the mobile phone does not enable the face recognition of the user 1.

It should be noted that fig. 8 exemplifies that the registration success prompt box 801 is displayed on the desktop. In practical applications, the registration success prompt box 801 may be displayed on another interface of the mobile phone, for example, the mobile phone may display the registration success prompt box 801 on a current display interface of the mobile phone when it is determined that the face registration process of the user 1 is completed.

Authentication of the first user in the second modality is supported if the electronic device determines that the user agrees to enable the means for identification of the second modality. For example, after the mobile phone completes the face registration process of the user 1, it is determined that the user 1 agrees to enable face recognition, and then the user 1 may be authenticated by using the face recognition. For example, after the mobile phone completes the voiceprint registration process of the user 1, and determines that the user 1 agrees to enable voiceprint recognition, the user 1 may be authenticated by using the voiceprint recognition.

In this way, the user 1 performs a registration process (fingerprint registration) in one modality on the mobile phone, the mobile phone actively acquires registration information (face image and voice) in two second modalities of the user 1, and then completes face registration and voiceprint registration of the user 1, and the mobile phone can authenticate the user 1 by using one or more of the three modalities (fingerprint identification, face identification and voiceprint identification).

According to the multi-mode identity recognition method provided by the embodiment of the application, a user registers in one mode on the electronic equipment, and the electronic equipment can support multi-mode identity recognition, so that a multi-mode registration process is simplified, and the convenience of user identity recognition is improved. Moreover, after the user registers in one mode, the electronic device actively acquires the registration information of other modes of the user, and the registration information of multiple modes acquired by the electronic device is automatically associated with the same user.

After the registration process of the first user in the multiple modalities is completed on the electronic device, the electronic device may authenticate the first user based on one or more of the multiple modalities. Illustratively, the user 1 registers a fingerprint on the mobile phone by using the method shown in fig. 1, and completes fingerprint registration. After the mobile phone successfully authenticates the user 1 based on the fingerprint registration each time, the face image and the voice of the user 1 are collected, and the face registration and the voiceprint registration of the user 1 are completed according to the collected face image and the voice of the user 1. The handset may authenticate the user 1 based on one or more of fingerprint recognition, face recognition and voiceprint recognition.

In one implementation, the electronic device may authenticate the first user based on any of a plurality of modalities. For example, the mobile phone may authenticate the user 1 based on fingerprint recognition, face recognition or voiceprint recognition.

In one implementation, after the electronic device fails to authenticate the first user based on one modality, the electronic device may authenticate the first user again by using another modality. For example, if the mobile phone fails to authenticate the user 1 based on fingerprint identification, the camera is opened to collect a face image of the user 1, and the user 1 is authenticated based on face identification according to the collected face image of the user 1. Therefore, the first user is authenticated based on multiple modes, and the accuracy of user identity identification can be improved.

In one implementation, after the electronic device fails to authenticate the first user based on one modality, the electronic device may authenticate the first user by using a combination of modalities. For example, if the mobile phone fails to authenticate the user 1 based on fingerprint identification, the camera is opened to collect a face image of the user 1; and according to the fingerprint image of the user 1 and the face image of the user 1, the user 1 is authenticated based on the combination of fingerprint identification and face identification.

For example, the confidence level may be used to characterize the matching degree between the features of the authentication information and the features of the registration information, i.e. the credibility of the authentication information belonging to a user; the confidence may be a score, the higher the confidence level; it will be appreciated that confidence may also be described as a confidence probability, a confidence score, and the like.

The mobile phone calculates the confidence 1 that the fingerprint image belongs to the user 1 according to the acquired fingerprint image, and if the mobile phone determines that the confidence 1 is smaller than a preset first confidence threshold, the mobile phone fails to authenticate the user 1 based on fingerprint identification. The mobile phone also calculates the confidence 2 that the face image belongs to the user 1 according to the acquired face image; the mobile phone weights and sums the confidence 1 and the confidence 2 to obtain a confidence 3; and if the confidence coefficient 3 is determined to be larger than a preset second confidence threshold value, determining that the joint authentication on the user 1 is successful.

Therefore, the first user is authenticated based on the multi-mode combination, and the accuracy of user identity identification can be improved.

The embodiment of the present application further provides a multi-modal identity recognition method, which may be applied to the electronic device 100 shown in fig. 5, and as shown in fig. 9, the method may include:

s901, the electronic device authenticates the first user based on the first mode.

On the electronic device, the registration process of the first user in the first modality has been completed. The electronic device may authenticate the first user based on the first modality.

Alternatively, the first user may be actively registered on the electronic device for the first modality. For example, the first modality is a voiceprint recognition modality. The user 1 performs voiceprint registration on the mobile phone by using the method shown in fig. 3, and the mobile phone can authenticate the user 1 based on a voiceprint recognition mode.

Optionally, the electronic device may collect registration information of the first user in the first modality, and complete a registration process of the first user in the first modality. For example, the first modality is a voiceprint recognition modality. After the mobile phone successfully authenticates the user 1 by using the fingerprint authentication each time, acquiring the voice information of the user 1, and completing voiceprint registration according to the acquired voice information of the user 1; the handset can authenticate the user 1 based on the voiceprint recognition mode.

Illustratively, user 1 initiates a voice command to the handset: "play a song. The mobile phone acquires the voice of the user 1 and performs voiceprint feature extraction on the voice. The mobile phone compares the voiceprint features of the collected voice with the stored voiceprint features of the registered voice, and if the collected voice is determined to belong to the voice of the user 1, the voiceprint authentication is successful. The handset may execute playing a song.

S902, after the electronic device successfully authenticates the first user based on the first modality, the electronic device collects registration information of the first modality and one or more second modalities of the first user.

On the electronic device, the registration process of the first user in the second modality has been completed. The electronic device may authenticate the first user based on the second modality. The second modality is different from the first modality, and the registration information of the second modality and the registration information of the first modality are associated to the same user (first user).

Alternatively, the first user may be actively registered on the electronic device for the second modality. For example, the second modality is a face recognition mode. The user 1 performs face registration on the mobile phone by using the method shown in fig. 2, and the mobile phone can authenticate the user 1 based on a face recognition mode.

Optionally, the electronic device may collect registration information of the first user in the second modality, and complete a registration process of the first user in the second modality. For example, the second modality is a face recognition mode. After the mobile phone successfully authenticates the user 1 by using a fingerprint identification mode each time, acquiring a face image of the user 1, and completing face registration according to the acquired face image of the user 1; the mobile phone can authenticate the user 1 based on a face recognition mode.

After the electronic device successfully authenticates the first user based on the first modality, the electronic device collects registration information of the first modality and one or more second modalities of the first user. In one implementation, after the electronic device successfully authenticates the first user based on the first modality, registration information of the first modality and the one or more second modalities of the first user is collected within a preset time period. For example, after the mobile phone successfully authenticates the user 1 based on voiceprint recognition each time, within a preset time period, the microphone is opened to collect the voice of the user 1, and the camera is opened to collect the face image of the user 1.

In some embodiments, prior to acquiring registration information for a first modality and one or more second modalities, the electronic device may confirm to the user whether to approve acquisition of registration information for the respective modalities. If it is determined that the user agrees to acquire registration information for each modality, registration information for a first modality of the user and one or more second modalities is acquired. In one example, the electronic device may confirm to the user whether to approve acquisition of registration information for each modality before acquiring registration information for a first modality of the first user and one or more second modalities for a first time. Fig. 7C and 7D may be referred to as a display interface for the electronic device to confirm whether the user agrees to collect registration information of each modality, which is not described herein again.

The electronic device determines that the user agrees to acquire the registration information of the first modality and the one or more second modalities, and acquires the registration information of the first modality of the first user and the one or more second modalities after the electronic device successfully authenticates the first user based on the first modality each time.

Exemplarily, after the mobile phone successfully authenticates the user 1 based on voiceprint recognition each time, within a preset time length, a microphone is opened to collect voice, and if the collected voice is determined to belong to the user 1, a plurality of samples of the voice of the user 1 are obtained; and the mobile phone opens the camera to collect the face image, and if the collected face image is determined to belong to the user 1, a plurality of samples of the face image of the user 1 are obtained. Optionally, the mobile phone may pre-process the collected voice, and take the pre-processed voice as a voice sample; optionally, the mobile phone may pre-process the collected face image, and use the pre-processed face image as a sample of the face image; this is not limited in the examples of the present application.

In an implementation manner, after the mobile phone collects samples of each modality each time, the mobile phone can screen the samples and delete unqualified samples. For a specific screening method, reference may be made to the description of the foregoing embodiments, which are not repeated herein.

Illustratively, the mobile phone determines that the user authorizes to acquire face and voice information, after the mobile phone successfully authenticates the user 1 based on voiceprint recognition each time, the microphone is turned on to acquire the voice of the user 1, and the camera is turned on to acquire a face image of the user 1. In this way, the mobile phone can acquire multiple samples of the voice of the user 1 and multiple samples of the face image of the user 1. Exemplarily, after the mobile phone successfully authenticates the voiceprint of the user 1 for the first time, 8 voice samples of the user 1 are collected, and 10 face image samples of the user 1 are collected; after the voiceprint authentication of the user 1 is successful for the second time, 10 voice samples of the user 1 are collected, and 15 face image samples of the user 1 are collected; after the voiceprint authentication of the user 1 is successful for the third time, 15 voice samples of the user 1 are collected, and 12 face image samples of the user 1 are collected; after the voiceprint authentication of the user 1 is successful for the fourth time, 20 voice samples of the user 1 and 18 face image samples of the user 1 are collected. After 4 times of acquisition, the mobile phone acquires 53 voice samples of the user 1 and 55 face image samples of the user 1. After the voiceprint authentication of the user 1 is successful in the fifth time, acquiring 16 voice samples of the user 1 and acquiring 10 face image samples of the user 1; at this time, the mobile phone acquires 65 face image samples of the user 1.

In some embodiments, the electronic device may also authenticate the first user based on a second modality. The multi-modal identification method provided by the embodiment of the application may further include S903.

S903, after the electronic device successfully authenticates the first user based on one second modality, the electronic device collects registration information of the first modality and one or more second modalities of the first user.

In one implementation, after the electronic device successfully authenticates the first user based on one second modality each time, the registration information of the first modality and the one or more second modalities of the first user is collected within a preset time period. For example, the first modality is voiceprint recognition and the second modality is face recognition. For example, after the mobile phone successfully authenticates the user 1 based on face recognition, a microphone is opened within a preset time length to collect voice, and if the collected voice is determined to belong to the user 1, a plurality of samples of the voice of the user 1 are obtained; and the mobile phone opens the camera to collect the face image, and if the collected face image is determined to belong to the user 1, a plurality of samples of the face image of the user 1 are obtained.

In some examples, the electronic device may authenticate the first user based on the second modality one or more times after authenticating the first user based on the first modality a plurality of times. After the electronic device successfully authenticates the first user based on the first modality each time, and after the electronic device successfully authenticates the first user based on one second modality each time, the electronic device collects registration information of the first modality of the first user and one or more second modalities within a preset time length.

In other examples, the electronic device may authenticate the first user based on the second modality one or more times between authenticating the first user based on the first modality a plurality of times. After the electronic device successfully authenticates the first user based on the first modality each time, and after the electronic device successfully authenticates the first user based on one second modality each time, the electronic device collects registration information of the first modality of the first user and one or more second modalities within a preset time length.

Illustratively, the handset determines that the user is authorized to capture facial and voice information. After the mobile phone successfully authenticates the user 1 based on voiceprint recognition each time, the microphone is opened to collect voice of the user 1, and the camera is opened to collect a face image of the user 1. After the mobile phone successfully authenticates the user 1 based on face recognition each time, the microphone is opened to collect voice of the user 1, and the camera is opened to collect face images of the user 1. In this way, the mobile phone can acquire multiple samples of the voice of the user 1 and multiple samples of the face image of the user 1.

In one example, the mobile phone authenticates the user 1 based on voiceprint recognition, and after the voiceprint authentication of the user 1 is successful for the first time, 8 voice samples of the user 1 and 10 face image samples of the user 1 are acquired; after the voiceprint authentication of the user 1 is successful for the second time, 10 voice samples of the user 1 are collected, and 15 face image samples of the user 1 are collected. Then, the mobile phone authenticates the user 1 based on face recognition, and after the face authentication of the user 1 is successful for the first time, 15 voice samples of the user 1 and 12 face image samples of the user 1 are collected; after the face authentication of the user 1 is successful for the second time, 20 voice samples of the user 1 are collected, and 18 face image samples of the user 1 are collected. After 4 times of acquisition, the mobile phone acquires 53 voice samples of the user 1 and 55 face image samples of the user 1. After the face authentication of the user 1 is successful for the third time, acquiring 16 voice samples of the user 1 and acquiring 10 face image samples of the user 1; at this time, the mobile phone acquires 65 face image samples of the user 1.

In another example, the mobile phone authenticates the user 1 based on voiceprint recognition, and after the voiceprint authentication of the user 1 is successful for the first time, 8 voice samples of the user 1 are collected, and 10 face image samples of the user 1 are collected. Then, the mobile phone authenticates the user 1 based on face recognition, and after the face authentication of the user 1 is successful for the first time, 10 voice samples of the user 1 and 15 face image samples of the user 1 are collected; after the face authentication of the user 1 is successful for the second time, 15 voice samples of the user 1 are collected, and 12 face image samples of the user 1 are collected. Then, the mobile phone authenticates the user 1 based on voiceprint recognition, and after the voiceprint authentication of the user 1 is successful for the second time, 20 voice samples of the user 1 and 18 face image samples of the user 1 are collected. After 4 times of acquisition, the mobile phone acquires 53 voice samples of the user 1 and 55 face image samples of the user 1. Then, the mobile phone authenticates the user 1 based on face recognition, and after the face authentication of the user 1 is successful for the third time, 16 voice samples of the user 1 and 10 face image samples of the user 1 are collected; at this time, the mobile phone acquires 65 face image samples of the user 1.

S904, the electronic equipment updates the algorithm model of the first mode according to the acquired registration information of the first mode of the first user; and the electronic equipment updates the algorithm model of the second mode according to the acquired registration information of the second mode of the first user.

In some embodiments, the electronic device obtains registration information of a first modality of the first user, and updates the algorithm model of the first modality according to the registration information of the first modality if it is determined that the registration information of the first modality reaches a preset update standard of the first modality.

For example, the preset update criteria of the first modality may include: the number of samples of the first modality acquired is greater than a first value (e.g., 50).

Illustratively, after 4 times of collection, the mobile phone obtains 53 voice samples of the user 1, and if it is determined that the number of the obtained voice samples of the user 1 is greater than a first value (50), the algorithm model of the voiceprint recognition is updated according to the obtained registration information of the voiceprint recognition. For example, the mobile phone trains the voiceprint recognition algorithm model by using 53 acquired voice samples of the user 1 to acquire an updated algorithm model. The specific method for training the voiceprint recognition algorithm model by using the voice sample can refer to the description in the conventional technology, and is not described herein again.

In some embodiments, the electronic device obtains registration information of a second modality of the first user, and updates the algorithm model of the second modality according to the registration information of the second modality if it is determined that the registration information of the second modality reaches a preset update standard of the second modality.

Exemplary, preset update criteria of the second modality may include: the number of samples of the second modality acquired is greater than a second value (e.g., 60). The first value and the second value may be the same or different.

Illustratively, after 5 times of acquisition, the mobile phone acquires 65 face image samples of the user 1, and determines that the number of the acquired face image samples of the user 1 is greater than a second value (60), the algorithm model of the face recognition is updated according to the acquired registration information of the face recognition. For example, the mobile phone trains the algorithm model for face recognition by using the acquired 65 face image samples of the user 1 to acquire an updated algorithm model. The specific method for training the algorithm model for face recognition by using the face image sample may refer to the description in the conventional technology, and is not described herein again.

According to the multi-modal identity recognition method provided by the embodiment of the application, after the electronic equipment successfully authenticates a user based on one mode, the registration information of multiple modes of the user can be collected, and the algorithm model of each mode is updated according to the collected registration information of multiple modes of the user. Therefore, the algorithm models of all the modes can be continuously updated according to a large number of samples of the registered user on the electronic equipment, and therefore the accuracy of multi-mode identity recognition is improved.

Referring to fig. 10, a flow chart of the multimodal identification method provided in the embodiment of the present application executed in an electronic device is shown.

As shown in fig. 10, the user registers a modality a in the electronic device, where the modality a may be one of recognition methods such as fingerprint recognition, face recognition, voiceprint recognition, iris recognition, and the like. The electronic device authenticates the user based on modality a. And after the authentication is successful, the electronic equipment acquires registration information of other modalities. For example, the electronic device may take a picture to obtain a face image of the user after the user is successfully authenticated based on fingerprint identification; for example, the electronic device may record the user after successfully authenticating the user based on face recognition and if detecting the lip movement of the user; for example, the electronic device may take a picture to obtain a face image of the user after the user is successfully authenticated based on iris recognition. Further, the electronic device may further perform sample information evaluation on the collected registration information of other modules, for example, filter the registration information of the user who is registered with modality a in the collected registration information. Further, the electronic device may adopt the screened sample of each mode to train the algorithm model of the corresponding mode to form a registration model of each mode; i.e. the registration process of the user in each modality is completed. In some examples, after multiple modalities of a user are successfully registered on the electronic device, the algorithm models of the multiple modalities may be continuously updated during the use of any modality, so as to improve the accuracy of identity recognition. In some examples, information of registered successful modalities may be used to assist in new modality sample information evaluation, to assist in screening the acquired samples of new modalities. And a multi-modal registration model is formed on the electronic equipment, so that multi-modal identity recognition can be performed on the user. For example, the electronic device completes registration of the user in the modality B by collecting registration information of the modality B, and the electronic device may authenticate the user based on the modality a and/or the modality B.

It is understood that the electronic device includes hardware structures and/or software modules for performing the functions in order to realize the functions. Those of skill in the art will readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present application.

In the embodiment of the present application, the electronic device may be divided into the functional modules according to the method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, in the embodiment of the present application, the division of the module is schematic, and is only one logic function division, and there may be another division manner in actual implementation.

In the case of an integrated unit, fig. 11 shows a schematic diagram of a possible structure of the electronic device involved in the above-described embodiment. The electronic device 1100 includes: a processing unit 1101, a display unit 1102, and a storage unit 1103.

The processing unit 1101 is configured to control and manage the operation of the electronic device 1100. For example, the method may be used to perform registration and authentication processes of each modality, collect registration information of each modality, and perform a registration processing step using the collected registration information; the method can also be used for executing a processing step of updating the algorithm model of each modality according to the collected registration information; and/or other processes for the techniques described herein.

The display unit 1102 is configured to display an interface of the electronic device. For example, an interface may be used to display registration information that asks the user whether the user agrees to acquire one or more modalities; the method can also be used for displaying registration success prompt information; and/or other processes for the techniques described herein.

The storage unit 1103 is used to store program codes and data of the electronic device 1100. For example, it may be used to store registration information of each acquired modality, and the like.

Of course, the unit modules in the electronic device 1100 include, but are not limited to, the processing unit 1101, the display unit 1102, and the storage unit 1103. For example, an input unit or the like may also be included in the electronic apparatus 1100. The input unit is used for collecting voice input, human face image input, fingerprint image input and the like of a user.

The processing unit 1101 may be a processor or a controller, such as a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an application-specific integrated circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor may include an application processor and a baseband processor. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor may also be a combination of computing functions, e.g., comprising one or more microprocessors, DSPs, and microprocessors, among others. The display unit 1102 may be a display screen. The storage unit 1103 may be a memory. The input unit may include a microphone, a camera, a fingerprint sensor, and the like.

For example, the processing unit 1101 is a processor (such as the processor 110 shown in fig. 5), the display unit 1102 is a display screen (such as the display screen 194 shown in fig. 5, the display screen 194 may be a touch screen, a display panel and a touch panel may be integrated in the touch screen), and the storage unit 1103 may be a memory (such as the internal memory 121 shown in fig. 5). The input unit may include a microphone (such as the microphone 170C shown in fig. 5), a camera (such as the camera 193 shown in fig. 5), a fingerprint sensor (such as the fingerprint sensor in the sensor module 180 shown in fig. 5). The electronic device 1100 provided by the embodiment of the present application may be the electronic device 100 shown in fig. 5. Wherein the above-mentioned processors, memories, etc. may be coupled together, e.g. by a bus connection.

The embodiment of the present application further provides a computer storage medium, in which computer program codes are stored, and when the processor executes the computer program codes, the electronic device executes the relevant method steps in fig. 6 or fig. 9 to implement the method in the foregoing embodiment.

Embodiments of the present application also provide a computer program product, which when run on a computer causes the computer to execute the relevant method steps in fig. 6 or fig. 9 to implement the method in the above-described embodiments.

In addition, the electronic device 1100, the computer storage medium, or the computer program product provided in the embodiment of the present application are all used for executing the corresponding method provided above, and therefore, the beneficial effects achieved by the electronic device 1100, the computer storage medium, or the computer program product may refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Through the above description of the embodiments, it is clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or partially contributed to by the prior art, or all or part of the technical solutions may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

The above description is only an embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A multi-modal identity recognition method, comprising:

the method comprises the steps that the electronic equipment receives registration operation of a user in a first mode, wherein the first mode is any one of a plurality of identity recognition modes supported by the electronic equipment;

the electronic device receiving an authentication operation of the user in the first modality;

if the electronic equipment successfully authenticates the user based on the first modality, responding to the authentication operation of the user in the first modality, and acquiring registration information of one or more second modalities of the user by the electronic equipment;

and the electronic equipment completes the registration process of the user in one or more second modalities according to the registration information of the one or more second modalities of the user.

2. The method of claim 1, wherein the electronic device collects registration information for one or more second modalities of the user, comprising:

and the electronic equipment collects registration information of one or more second modalities of the user within a preset time length.

3. The method of claim 1 or 2, wherein prior to the electronic device acquiring registration information for one or more second modalities of the user, the method further comprises:

the electronic equipment receives confirmation operation of the user on acquisition of registration information of one or more second modalities of the user.

4. The method according to any one of claims 1-3, wherein after the electronic device receives a user registration operation in a first modality, the method further comprises:

and if the user is determined to be successfully registered in the first modality, responding to the registration operation of the user in the first modality, and acquiring the registration information of one or more second modalities of the user by the electronic equipment.

5. The method according to any one of claims 1-4, wherein the modalities include: fingerprint recognition, face recognition, voiceprint recognition or iris recognition.

6. A multi-modal identity recognition method, comprising:

the electronic equipment receives an authentication operation of a user in a first mode; the first mode is any one of a plurality of identity recognition modes supported by the electronic equipment;

if the electronic equipment successfully authenticates the user based on the first modality, responding to the authentication operation of the user in the first modality, and acquiring registration information of the first modality and one or more second modalities of the user by the electronic equipment;

the electronic equipment updates an algorithm model of a first modality according to registration information of the first modality of the user; and the electronic equipment updates the algorithm model of each second mode according to the registration information of each second mode.

7. The method of claim 6, wherein the electronic device collects registration information for a first modality of the user and one or more second modalities, comprising:

the electronic equipment collects registration information of a first modality and one or more second modalities of the user within a preset time.

8. The method of claim 6 or 7, wherein prior to the electronic device acquiring registration information of the first modality of the user and one or more second modalities, the method further comprises:

the electronic equipment receives confirmation operation of the user on acquisition of registration information of a first modality and one or more second modalities of the user.

9. The method according to any one of claims 6-8, wherein the modalities include: fingerprint recognition, face recognition, voiceprint recognition or iris recognition.

10. The method according to any one of claims 1-9, further comprising:

the electronic equipment receives the operation of fingerprint authentication of a user on the electronic equipment;

and if the electronic equipment successfully performs fingerprint authentication on the user, responding to the operation of performing fingerprint authentication on the electronic equipment by the user, and acquiring a face image of the user by the electronic equipment.

11. The method according to any one of claims 1-9, further comprising:

the electronic equipment receives input voice of a user;

the electronic equipment performs voiceprint authentication on the user according to the input voice of the user;

and if the electronic equipment successfully performs voiceprint authentication on the user, the electronic equipment acquires a face image of the user.

12. A graphical user interface, GUI, stored in an electronic device, the graphical user interface comprising:

in response to an electronic device successfully authenticating a user based on a first modality, the electronic device displaying a first GUI on a first interface;

the first GUI comprises first information, the first information is used for prompting to collect registration information of the user in at least one modality, and the modality comprises: fingerprint identification, face identification, voiceprint identification or iris identification;

the first GUI further comprises a first button and a second button, wherein the first button is used for triggering the electronic equipment to confirm that the user agrees to acquire registration information of the user in at least one modality; the second button is used for triggering the electronic equipment to confirm that the user does not agree to acquire the registration information of the user in at least one mode.

13. A graphical user interface, GUI, stored in an electronic device, the graphical user interface comprising:

in response to a user successfully registering in a first modality on an electronic device, the electronic device displaying a first GUI on a first interface;

14. A graphical user interface, GUI, according to claim 12 or 13, characterized in that the graphical user interface further comprises:

responding to the clicking operation of the first button by the user, and displaying a second GUI on the first interface by the electronic equipment; the second GUI includes a plurality of options, wherein each of the options corresponds to one of the modalities.

15. A GUI according to claim 12 or claim 13, further comprising a plurality of options in the first GUI, wherein each option corresponds to one of the modalities.

16. An electronic device, comprising:

the processor is used for receiving a registration operation of a user in a first modality, wherein the first modality is any one of a plurality of identification modes supported by the electronic equipment;

the processor is further configured to receive an authentication operation of the user in the first modality;

the input unit is used for collecting registration information of one or more second modalities of the user in response to the authentication operation of the user in the first modality if the processor determines that the electronic equipment successfully authenticates the user based on the first modality;

the processor is further configured to complete a registration process of the user in one or more second modalities according to registration information of the one or more second modalities of the user.

17. The electronic device of claim 16,

the processor is further configured to receive a confirmation operation of the user to collect registration information of one or more second modalities of the user.

18. An electronic device, comprising:

a processor for receiving an authentication operation of a user in a first modality; the first mode is any one of a plurality of identity recognition modes supported by the electronic equipment;

the input unit is used for collecting registration information of a first modality and one or more second modalities of the user in response to the authentication operation of the user in the first modality if the processor determines that the electronic equipment successfully authenticates the user based on the first modality;

the processor is further configured to update the algorithm model of the first modality according to registration information of the first modality of the user; and the electronic equipment updates the algorithm model of each second mode according to the registration information of each second mode.

19. A computer storage medium comprising computer instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1-11.

20. A computer program product, characterized in that it causes a computer to carry out the method according to any one of claims 1-11, when said computer program product is run on the computer.