JP7402239B2 - Face recognition method, neural network training method, face recognition device, electronic device and computer readable storage medium - Google Patents

Description

The present disclosure relates to the technical field of image processing, and particularly relates to a face recognition method, a neural network training method, an apparatus, and an electronic device.

With the rapid development of artificial intelligence and the vehicle industry, applying the latest artificial intelligence technology to mass-produced vehicles has already become the most promising direction in the market. Artificial intelligence products in demand in the current vehicle market include, but are not limited to, driver assistance systems, driver monitoring systems, vehicle operation management systems, etc. In order to meet these market needs, it is usually necessary to recognize the driver's face and to perform subsequent management and control based on this recognition.

The present disclosure provides technical means for facial recognition and neural network training.

In a first aspect, a face recognition method according to an embodiment of the present disclosure includes:
acquiring a first facial image with a first camera;
extracting a first facial feature of the first facial image;
The first facial features are compared with pre-stored second facial features extracted from the features of a second facial image acquired by a second camera that is a different type of camera from the first camera. , obtaining reference similarity;
determining whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity.

In the embodiment of the present disclosure, when performing face recognition, recognition of a first face image and a second face image acquired by different types of cameras can be realized. In implementing the embodiments of the present disclosure, the first face image and the second face image are acquired by different types of cameras, thereby applying the face recognition method according to the embodiments of the present disclosure to more scenes. This not only simplifies face recognition, but also improves the convenience of face registration by not limiting the number of cameras used during face registration.

In a second aspect, a neural network training method according to an embodiment of the present disclosure includes:
obtaining a first type of image sample and a second type of image sample that are taken by different types of cameras and include a face;
training a neural network based on the first type of image samples and the second type of image samples.

In embodiments of the present disclosure, by training a neural network using facial images captured by different types of cameras, the accuracy with which the neural network outputs facial features can be effectively improved, and facial When performing recognition, using the neural network to extract facial features can effectively improve the accuracy of facial recognition.

In a third aspect, a face recognition device according to an embodiment of the present disclosure includes:
a first acquisition unit that acquires a first face image with a first camera;
a first extraction unit that extracts a first facial feature of the first facial image;
The first facial features are compared with pre-stored second facial features extracted from the features of a second facial image acquired by a second camera that is a different type of camera from the first camera. , a contrast unit that obtains the reference similarity;
a determining unit for determining whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity.

In a fourth aspect, a neural network training device according to an embodiment of the present disclosure includes:
an acquisition unit for acquiring a first type of image sample and a second type of image sample taken by different types of cameras and including a face;
a training unit for training a neural network based on the first type of image samples and the second type of image samples.

In a fifth aspect, an electronic device according to an embodiment of the present disclosure includes:
a processor and a memory, the memory coupled to the processor and further storing program instructions, the processor configured to support the electronic device in performing the corresponding functions in the method of the first aspect. It is composed of

In a sixth aspect, an electronic device according to an embodiment of the present disclosure includes a processor and a memory, the memory is coupled to the processor and further stores program instructions, and the processor is configured to and configured to support performing the corresponding functions in the method of aspect 2.

In a seventh aspect, a face recognition system according to an embodiment of the present disclosure includes a neural network training device and a face recognition device, and the neural network training device is coupled to the face recognition device;
The neural network training device trains a neural network,
The face recognition device applies a neural network trained by the neural network training device.

In an eighth aspect, a computer readable storage medium according to an embodiment of the present disclosure stores instructions that, when executed in a computer, cause the computer to perform the methods described in each aspect above.

In a ninth aspect, a computer program comprising instructions according to an embodiment of the present disclosure causes the computer to perform the method described in each aspect above when the instructions are executed on a computer.

In the process of implementing the embodiments of the present disclosure, the applicant of the present disclosure often restricts the image type that is authenticated and the image type that is registered in conventional face recognition methods to be the same image type; For example, if RGB images are used in the registration process, RGB images must also be used in the authentication process, thus limiting the application of traditional facial recognition solutions in scenes involving multiple types of cameras. I discovered that. Embodiments of the present disclosure provide a facial image recognition solution for scenes involving multiple types of cameras, and implementations of embodiments of the present disclosure provide a facial image recognition solution for scenes involving multiple types of cameras; Implementation of the present disclosure by taking a face image as a base library feature, comparing the face feature of a first face image acquired by another type of camera with the base library feature, and realizing face recognition based on the comparison result. The face recognition method according to the example can be applied to more scenes, which not only makes face recognition easier, but also the camera used during face registration is not limited, improving the convenience of face registration.

In order to more clearly explain the technical means in the embodiments of the present disclosure or the background art, drawings that need to be used in the embodiments or background technology of the present disclosure will be described below.
3 is a flowchart of a face recognition method according to an embodiment of the present disclosure. 3 is a flowchart of a neural network training method according to an embodiment of the present disclosure. 1 is a schematic diagram of a training process according to an embodiment of the present disclosure; FIG. 1 is a schematic configuration diagram of a face recognition device according to an embodiment of the present disclosure. FIG. 3 is a schematic configuration diagram of another face recognition device according to an example of the present disclosure. FIG. 1 is a schematic configuration diagram of a neural network training device according to an embodiment of the present disclosure. FIG. 1 is a schematic configuration diagram of a training unit according to an example of the present disclosure. FIG. 2 is a schematic configuration diagram of another neural network training device according to an embodiment of the present disclosure. 1 is a schematic configuration diagram of an electronic device according to an example of the present disclosure.

In order to make the objects, technical means, and advantages of the present disclosure more clear, the present disclosure will be described in further detail with reference to the drawings below.

The terms "first", "second", etc. in the specification and claims of the present disclosure and the above drawings are used to distinguish different objects and are not intended to describe a particular order. Also, the terms "comprising" and "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or apparatus comprising a series of steps or units is not limited to the steps or units shown, but preferably further comprises steps or units not shown, or preferably: It may further include other steps or units specific to these processes, methods or apparatuses.

Referring to FIG. 1, FIG. 1 is a flowchart of a face recognition method according to an embodiment of the present disclosure, and the face recognition method can be applied to a face recognition device, and further can be applied to an electronic device, and the electronic device includes: It may include a server or a terminal device, and the server may include any type of server, such as a cloud server, and is not limited by the embodiments of this disclosure. The terminal device may include a mobile phone, a tablet computer, a desktop computer, an in-vehicle device, a driver status monitoring system, a ride management system, an automobile rental management system, an online vehicle dispatch management system, etc. The specific form of the terminal device is not uniquely limited. The application of the face recognition method to electronic equipment will be described below as an example.

As shown in FIG. 1, the face recognition method includes the following steps 101-104.

In step 101, a first facial image is acquired by a first camera.

In embodiments of the present disclosure, the first facial image may include an image captured by a first camera, or the first facial image may include video stream data captured by a first camera. The embodiment of the present disclosure does not limit the source of the first face image.

In embodiments of the present disclosure, the first camera may be a thermal camera or the first camera may be a visible light camera. When the first camera is a thermo camera, the second camera may be another camera different from the thermo camera, for example, the second camera may be a visible light camera. When the first camera is a visible light camera, the second camera may be another camera different from the visible light camera, for example, the second camera may be a thermo camera. In one example, the visible light camera may include a red green blue (RGB) camera. Thermal cameras may include infrared radiation (IR) cameras. The imaging of IR camera is not affected by the interference of environmental light, and there is no difference in quality regardless of whether it is daytime or night, whether it is sunny, cloudy or rainy, and whether it is different application scenes such as external roads or tunnels. Can collect images that are not large. RGB cameras are low in price and widely used, RGB cameras are installed in many terminals or scenes, and RGB images are also very popular and widely used. Therefore, in the car application scene, the on-vehicle camera may be an IR camera. This will improve the convenience and flexibility of registration by using RGB cameras for face registration, perform face recognition using IR cameras, and collect real-time images using in-vehicle cameras to improve the convenience and flexibility of registration. This makes it easy to perform processes such as unlocking, authority control, personnel/vehicle management, etc. Note that the above is only an example, and a specific implementation may further include other types of cameras, and will not be repeatedly described herein.

In one example, the first camera may be an external camera connected to an electronic device, or a built-in camera of the electronic device, etc., and embodiments of the present disclosure describe how the first camera The specific implementation form is not limited. Accordingly, in one example of the second camera, the first camera and the second camera may be different types of onboard cameras, i.e. the first facial image is acquired in the driving area of the vehicle by the onboard camera. It may be a face image that has been In addition, in a specific implementation, when the first camera and the second camera are built into various electronic devices, for example, the first camera may be built into the camera, or the first camera may be built into a mobile phone. The first camera and the second camera may be built into an in-vehicle device or the like, and the embodiments of the present disclosure do not uniquely limit the specific forms of the first camera and the second camera.

In one example, when the first camera is an in-vehicle camera, obtaining the first facial image with the first camera includes:
The method includes acquiring a first facial image by an on-vehicle camera, the first facial image including a facial image of a vehicle user of the vehicle.

In this embodiment, the vehicle may include a car, a light vehicle, a passenger car, a truck, a scheduled vehicle, a taxi, a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle or a vehicle with more than four wheels, a small vehicle, a vehicle-type robot, a radio-controlled model vehicle, etc. , embodiments of the present disclosure do not limit the specific type of vehicle.

In this example, vehicle users may include one or more of the following: a person who drives the vehicle, a person who rides the vehicle, a person who repairs the vehicle, a person who refuels the vehicle, and a person who controls the vehicle. The person who controls the vehicle may be a person who controls a radio-controlled model car, the person who refuels the vehicle may be a refueling worker, the person who repairs the vehicle may be an automobile repair worker, and the person who rides the vehicle may be a person who controls a radio-controlled model car. The person may be a person riding a taxi or a scheduled vehicle, etc., and the person driving the vehicle may be a driver, etc. Embodiments of the present disclosure do not limit the specific types of vehicle users described above.

In one example, if the vehicle users include a person driving the vehicle, embodiments of the present disclosure further provide a triggering condition of when the electronic device acquires the first facial image, e.g. The step of acquiring the first facial image is as follows:
If the trigger command is received, acquiring a first facial image by the in-vehicle camera;
Alternatively, a step of acquiring a first facial image using an on-vehicle camera while the vehicle is running;
Alternatively, the method includes the step of acquiring a first facial image using an on-vehicle camera when the traveling speed of the vehicle reaches a reference speed.

In this embodiment, the trigger command may be a trigger command received by the electronic device and input by a user, or may be a trigger command sent by another electronic device connected to the electronic device. The embodiments do not limit the origin and specific form of the trigger instruction.

In this embodiment, when the vehicle is running, it can be understood that when the vehicle is ignited, that is, when the electronic device detects that the vehicle has started running, the electronic device detects the user's ignition in the driving area of the vehicle. A facial image, ie, a first facial image, can be obtained.

In this embodiment, the reference speed is used to determine at what speed the vehicle travels when the electronic device acquires the first face image. Not limited. The reference speed may be set by a user, may be set by a device connected to an electronic device and measures the running speed of the vehicle, or may be set by an electronic device, and this embodiment is not limited to do not.

In the implementation of this embodiment, by setting the trigger condition and acquiring the first facial image, it is possible to identify the identity of the vehicle user and effectively improve the efficiency with which the electronic device performs facial recognition. can.

In step 102, a first facial feature of the first facial image is extracted.

In embodiments of the present disclosure, the electronic device may extract the first facial feature of the first facial image by any method, for example, the electronic device may extract the first facial feature of the first facial image using a feature point extraction algorithm. For the feature point extraction, susan operator feature extraction, harris operator feature extraction, sift feature extraction, neural network feature extraction, or the like can be used, but the method is not limited thereto. Further, for example, the electronic device extracts the first facial feature etc. by a facial feature extraction method based on geometric features or a facial feature extraction method based on template matching, and the embodiment of the present disclosure There is no limitation on how the first facial feature is extracted.

In one example, the electronic device may extract the first facial feature of the first face with a neural network, and the neural network may be a pre-trained neural network. The pre-trained neural network may be a neural network trained by an electronic device in an embodiment of the present disclosure, or may be a neural network trained by another device, and then a neural network trained by an electronic device in an embodiment of the present disclosure. The electronic device in the example may be a neural network acquired from the other device, and the embodiments of the present disclosure are not limited thereto.

In the embodiments of the present disclosure, the neural network may be designed to be stacked in a certain manner with network layers such as convolutional layers, nonlinear layers, pooling layers, etc., and the embodiments of the present disclosure do not limit the specific network structure. After designing the structure of the neural network, gradient error backpropagation is performed on the neural network designed using the supervised or weakly supervised method based on images with labeling information until the predetermined training completion condition of the neural network is met. The network parameters of the neural network may be adjusted by performing repeated training thousands or even tens of thousands of times. The embodiments of the present disclosure are not limited to specific training methods.

Feature extraction of a facial image by a neural network can realize output between terminals, for example, a first facial image is input to a pre-trained neural network, and the neural network extracts features from the first facial image. The feature map obtained by extraction is output, that is, the process of facial image feature extraction is realized between terminals. Facial feature extraction refers to performing facial feature extraction on some features of a face, and the facial features may also be referred to as facial features. As an example, facial feature extraction using a neural network may specifically involve extracting deep-level abstract features of a face using a deep neural network.

In one example, a first facial image is input to a pre-trained neural network, the neural network outputs a first facial feature of the first facial image, and the neural network includes a first facial image captured by a different type of camera; It is obtained by training based on a first type of image sample and a second type of image sample containing a face. In this disclosure, by training a neural network using image samples taken by two different types of cameras, the neural network can learn the feature extraction ability of different types of images, thereby training The face recognition ability for different types of images is realized based on the developed neural network.

In step 103, the first facial features are replaced with pre-stored second facial features extracted from the features of a second facial image acquired by a second camera that is a different type of camera from the first camera. Contrast and obtain reference similarity.

In embodiments of the present disclosure, the first facial image may be understood as a facial image that needs to undergo facial recognition, or that needs to be searched. It may be understood as a certain facial image. The second facial image may be understood as a facial image during face registration or may be understood as a facial image stored in an identity-based library. The identity base library stores identity information and facial features corresponding to each identity information. Hereinafter, an example will be described in which the first face image is a face image that requires face recognition, and the second face image is a face image at the time of face registration.

In the embodiments of the present disclosure, the first camera and the second camera belong to different types of cameras, that is, the first facial image and the second facial image can be understood as different types of facial images. For example, if the first facial image is an RGB facial image, the second facial image may be an IR facial image. Alternatively, the first face image is an IR face image and the second face image is an RGB face image. Alternatively, it may be another type of facial image, etc., and the embodiments of the present disclosure are not limited thereto. For specific descriptions of the first camera and the second camera, reference can be made to the previous embodiments, and they will not be described in detail one by one in this specification.

In an embodiment of the present disclosure, the reference similarity is the similarity where the first facial image belongs to the user who corresponds to the second facial image in the identity-based library, i.e., the reference similarity is the degree of similarity between the first facial image and the second facial image in the identity-based library. It is possible to indicate the degree of similarity in which the second facial images correspond to the same person. For example, an electronic device can obtain reference similarity by comparing facial features.

In one example, when the second facial image is an image for performing facial registration for a vehicle user, embodiments of the present disclosure further provide a method for obtaining facial features of the second facial image, For example, before comparing a first facial feature with a pre-stored second facial feature, the method shown in FIG.
acquiring a second facial image with a second camera;
extracting a second facial feature of the second facial image;
and storing a second facial feature of the second facial image.

The present embodiment does not limit the method of extracting the facial features of the second facial image. For example, the second facial features are extracted by extracting the facial features of the second facial image using a pre-trained neural network. You may obtain it. Alternatively, facial features of the second facial image may be extracted using, for example, a local binary patterns (LBP) method. Further, the facial features of the second facial image may be extracted by, for example, a sift feature extraction method, a facial feature extraction method based on geometric features, a facial feature extraction method based on template matching, or the like. Note that this embodiment does not uniquely limit how the facial features of the second facial image are extracted. In this embodiment, the second facial feature can be stored in the electronic device to provide security for facial recognition.

In step 104, it is determined whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity.

In embodiments of the present disclosure, when determining whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity, for example, the reference similarity and the similarity threshold (static (understood as a similarity threshold), and if the reference similarity is greater than or equal to the similarity threshold, then the first facial feature and the second facial feature are the same person. It can be determined that it corresponds to If the reference similarity is less than the similarity threshold, it may be determined that the first facial feature and the second facial feature correspond to different people.

In one example, embodiments of the present disclosure further provide a method for determining whether a first facial feature and a second facial feature correspond to the same person using a dynamic similarity threshold. For example, determining whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity may include:
determining whether the first facial feature and the second facial feature correspond to the same person based on a reference similarity, a reference false alarm rate, and a similarity threshold, where different false alarm rates correspond to different degrees of similarity; Corresponds to the threshold.

In this example, different false alarm rates correspond to different similarity thresholds, that is, there is a correspondence between false alarm rates and similarity thresholds. The embodiments of the present disclosure do not limit the similarity thresholds to which different false alarm rates specifically correspond. For example, the correspondence between the false alarm rate and the similarity threshold may be set by the user or may be set independently by the electronic device, and the embodiments of the present disclosure are not limited thereto. Note that the reference false alarm rate in the embodiment of the present disclosure is a false alarm rate determined by the electronic device, and is, for example, one false alarm rate determined by the electronic device from the correspondence between the false alarm rate and the similarity threshold.

For example, the relationship between the false alarm rate and the similarity threshold is as follows: if the false alarm rate is 1 in 10,000, then the similarity threshold is 0.7, and the false alarm rate is 1 in 100,000. , the similarity threshold may be 0.8, and if the false alarm rate is 1 in 1 million, the similarity threshold may be 0.9, and if the false alarm rate is 1 in 1 million, The similarity threshold may be 0.98. Accordingly, after determining the reference false alarm rate, the electronic device can determine the similarity threshold based on the reference false alarm rate, thereby determining the similarity threshold based on the obtained reference similarity and the determined similarity threshold. It may be determined whether the first facial feature and the second facial feature correspond to the same person. Note that embodiments of the present disclosure do not limit how the electronic device determines the reference false alarm rate; for example, the electronic device may determine the reference false alarm rate based on a determination instruction input by a user, or may determine the reference false alarm rate by other methods. You may decide.

In the implementation of this embodiment, determining whether the first facial feature and the second facial feature correspond to the same person by obtaining different similarity thresholds with different false alarm rates is based on a fixed similarity threshold. By avoiding face recognition solutions using a degree threshold method, the degree of similarity for determining the relationship between two face images can be dynamically determined, improving the accuracy of face recognition. .

In one example, embodiments of the present disclosure further provide a method for determining whether a first facial feature and a second facial feature correspond to the same person, e.g. The step of determining whether the facial feature and the second facial feature correspond to the same person includes:
determining a normalized reference similarity based on the reference similarity and the threshold information;
determining whether the first facial feature and the second facial feature correspond to the same person based on the normalized reference similarity.

In this embodiment, the threshold information is obtained based on the similarity of positive sample pairs, the similarity of negative sample pairs and different preset false alarm rates, and the positive sample pairs and negative sample pairs are of the first type. image and a second type of image, each positive sample pair includes two images, and the faces in the two images correspond to the same person, and each negative sample pair includes two images. The faces in the two images correspond to different people, and the similarity between the positive sample pair and the negative sample pair is determined by a pre-trained neural network. In one example, the threshold information may include a first threshold and a second threshold, whereby the electronic device normalizes the reference similarity based on the first threshold and the second threshold. a reference similarity between a first threshold and a second threshold; closest to the threshold of 2. The first type of image and the second type of image are each obtained by a different type of camera.

In this embodiment, the first threshold value and the second threshold value may be determined from the threshold information based on the reference similarity, so that the electronic device can determine the regularity based on the first threshold value and the second threshold value. The standardized reference similarity can be determined. The normalized reference similarity is the final similarity for determining whether the first facial feature and the second facial feature correspond to the same person. For example, if the reference similarity value is between T(n-1) (e.g., the first threshold) and T(n) (e.g., the second threshold), then the normalized reference similarity The degree can be determined to be 0.3+(n-1) ^/10+0.1* (reference similarity-T(n-1)) ^* (T(n)-T(n-1)). Note that the above is only an example of the normalization method, and should not be understood as limiting the present embodiment.

Note that, after determining the normalized reference similarity, the electronic device in this embodiment uses a fixed similarity threshold to identify a person whose first facial feature and second facial feature are the same. You may decide whether or not to comply with the above. In one example, the electronic device further uses a dynamic similarity threshold (i.e., obtains different similarity thresholds based on different false alarm rates) to match the first facial feature and the second facial feature to the same person. You may decide whether to respond or not.

By implementing this embodiment, it is possible to further improve the accuracy of the degree of similarity for determining whether or not the first facial feature and the second facial feature correspond to the same person, thereby improving the accuracy of facial recognition. can.

In one example, the facial images in the identity-based library may be plural, i.e., the pre-stored second facial features may correspond to multiple individuals, so that the number of the second facial images is may include at least two, and the number of reference similarities may include at least two. Accordingly, embodiments of the present disclosure further provide a facial recognition method, determining whether the first facial feature and the second facial feature correspond to the same person based on a reference similarity, a reference false alarm rate, and a similarity threshold. The step to decide whether
determining a similarity threshold based on the reference false alarm rate, and determining a second facial feature that is most similar to the first facial feature from the at least two reference similarities;
If the reference similarity between the second facial feature with the highest similarity and the first facial feature is greater than the similarity threshold, the second facial feature with the highest similarity is the same as the first facial feature. and determining that the method corresponds to.

In this embodiment, the electronic device obtains the reference similarities between the first facial feature and the at least two second facial features, thereby determining the similarity between the first facial feature and the first facial feature from the at least two reference similarities. determine the second facial feature with the highest degree of similarity, and then determine whether the reference similarity between the second facial feature with the highest degree of similarity and the first facial feature is greater than a similarity threshold (different false alarms); If the similarity is large, it indicates that the second facial feature and the first facial feature with the highest degree of similarity correspond to the same person.

Furthermore, the number of normalized reference similarities obtained by the electronic device based on the reference similarities between the first facial feature and at least two second facial features also includes at least two. Therefore, after determining the normalized reference similarity, the electronic device further determines a second facial feature that has the highest similarity to the first facial feature based on the normalized reference similarity. and the normalized reference similarity between the second facial feature and the first facial feature that has the highest similarity with the first facial feature is set to a similarity threshold (obtained based on different false alarm rates). ), it can be determined that the second facial feature with the highest degree of similarity to the first facial feature and the face in the first facial feature belong to the same face, that is, they correspond to the same person. can.

In the embodiment of the present disclosure, the electronic device needs to compare whether the face image at the time of registration and the face image at the time of face recognition correspond to the same person. There is no limitation as to whether the electronic device that acquires the facial image during face recognition and the electronic device that acquires the facial image during face recognition are of the same type (or the same).

In the process of implementing the embodiments of the present disclosure, the applicant of the present disclosure often restricts the image type that is authenticated and the image type that is registered in conventional face recognition methods to be the same image type; For example, if RGB images are used in the registration process, RGB images must also be used in the authentication process, thus limiting the application of traditional facial recognition solutions in scenes involving multiple types of cameras. I discovered that. Embodiments of the present disclosure provide a facial image recognition solution for scenes involving multiple types of cameras, and implementations of embodiments of the present disclosure provide a facial image recognition solution for scenes involving multiple types of cameras; Implementation of the present disclosure by taking a face image as a base library feature, comparing the face feature of a first face image acquired by another type of camera with the base library feature, and realizing face recognition based on the comparison result. The face recognition method according to the example can be applied to more scenes, which not only makes face recognition easier, but also the camera used during face registration is not limited, improving the convenience of face registration.

As an example, a scene to which the embodiment of the present disclosure is applied will be described below.

scene 1

(1) Regarding the registration process, the user may complete the face registration by mobile phone, that is, the registration flow may be performed by the face image taken by the camera installed on the mobile phone, and the mobile phone generally has a visible light camera. (for example, an RGB camera), the registered face image is a second face image taken by the RGB camera. Thereafter, the registered facial image is transmitted to the in-vehicle device by the server by or via the mobile phone, and the in-vehicle device stores the registered facial image, that is, the in-vehicle device transmits the second facial image. Remember. Alternatively, the user captures a second facial image using a visible light camera installed on a mobile phone, and sends the second facial image to the server or in-vehicle device, thereby capturing the second facial image in the server or in-vehicle device. The registration of the facial image may be completed, and the second facial feature extracted based on the second facial image may be saved after the registration is completed.

(2) Regarding the recognition process, the user collects a facial image that needs to be recognized by the in-vehicle device (i.e., the first facial image), and an infrared camera may be installed in the in-vehicle device, so that the infrared camera captures the facial image. Alternatively, for example, the IR face image may be used as the image of the target whose face is to be recognized, or the first face image. The in-vehicle device extracts the facial features of the IR image and compares the facial features of the IR image with the facial features of the registered RGB image to determine if the user to be identified is the same as the registered user. Determine whether a person is a person or not.

Note that scene 1 shown in the embodiment of the present disclosure can be applied to a driver condition monitoring system, a ride management system, an automobile rental management system, an online vehicle dispatch management system, and the like. For example, for a driver condition monitoring system, the system generally includes several modules such as facial recognition, driver motion detection, driver fatigue detection, and driver attention monitoring. Therefore, the above system can determine the identity of the driver, the identity of the person renting the vehicle, etc. by performing facial recognition using the method shown in scene 1 in the facial recognition step.

scene 2

The face recognition method according to the embodiment of the present disclosure can be further applied to a small area access management system, a security setting control system, etc., for example, a small area access management system is taken as an example.

(1) Regarding the registration process, a user (a resident in a certain sub-area) registers his or her face using a mobile phone, that is, a visible light camera (e.g. You may obtain facial images of Thereafter, the registered facial image is transmitted by the server to the entrance/exit control device by or via the mobile phone, and the entrance/exit control device stores the second facial image. Alternatively, the user can capture a second face image using a visible light camera installed on a mobile phone and transmit the second face image to the access control device, so that the access control device can display the second face image. The image registration may be completed, and the second facial features extracted based on the second facial image may be stored after the registration is completed.

(2) For the recognition process, when a user needs to enter or exit a sub-area, the access control device may obtain a facial image of the user, ie a first facial image, by an infrared camera (e.g. IR camera). After that, the entrance/exit control device extracts the facial features of the first facial image to obtain the first facial features, and the entrance/exit control device extracts the facial features of the first facial image, and the entrance/exit control device extracts the facial features of the first facial image. determine whether the user attempting to enter or leave the subarea is the same person as the registered user by comparing the facial features of the RGB image captured by the registered RGB camera. .

Note that the above are only some scenes illustrated in the embodiments of the present disclosure, and in concrete implementation, the method according to the embodiments of the present disclosure can be applied to many more scenes, for example, in the implementation of the present disclosure. The example method can further be applied to unlocking a terminal such as unlocking a mobile phone, bank identification system, etc., and the embodiments of the present disclosure are not limiting. For the application in unlocking a mobile phone, the user may perform face registration with the mobile phone, and then in the process of using the mobile phone each time, the mobile phone will be automatically registered according to the embodiments of the present disclosure. The method may be implemented to realize facial image identification. Note that the mobile phone at the time of face registration and the mobile phone at the time of face recognition may be different mobile phones, that is, the camera of the mobile phone at the time of face registration and the camera of the mobile phone at the time of face recognition may be of different types. It's fine. In addition, when a user opens a bank account for a bank ID authentication system, which is a financial ID authentication system, the user may register his or her face using a bank terminal. The terminal can perform the method according to the embodiment of the present disclosure to realize facial image recognition and ensure the security of the user's banking business. The types of cameras in the bank terminal during face registration and the bank terminal during face recognition may be different.

Therefore, the scenes shown in the embodiments of the present disclosure should not be understood as limiting the embodiments of the present disclosure.

The applicant of the present disclosure proposes that in the process of implementing the embodiments of the present disclosure, the same person takes two facial images using an RGB camera and an IR camera, and uses these two images to create an RGB image and an IR image. , formed three sets of mixed comparison combinations: mixed images of RGB and infrared images, and further discovered that there may be differences in the similarity obtained through the same neural network.

At the same time, two different people take four facial images with an RGB camera and an IR camera, respectively, and use these four facial images to create different image combinations, such as user 1 RGB image, user 2 RGB image, user 1 IR The image and the user 2 IR image, the user 1 RGB image and the user 2 IR image, and the user 1 IR image and the user 2 RGB image are configured. There may be differences in the degree of similarity obtained through the same neural network among these four mixed comparison combinations.

Accordingly, embodiments of the present disclosure provide a technical means for training a neural network (NN), which can effectively reduce or avoid the problems encountered above. Note that the neural network in the embodiments of the present disclosure may include a deep neural network (DNN), a convolutional neural network, etc., and the embodiments of the present disclosure do not limit the specific form of the neural network. .

Hereinafter, technical means for training a neural network according to an embodiment of the present disclosure will be described in detail.

Referring to FIG. 2, FIG. 2 is a flowchart of a neural network training method according to an embodiment of the present disclosure, the neural network training method can be applied to a neural network training device, and the neural network training method can be applied to a neural network training device. The method can be applied to an electronic device, and the electronic device may include a server or a terminal device, and the terminal device may include a mobile phone, a tablet computer, a desktop computer, a palm-sized personal computer, a vehicle-mounted device, a vehicle-mounted robot, etc. The embodiments of the present disclosure do not uniquely limit the specific form of the electronic device. Note that the neural network training method can be further applied to a face recognition device. That is, the method shown in FIG. 2 and the method shown in FIG. 1 may be performed by the same type of electronic device or by different types of electronic devices, and the embodiments of the present disclosure are not limiting. The method shown in FIG. 1 may be carried out by the same type of electronic equipment, ie the method shown in FIG. 2 may be carried out by a terminal device. Alternatively, the method shown in FIG. 2 may be performed by the same apparatus as the method shown in FIG. 1, and embodiments of the present disclosure are not limiting.

Hereinafter, the training images used in the training process of the neural network will be referred to as image samples, and the image samples include labeling information, and the labeling information includes the face ID in the image (which may be understood as face labeling information), the image The face ID corresponding to the same person is the same, and the type of image characterizes the camera type corresponding to the image collection, including, but not limited to, the type of image. As shown in FIG. 2, the neural network training method includes the following steps 201-202.

In step 201, a first type of image sample and a second type of image sample that are captured by different types of cameras and include a face are obtained.

In embodiments of the present disclosure, the first type of image sample includes at least a facial image, and the second type of image sample includes at least a facial image, and the first type of image sample and the second type of image sample include at least a facial image. A second type of image sample is acquired by a different type of camera. For example, if a first type of image sample is acquired by an RGB camera, a second type of image sample may be acquired by another type of camera, such as an IR camera. It should be noted that for specific embodiments of different types of cameras, reference may be made to the embodiment shown in FIG. 1 and will not be repeatedly described herein.

Note that the embodiment of the present disclosure does not limit the quantity of the first type of image samples and the quantity of the second type of image samples. The number of image samples of the first type and the number of image samples of the second type may be used as a metric for the degree of training of the neural network.

Step 202 trains a neural network based on the first type of image samples and the second type of image samples.

In embodiments of the present disclosure, the electronic device may train the neural network using image samples taken by two different types of cameras, such that the neural network uses two different types of images in the training process. It is possible to learn feature extraction ability for images. After the training of the neural network is completed, perform feature extraction on any one of these two different types of images based on the trained neural network, and ensure the accuracy of feature extraction. This satisfies the application needs of face registration and recognition based on these two types of cameras. In other words, when training a neural network, instead of simply training with one type of image samples, by training with multiple types of image samples, the trained neural network can be trained with different types of image samples. characteristics can be effectively acquired.

In one example, taking supervised training as an example, the training method of the neural network includes, for example, inputting a first type image sample and a second type image sample to the neural network, and passing the first type image sample through the neural network. and after processing the second type image samples, the neural network outputs the prediction results for the first type image samples and the prediction results for the second type image samples (i.e., the face prediction results for each image sample), respectively. do. Then, calculate the loss between the face prediction result of the image sample and the labeling information of the image sample (i.e., the true face labeling information) by a predetermined loss function, i.e., the loss of the first type of image sample by a predetermined loss function. a loss between the prediction result of the second type of image sample and the labeling information and a loss between the prediction result of the second type of image sample and the labeling information. Thereby, the loss is back-propagated to the neural network, which adjusts parameter values of network parameters such as convolution kernels, weights, etc. based on the back-propagated loss. Generally, after the adjustment, the images of the other first type are Further samples and other second type image samples may be input and the training process described above may be repeated. Note that the above is only a training process according to the embodiment of the present disclosure, and in concrete implementation, other methods may be included, and the embodiment of the present disclosure is not limited.

In one example, training the neural network based on the first type of image samples and the second type of image samples includes:
pairing the first type of image sample and the second type of image sample to obtain a mixed type of image sample of the first type of image sample and the second type of image sample;
training a neural network based on the first type of image samples, the second type of image samples and the mixed type of image samples.

In this example, a first type of image sample and a second type of image sample are paired, that is, each pair of mixed type image samples in the mixed type image sample is an image sample of the first type. sample and a second type of image sample, thereby forming two different types of training image samples. By training a neural network with image samples of the first type, image samples of the second type, and image samples of mixed types, the neural network can learn the feature extraction ability of each single type of image. Not only can the feature extraction ability of these two different types of images be better jointly learned, but by improving the feature extraction accuracy of the neural network, the trained neural network can improve the performance of the implementation of the present disclosure. It can be effectively applied to the face recognition method according to the example.

In one example, training the neural network based on the first type of image samples, the second type of image samples, and the mixed type of image samples includes:
obtaining face prediction results for a first type of image sample, a face prediction result for a second type of image sample, and a face prediction result for a mixed type image sample by a neural network;
The difference between the face prediction result and face labeling result of the first type image sample, the difference between the face prediction result and face labeling result of the second type image sample, and the face prediction result and face labeling of the mixed type image sample. training a neural network based on the differences with the results.

In this embodiment, the electronic device uses a neural network to obtain face prediction results for the first type image sample, face prediction results for the second type image sample, and face prediction results for the mixed type image sample. good. Thereafter, the difference between the face prediction result and the face labeling result of the first type image sample, the difference between the face prediction result and the face labeling result of the second type image sample, and the face prediction of the mixed type image sample. Train a neural network based on the differences between the results and the face labeling results. For example, training a neural network based on the loss between the face prediction result and the face labeling result of a first type of image sample, and the loss between the face prediction result and the face labeling result of a second type of image sample. The neural network may be trained based on the loss between the face prediction result and the face labeling result for the mixed type image samples. Note that for specific training methods, reference can be made to the description of the above-mentioned embodiments, and each training method will not be described in detail in this specification.

In one example, to further improve the accuracy with which the neural network extracts facial features, embodiments of the present disclosure further provide a training method, e.g., the neural network includes a first classifier, a second classifier. and a mixture classifier, the step of obtaining face prediction results for the first type image sample, the second type image sample and the mixture type image sample by the neural network,
inputting features of the first type of image sample into a first classifier to obtain a face prediction result of the first type of image sample;
inputting a second type of image sample into a second classifier to obtain a face prediction result for the second type of image sample;
inputting features of the mixed type image sample to a mixture classifier to obtain a face prediction result for the mixed type image sample.

In this example, the classifier may classify the face prediction results of image samples of different types, e.g. by being able to obtain the face prediction results of the image samples input to the classifier by the classifier. The loss can be determined based on the output of the device and further backpropagated to the neural network to train the neural network. In one example, a first classifier outputs a face prediction result for a first type of image sample, a second classifier outputs a face prediction result for a second type of image sample, and the mixture classification The device may output face prediction results for mixed type image samples. By outputting the face prediction result by the classifier corresponding to the type of image sample, it can effectively improve the accuracy of the classifier outputting the face prediction result, and furthermore, based on the output result of the classifier, the image features of mixed types can be output. The training accuracy of the neural network supporting extraction can be improved, and the accuracy and robustness of the neural network in extracting facial features of different types of images can be improved.

A specific implementation for training a neural network based on first type image samples, second type image samples and mixed type image samples may be as shown in FIG. , is a schematic diagram of a training process according to an embodiment of the present disclosure. The first type of image sample is an RGB image sample, each image sample included in the RGB image sample library is an RGB image sample, and the second type of image sample is an IR image sample, and each image sample included in the RGB image sample library is an IR image sample. Each image sample included in the sample library is an IR image sample. For example, a mixed type image sample is an RGB & IR image sample. The mixed type image sample library includes some RGB image samples and A partial IR image sample is included and displayed as an RGB&IR image sample. At the same time, take as an example that the first classifier is an RGB classifier, the second classifier is an IR classifier, and the mixed classifier is an RGB&IR classifier, the RGB classifier is The classification result of an RGB image sample indicates the probability that the face in the RGB image sample belongs to each face ID category in the RGB image sample library, and the IR classifier classifies the IR images in the IR image sample library. The classification result of a given IR image sample indicates the probability that the face in the IR image sample belongs to each face ID category in the IR image sample library, and the RGB&IR classifier classifies the RGB images in the mixed type image sample library. The samples and IR image samples are classified, and a certain RGB&IR image sample indicates the probability that the face in the image sample belongs to each face ID category in the mixed type image sample library. "&" in RGB&IR image sample indicates that the image sample input to the RGB&IR classifier, which is a mixed classifier, may be an RGB type image sample and may also be an IR type image sample. It's fine. Therefore, "&" in this disclosure should not be understood as limiting this disclosure.

As shown in FIG. 3, after inputting RGB image samples, IR image samples and RGB&IR image samples respectively into the neural network, the neural network uses a feature extractor to extract the features of the RGB image samples, the features of the IR image samples and the RGB&IR images. Sample characteristics may be output. Thereafter, the features of the RGB image samples, the features of the IR image samples, and the features of the RGB&IR image samples are input into the RGB classifier, the IR classifier, and the RGB&IR classifier, respectively. As a result, the neural network outputs the face prediction result of each image sample using the classifier, and compares the face prediction result of each image sample with the face labeling result of each image sample to predict the face of each image sample. Obtain the loss between the result and the face labeling result of each image sample, and further backpropagate the loss to the entire neural network to train the entire neural network. In this example, by adding a classifier to the neural network, the classifier helps train the entire neural network, so that the trained neural network can accurately and effectively distinguish the features of different types of images. extraction, thereby improving the accuracy and efficiency of face recognition.

Note that after completing the training of the entire neural network using the first classifier, the second classifier, and the mixed classifier, the method shown in FIG.
The method further includes the step of removing a first classifier, a second classifier, and a mixed classifier from the trained neural network to obtain a neural network for performing face recognition.

That is, the first classifier, the second classifier, and the mixed classifier can support training of a neural network, but in a specific application, for example, by applying the method according to the embodiment of the present disclosure. In the process of performing face recognition via a neural network, the neural network may not include the first classifier, the second classifier and the mixed classifier, so that the training of the neural network training device is Once completed, the first classifier, second classifier and mixed classifier in the neural network can be removed.

In embodiments of the present disclosure, by training a neural network using images taken by different types of cameras, the efficiency of the neural network to output features can be effectively improved, and images of different types can be trained using the same neural network. Similarity differences due to feature extraction by a network can be reduced or avoided.

In one example, to obtain threshold information for determining the normalized reference similarity shown in FIG. 1, embodiments of the present disclosure further provide a method for obtaining threshold information, as shown below. be.

After training the neural network based on the first type of image samples, the second type of image samples and the mixed type of image samples, the method shown in FIG.
acquiring a first type of image and a second type of image acquired by different cameras;
a positive sample pair based on the first type of image and the second type of image, each including two images, and the faces in the two images correspond to the same person; obtaining negative sample pairs corresponding to people whose faces in the two images are different;
determining the similarity of the positive sample pairs and the similarity of the negative sample pairs, respectively, via the trained neural network;
The method further includes determining threshold information including a first threshold value and a second threshold value based on the similarity of the positive sample pair, the similarity of the negative sample pair, and different preset false alarm rates.

In this example, the first type of image includes at least a face image, the second type of image includes at least a face image, and the first type of image and the second type of image Images are acquired by different types of cameras. For example, an RGB camera can acquire RGB images of multiple people, and an IR camera can acquire IR images of multiple people, including, for example, N people, each person having M RGB images and M images. Contains IR images. Both M and N are integers of 2 or more.

In one example, obtaining positive sample pairs based on the first type of image and the second type of image includes:
pairing the first type of images to obtain a positive sample pair of the first type of images;
pairing the second type of images to obtain a positive sample pair of the second type of images;
pairing the first type of image and the second type of image to obtain a positive sample pair of mixed images.

In this embodiment, for example, if there are N people and each person includes M RGB images and M IR images, the M RGB images included by each person are paired and M ^* ( M-1)/2 RGB positive sample pairs are obtained, and M ^* (M-1)/2 IR positive sample pairs are obtained by pairing the M IR images included in each person. M*M RGB&IR positive sample pairs are obtained by pairing M RGB images and M IR images of a person.

For negative sample pairs, images of each person can be paired with images of other different people to form negative sample pairs. For example, taking a first user and a second user as an example, the negative sample pairs include an IR image of the first user and an IR image of the second user, and an RGB image of the first user and the second user. , an IR image of the first user, and an RGB image of the second user.

After obtaining the positive sample pair and the negative sample pair, they can be tested on the trained neural network shown in Figure 2, for example, the facial features of the positive sample pair and the face of the negative sample pair can be tested by the trained neural network. outputting features, obtaining the similarity of the positive sample pair based on the facial features of the positive sample pair, and obtaining the similarity of the negative sample pair based on the facial features of the negative sample pair. Thereby, the passing rate and the corresponding threshold value at different false alarm rates are obtained. Specifically, for example, when the similarity of positive sample pairs and the similarity of negative sample pairs are obtained, the sample pair with the lowest similarity is found from the similarity of each sample pair based on the target false alarm rate, and The sample pair with the lowest degree of similarity belongs to the sample pair of false alarms, so that the degree of similarity corresponding to the sample pair with the lowest degree of similarity can be set as the threshold value corresponding to the target false alarm rate. Note that the target false alarm rate can be understood as one preset or selected false alarm rate, and the embodiments of the present disclosure do not limit the specific value of the target false alarm rate.

As shown in Table 1, Table 1 is threshold information according to an example of the present disclosure.

The false alarm rate may be independently set by the electronic device, and the embodiments of the present disclosure do not limit the specific value of the false alarm rate.

For example, if the target false alarm rate is 0.00001 and the total number of positive sample pairs and negative sample pairs is 1,000,000, it can be seen that there are 10 false alarm sample pairs in total. It is possible to find the 10 false alarm sample pairs from the positive sample pairs and negative sample pairs, and then extract the sample pair with the lowest degree of similarity from among them, thereby determining the degree of similarity of the sample pair with the lowest degree of similarity. is a threshold corresponding to the target false alarm rate. For Table 1, the threshold corresponding to the target false alarm rate of 0.00001 is T(5). Note that the embodiments of the present disclosure do not limit the method of selecting the sample pair with the lowest degree of similarity; for example, the sample pair with the lowest degree of similarity can be obtained by a ranking method. . Each correspondence relationship shown in Table 1 is only an example, and the embodiments of the present disclosure do not limit specific values.

In this embodiment, after training the neural network, thresholds at different false alarm rates can be obtained by further testing the neural network based on a large number of images, that is, threshold information can be obtained, and furthermore, the neural network can be tested based on a large number of images. When applying the network, the normalized reference similarity may be determined using different threshold information (eg, a first threshold and a second threshold) based on different false alarm rates. Note that for a specific application of Table 1, reference can be made to the implementation form shown in FIG. By determining a normalized reference similarity based on the reference similarity of the second facial feature, the first facial feature and the second facial feature are determined based on the normalized reference similarity. You may decide whether to respond to the same person or not.

It should be noted that each of the above embodiments has its own importance, and implementation forms that are not explained in detail in one embodiment can be referred to correspondingly to implementation forms of other embodiments, and in this specification, 1. I will not elaborate on each one.

The above describes the method of the embodiment of the present disclosure in detail, and below, the apparatus and electronic device of the embodiment of the present disclosure are provided and in view of the brief description, technical principles, technical effects, etc. For parts of the apparatus that are not explained in detail in the above, reference can be made to the corresponding description of the method embodiments above, and the explanation will be omitted.

Referring to FIG. 4, FIG. 4 is a schematic configuration diagram of a face recognition device according to an embodiment of the present disclosure, and the face recognition device can execute the face recognition method shown in FIG. 1. As shown in FIG. 4, the face recognition device includes:
a first acquisition unit 401 that acquires a first face image with a first camera;
a first extraction unit 402 that extracts a first facial feature of a first facial image;
The first facial feature is compared with a second facial feature extracted from the features of a second facial image acquired by a second camera that is a different type of camera from the first camera, and a reference similarity is obtained. A comparison unit 403 that
a determining unit 404 for determining whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity.

In the process of implementing the embodiments of the present disclosure, the applicant of the present disclosure often restricts the image type that is authenticated and the image type that is registered in conventional face recognition methods to be the same image type; For example, if RGB images are used in the registration process, RGB images must also be used in the authentication process, thus limiting the application of traditional facial recognition solutions in scenes involving multiple types of cameras. I discovered that. Embodiments of the present disclosure provide a facial image recognition solution for scenes involving multiple types of cameras, and implementations of embodiments of the present disclosure provide a facial image recognition solution for scenes involving multiple types of cameras; Implementation of the present disclosure by taking a face image as a base library feature, comparing the face feature of a first face image acquired by another type of camera with the base library feature, and realizing face recognition based on the comparison result. The face recognition method according to the example can be applied to more scenes, which not only makes face recognition easier, but also the camera used during face registration is not limited, improving the convenience of face registration.

In one example, the first camera is a thermal camera and the second camera is a visible light camera, or the first camera is a visible light camera and the first camera is a thermal camera.

In one example, the determining unit 404 specifically determines whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity, the reference false alarm rate, and the similarity threshold. However, different false alarm rates correspond to different similarity thresholds.

In implementations of embodiments of the present disclosure, determining whether the first facial feature and the second facial feature correspond to the same person by obtaining different similarity thresholds with different false alarm rates may be fixed. By bypassing solutions that authenticate faces in a manner that uses a similarity threshold of improve efficiency and accuracy.

In one example, the determining unit 404 specifically determines the normalized reference similarity based on the reference similarity and the threshold information, and determines the first facial feature based on the normalized reference similarity. and the second facial feature correspond to the same person.

In the embodiment of the present disclosure, the normalized reference similarity is determined based on the reference similarity and preset information, and the first facial feature is determined based on the normalized reference similarity. It is determined whether the second facial feature corresponds to the same person. By effectively improving the face recognition solution in a manner that uses a fixed similarity threshold, the similarity measure (i.e., the normalized reference similarity measure) for determining the relationship between two facial images. can be determined dynamically, improving the efficiency and accuracy of face recognition.

In one example, the first extraction unit 402 specifically inputs the first facial image to a pre-trained neural network, outputs the first facial feature of the first facial image by the neural network, The neural network is obtained by training on a first type of image samples and a second type of image samples that are taken by different types of cameras and include faces.

Embodiments of the present disclosure effectively improve the efficiency and accuracy of face recognition, such as applying the neural network to recognize faces, by training the neural network with different types of image samples.

In one example, the neural network is obtained by training on image samples of a first type, image samples of a second type and image samples of a mixed type, where the image samples of a mixed type are of a first type. It is obtained by pairing the image sample and the second type of image sample.

In one example, the first camera includes an on-vehicle camera, and the first acquisition unit 401 specifically captures the first facial image by the on-vehicle camera, and the first facial image is a vehicle user of the vehicle. Contains facial images.

Embodiments of the present disclosure can be effectively applied to a driver monitoring system to improve the efficiency of driver face recognition.

In one example, vehicle users include one or more of a person who drives the vehicle, a person who rides the vehicle, a person who repairs the vehicle, a person who refuels the vehicle, and a person who controls the vehicle.

In one example, when the vehicle user includes a person driving a vehicle, the first acquisition unit 401 specifically acquires a first facial image by an on-vehicle camera when receiving a trigger command;
Alternatively, the first acquisition unit 401 specifically acquires the first facial image using an on-vehicle camera while the vehicle is running;
Alternatively, the first acquisition unit 401 specifically acquires the first face image using the vehicle-mounted camera when the traveling speed of the vehicle reaches the reference speed.

In one example, the second face image is an image for the vehicle user to perform face registration, and as shown in FIG.
a second acquisition unit 405 that acquires a second facial image with a second camera;
a second extraction unit 406 for extracting a second facial feature of the second facial image;
and a storage unit 407 for storing second facial features of the second facial image.

As should be explained, the implementation of each unit may also be referred to correspondingly in the corresponding description of the method embodiment shown in FIG.

Referring to FIG. 6, FIG. 6 is a schematic configuration diagram of a neural network training device according to an embodiment of the present disclosure, and the neural network training device is capable of executing the face recognition method shown in FIG. can. As shown in FIG. 6, the neural network training device includes:
an acquisition unit 601 for acquiring a first type of image sample and a second type of image sample taken by different types of cameras and containing a face;
a training unit 602 for training the neural network based on the first type of image samples and the second type of image samples.

In embodiments of the present disclosure, by training the neural network using facial images taken by different types of cameras, the accuracy of the neural network outputting facial features can be effectively improved, and facial images can be effectively recognized. When performing facial recognition, extracting facial features using the neural network effectively improves the accuracy of facial recognition.

In one example, as shown in FIG. 7, training unit 602 includes:
Pairing subunit 6021 for pairing a first type of image sample and a second type of image sample to obtain a mixed type image sample of the first type of image sample and the second type of image sample. and,
a training subunit 6022 for training the neural network based on the first type of image samples, the second type of image samples, and the mixed type of image samples.

In this example, a first type of image sample and a second type of image sample are paired, that is, each pair of mixed type image samples in the mixed type image sample is an image sample of the first type. sample and a second type of image sample, thereby forming two different types of training image samples. By training a neural network with image samples of the first type, image samples of the second type, and image samples of mixed types, the neural network can learn the feature extraction ability of each single type of image. Not only can the feature extraction ability of these two different types of images be better jointly learned, but by improving the feature extraction accuracy of the neural network, the trained neural network can improve the performance of the implementation of the present disclosure. It can be effectively applied to the face recognition method according to the example.

In one example, the training subunit 6022 specifically uses a neural network to generate face prediction results for the first type of image sample, face prediction results for the second type of image sample, and face prediction results for the mixed type image sample. and the difference between the face prediction result and the face labeling result of the first type of image sample, the difference between the face prediction result and the face labeling result of the second type of image sample, and the face of the mixed type image sample. Train a neural network based on the difference between the prediction result and the face labeling result.

In one example, the neural network includes a first classifier, a second classifier, and a mixture classifier, and the training subunit 6022 specifically combines features of the first type of image sample with the first classifier. A second type of image sample is input to a second classifier to obtain a face prediction result of the first type of image sample, and a second type of image sample is input to a second classifier to obtain a face prediction result of the second type of image sample. Obtaining the result, and inputting the facial features of the mixed type image sample to a mixture classifier to obtain the face prediction result of the mixed type image sample.

In one example, as shown in FIG.
The method further includes a neural network application unit that removes the first classifier, the second classifier, and the mixed classifier from the trained neural network to obtain a neural network for performing face recognition.

As such, the implementation of each unit may also be referred to correspondingly in the corresponding description of the method embodiment shown in FIG. 2.

Referring to FIG. 9, FIG. 9 is a schematic configuration diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 9, the electronic device includes a processor 901, a memory 902, and an input/output interface 903 that are connected to each other by a connection line that may include a bus or the like.

The input/output interface 903 can input data and/or signals and output data and/or signals.

Memory 902 may be random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or compact read-only memory. (compact The memory 902 is used for related instructions and data, including but not limited to disc read-only memory (CD-ROM).

The processor 901 may be one or more, and the one or more processors may be one or more central processing units (CPUs) and/or one or more acceleration units. The CPU may be a single-core CPU or a multi-core CPU. The acceleration unit includes, but is not limited to, a graphics processing unit (GPU), a field-programmable gate array (FPGA), and the like.

Note that the processor according to the embodiment of the present disclosure may be another type of processor, and the embodiment of the present disclosure does not uniquely limit the type of the processor. The memory according to embodiments of the present disclosure may be other types of memory, and embodiments of the present disclosure do not limit the type of memory.

In one example, the implementation of each operation can further refer to the corresponding description of the method embodiment shown in FIG. The implementation of each operation can be referred to correspondingly in the corresponding description of the method embodiment further illustrated in FIG. Alternatively, the implementation of each operation can further refer correspondingly to the corresponding description of the embodiments shown in FIGS. 4 and 5. The implementation of each operation can also be correspondingly referred to the corresponding description of the embodiments shown in FIGS. 6 to 8.

In one embodiment, processor 901 may perform the method shown in steps 101-104. As an example, the processor may further control the input/output interface to obtain a first facial image, etc., and embodiments of the present disclosure uniquely determine how to obtain the first facial image. Not limited.

Also for example, in one embodiment, the processor may perform the method shown in steps 201 and 202.

Also, for example, the processor 901 may perform the method performed by the first acquisition unit 401, as well as the first extraction unit 402, the comparison unit 403, the determination unit 404, etc.

Also, for example, processor 901 may further perform methods such as those performed by acquisition unit 601 and training unit 602.

As a result, the specific implementation form of the electronic device shown in FIG. 9 can be referred to in a corresponding manner to the description of each of the above-mentioned embodiments, and will not be described in detail one by one in this specification.

Embodiments of the present disclosure further provide a computer-readable storage medium. All or some of the flows in the method embodiments described above may be completed by instructing associated hardware by a computer program, which program may be stored on the computer storage medium, and during execution, the program may execute the steps in each of the method embodiments described above. It may contain the flow of The computer-readable storage medium may be an internal storage unit of the facial recognition device or neural network training device of any of the embodiments described above, such as a hard disk or memory of the facial recognition device or neural network training device. The computer-readable storage medium may be an external storage device of the facial recognition device or neural network training device, such as an insertable hard disk, a smart memory card, or a smart media card provided in the facial recognition device or neural network training device. SMC), secure digital (SD) card, flash memory card (flash card), etc. may be used. Furthermore, the computer-readable storage medium may include not only an internal storage unit of the facial recognition device or neural network training device, but also an external storage device. The computer-readable storage medium stores the computer program and other programs and data necessary for the facial recognition device or neural network training device. The computer-readable storage medium may temporarily store data that has been output or is to be output.

In one or more optional embodiments, embodiments of the present disclosure may be implemented using a computer program storing computer readable instructions that, when executed, cause a computer to perform the method described in any one of the embodiments above. Provide more programs .

Specifically, the computer program can be implemented using hardware, software, or a combination thereof. In one alternative example, the computer program is specifically embodied as a computer storage medium, and in another alternative, the computer program is specifically embodied in software , such as a software development packet ( It is realized as a software development kit (SDK), etc.

The embodiments described above may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in a software program, it may be implemented in whole or in part in the form of a computer program . The computer program includes one or more computer instructions. When loaded and executed on a computer, the computer program instructions may be generated in whole or in part based on the flows or functionality described in the embodiments of the present disclosure. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions may be stored on or transmitted by a computer-readable storage medium. The computer-readable storage medium may be any available computer-accessible media or data storage device such as a server, data center, etc. that includes one or more collections of available media. The available media may be magnetic media (e.g., soft disk, hard disk, magnetic tape), optical media (e.g., DVD (registered trademark) ), or semiconductor media (e.g., solid state disk (SSD)). It's fine.

The steps in the method of the embodiments of the present disclosure can be reordered, combined, and deleted according to actual needs.

Modules in the embodiment device of the present disclosure can be merged, divided, and deleted according to actual needs.

As mentioned above, it should be explained that the above embodiments are only for illustrating the technical means of the present disclosure and not limiting, and the present disclosure will be described in detail with reference to the above embodiments. Having explained this, it should be understood by those skilled in the art that the technical means described in each of the above-mentioned embodiments can still be modified or equivalent substitutions can be made to some of the technical features thereof, and these Due to modifications and substitutions, the essence of the corresponding technical means does not depart from the scope of the technical means according to the embodiments of the present disclosure.

Claims (14)

acquiring a first facial image with a first camera;
extracting a first facial feature of the first facial image;
The first facial features are compared with pre-stored second facial features extracted from the features of a second facial image acquired by a second camera that is a different type of camera from the first camera. , obtaining a reference similarity indicating a similarity between the first facial feature and the second facial feature;
determining whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity;
The first camera is a thermo camera and the second camera is a visible light camera, or the first camera is a visible light camera and the second camera is a thermo camera,
The step of determining whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity,
determining whether the first facial feature and the second facial feature correspond to the same person based on a reference false alarm rate and a similarity threshold in addition to the reference similarity; , the similarity threshold is a threshold of the reference similarity, and different reference false alarm rates correspond to different similarity thresholds, or
In addition to the reference similarity, a normalized reference similarity is determined based on threshold information, and the first facial feature and the second face are determined based on the normalized reference similarity. determining whether the features correspond to the same person , the threshold information being obtained based on the similarity of the positive sample pair, the similarity of the negative sample pair, and different preset false alarm rates. , the threshold information includes a first threshold and a second threshold, the reference similarity is between the first threshold and the second threshold, and the normalized reference similarity is between the first and second thresholds. a reference similarity, determined based on the first threshold and the second threshold ;
determining whether the first facial feature and the second facial feature correspond to the same person based on a reference false alarm rate and a similarity threshold in addition to the reference similarity;
determining the similarity threshold based on the reference false alarm rate and obtaining the reference similarity between the first facial feature and at least two second facial features; determining a second facial feature with the highest degree of similarity;
If the reference similarity between the second facial feature having the highest degree of similarity and the first facial feature is greater than the similarity threshold, the second facial feature having the highest degree of similarity and the first facial feature have the highest degree of similarity. determining that the facial features correspond to the same person;
determining whether the first facial feature and the second facial feature correspond to the same person based on the normalized reference similarity;
the first facial feature having the highest similarity to said first facial feature based on at least two normalized reference similarities obtained based on reference similarities between said first facial feature and at least two second facial features; A second facial feature is determined, and the normalized reference similarity between the second facial feature and the first facial feature having the highest degree of similarity with the first facial feature is the similarity threshold. , the step of determining that a second facial feature having the highest degree of similarity to the first facial feature and the first facial feature correspond to the same person. Method. The step of extracting a first facial feature of the first facial image includes:
inputting the first facial image into a pre-trained neural network, and outputting a first facial feature of the first facial image by the neural network, the neural network is configured to obtained by training on image samples and a second type of image sample , the first type of image sample being taken by a first type of camera, and the second type of image sample being a second type of image sample. taken by the camera of
The first type of camera is a thermal camera and the second type of camera is a visible light camera, or the first type of camera is a visible light camera and the second type of camera is a visible light camera. is a thermo camera,
the first type of image sample and the second type of image sample include a face;
The face recognition method according to claim 1 , wherein the neural network has feature extraction capabilities for two different types of images . The neural network is trained based on the first type of image samples, the second type of image samples and mixed type image samples, the mixed type image samples being the first type of image samples. 3. The face recognition method according to claim 2, wherein the face recognition method is obtained by pairing the second type of image sample and the second type of image sample. The first camera includes an in-vehicle camera, and the step of acquiring a first facial image with the first camera includes:
4. The method according to claim 1, further comprising the step of acquiring the first facial image using the on-vehicle camera, wherein the first facial image includes a facial image of a vehicle user of the vehicle . Facial recognition method described in Section. The vehicle user includes one or more of the following: a person who drives the vehicle, a person who rides the vehicle, a person who repairs the vehicle, a person who refuels the vehicle, and a person who controls the vehicle. The face recognition method according to claim 4 . The vehicle user includes a person who drives the vehicle, and the step of acquiring the first facial image with the in-vehicle camera includes:
If a trigger command is received, acquiring the first facial image of the person driving the vehicle using the in-vehicle camera;
Alternatively, while the vehicle is running, acquiring the first facial image of the person driving the vehicle using the in-vehicle camera;
Alternatively, the method further includes the step of acquiring the first facial image of the person driving the vehicle using the in-vehicle camera when the traveling speed of the vehicle reaches a reference speed;
5. The face recognition method according to claim 4, wherein the trigger command is a trigger command input by a user or a trigger command transmitted from another electronic device. The second face image is an image for the vehicle user to perform face registration, and the second face image is an image for the vehicle user to perform face registration, and the second face image is an image for the vehicle user to perform face registration. The recognition method is
acquiring the second facial image with the second camera;
extracting a second facial feature of the second facial image;
The face recognition method according to any one of claims 4 to 6, further comprising the step of storing a second facial feature of the second facial image. obtaining a first type of image sample and a second type of image sample, each of which includes facial images captured by different types of cameras;
training a neural network based on the first type of image samples and the second type of image samples;
The different types of cameras include a first camera and a second camera, the first type of image sample being taken by the first camera, and the second type of image sample being taken by the second camera. ,
The first camera is a thermo camera and the second camera is a visible light camera, or the first camera is a visible light camera and the second camera is a thermo camera,
The neural network has feature extraction capabilities for two different types of images,
The step of training a neural network based on the first type of image samples and the second type of image samples includes:
pairing the first type of image sample and the second type of image sample to obtain a mixed type image sample of the first type of image sample and the second type of image sample; the mixed-type image samples include a plurality of pairs of mixed-type image samples, each pair of mixed-type image samples including a first type of image sample and a second type of image sample; step and
A method for training a neural network, comprising the step of training the neural network based on the first type of image sample, the second type of image sample, and the mixed type of image sample. The step of training the neural network based on the first type of image samples, the second type of image samples and the mixed type of image samples includes:
obtaining a face prediction result of the first type image sample, a face prediction result of the second type image sample, and a face prediction result of the mixed type image sample by the neural network;
The difference between the face prediction result and face labeling result of the first type image sample, the difference between the face prediction result and face labeling result of the second type image sample, and the face prediction result of the mixed type image sample. and a face labeling result , the face labeling result being labeling information of a face included in an image sample. A neural network training method described in . The neural network includes a first classifier, a second classifier, and a mixture classifier, and the neural network calculates a face prediction result for the first type of image sample and a face prediction result for the second type of image sample. The step of obtaining a face prediction result and a face prediction result of the mixed type image sample comprises:
inputting facial features of the first type of image sample into the first classifier to obtain a face prediction result of the first type of image sample;
inputting facial features of the second type of image sample to the second classifier to obtain a face prediction result of the second type of image sample;
The neural network according to claim 9 , further comprising inputting facial features of the mixed type image sample to the mixture classifier to obtain a face prediction result of the mixed type image sample. training method. After said step of training said neural network based on said first type of image samples, said second type of image samples and said mixed type of image samples;
Claim further comprising the step of removing the first classifier, the second classifier, and the mixed classifier from the neural network to obtain a neural network for performing face recognition. 10. The neural network training method described in 10 . a first acquisition unit that acquires a first face image with a first camera;
a first extraction unit that extracts a first facial feature of the first facial image;
The first facial features are compared with pre-stored second facial features extracted from the features of a second facial image acquired by a second camera that is a different type of camera from the first camera. , a comparison unit that obtains a reference similarity indicating the similarity between the first facial feature and the second facial feature;
a determining unit for determining whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity;
The first camera is a thermo camera and the second camera is a visible light camera, or the first camera is a visible light camera and the second camera is a thermo camera,
The determining unit determines whether the first facial feature and the second facial feature correspond to the same person based on the reference similarity,
determining whether the first facial feature and the second facial feature correspond to the same person based on a reference false alarm rate and a similarity threshold in addition to the reference similarity; , the similarity threshold is a threshold of the reference similarity, and different reference false alarm rates correspond to different similarity thresholds, or
In addition to the reference similarity, a normalized reference similarity is determined based on threshold information, and the first facial feature and the second face are determined based on the normalized reference similarity. and determining whether the features correspond to the same person, the threshold information being obtained based on the similarity of the positive sample pair, the similarity of the negative sample pair and different preset false alarm rates. , the threshold information includes a first threshold and a second threshold, the reference similarity is between the first threshold and the second threshold, and the normalized reference similarity is between the first and second thresholds. a reference similarity, determined based on the first threshold and the second threshold ;
Determining whether the first facial feature and the second facial feature correspond to the same person based on the reference false alarm rate and a similarity threshold in addition to the reference similarity,
determining the similarity threshold based on the reference false alarm rate and obtaining the reference similarity between the first facial feature and at least two second facial features; determining a second facial feature with the highest degree of similarity;
If the reference similarity between the second facial feature having the highest degree of similarity and the first facial feature is greater than the similarity threshold, the second facial feature having the highest degree of similarity and the first facial feature have the highest degree of similarity. determining that the facial features correspond to the same person;
determining whether the first facial feature and the second facial feature correspond to the same person based on the normalized reference similarity;
the first facial feature having the highest similarity to said first facial feature based on at least two normalized reference similarities obtained based on reference similarities between said first facial feature and at least two second facial features; A second facial feature is determined, and the normalized reference similarity between the second facial feature and the first facial feature having the highest degree of similarity with the first facial feature is the similarity. Face recognition comprising: determining that a second facial feature having the highest degree of similarity to the first facial feature and the first facial feature correspond to the same person if the second facial feature is larger than a threshold. Device. comprising a processor and a memory, the processor coupled to the memory, the memory, when executed by the processor, causing the processor to perform the face recognition method of any one of claims 1 to 7 ; Or an electronic device storing a program instruction for causing the processor to execute the neural network training method according to any one of claims 8 to 11 . When executed by a processor, it causes the processor to perform a face recognition method according to any one of claims 1 to 7 , or causes the processor to perform a neural network according to any one of claims 8 to 11 . A computer-readable storage medium having stored thereon a computer program containing program instructions for carrying out a training method.