CN110335389B

CN110335389B - Vehicle door unlocking method, vehicle door unlocking device, vehicle door unlocking system, electronic equipment and storage medium

Info

Publication number: CN110335389B
Application number: CN201910586845.6A
Authority: CN
Inventors: 李轲; 林琪钧
Original assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Current assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Priority date: 2019-07-01
Filing date: 2019-07-01
Publication date: 2021-10-12
Anticipated expiration: 2039-07-01
Also published as: WO2021000587A1; JP2022118730A; KR20220070581A; CN110335389A; KR20220016184A; JP2022537923A

Abstract

The disclosure relates to a vehicle door unlocking method, a vehicle door unlocking device, a vehicle door unlocking system, a vehicle, electronic equipment and a storage medium. The method comprises the following steps: searching a Bluetooth device with a preset identifier through a Bluetooth module arranged on the vehicle; responding to the Bluetooth device with the preset identifier, and establishing Bluetooth pairing connection between the Bluetooth module and the Bluetooth device with the preset identifier; responding to the successful Bluetooth pairing connection, awakening and controlling an image acquisition module arranged on the vehicle to acquire a first image of a target object; performing face recognition based on the first image; and responding to the success of the face recognition, and sending a door unlocking instruction to at least one door lock of the vehicle. The embodiment of the disclosure can meet the requirements of low-power operation, rapid door opening and improvement of user experience.

Description

Vehicle door unlocking method, vehicle door unlocking device, vehicle door unlocking system, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of vehicle technologies, and in particular, to a method, an apparatus, a system, a vehicle, an electronic device, and a storage medium for unlocking a vehicle door.

Background

Brushing the face to drive the car door is a new technology of the intelligent vehicle. At present, in order to detect a human face in time, a camera needs to be kept in an open state; whether the person who is close to the vehicle is the car owner in order to can in time judge, need carry out real-time processing to the image that the camera was gathered to quick discernment car owner is with opening the door fast. However, the operation power consumption of this method is high, and long-time high power consumption operation may cause that the vehicle cannot be started due to insufficient electric quantity, which may affect normal use of the vehicle by the user and user experience.

How to reduce the power consumption of the mode of brushing the face and opening the vehicle door is a problem to be solved urgently.

Disclosure of Invention

The present disclosure provides a technical scheme for unlocking a vehicle door.

According to a first aspect of the present disclosure, there is provided a vehicle door unlocking method including:

searching a Bluetooth device with a preset identifier through a Bluetooth module arranged on the vehicle;

responding to the Bluetooth device with the preset identifier, and establishing Bluetooth pairing connection between the Bluetooth module and the Bluetooth device with the preset identifier;

responding to the successful Bluetooth pairing connection, awakening and controlling an image acquisition module arranged on the vehicle to acquire a first image of a target object;

performing face recognition based on the first image;

and responding to the success of the face recognition, and sending a door unlocking instruction to at least one door lock of the vehicle.

In one possible implementation manner, the bluetooth device that searches for the preset identifier via the bluetooth module provided to the vehicle includes:

when the vehicle is in a flameout state or in a flameout and vehicle door locking state, the Bluetooth device with the preset identifier is searched through the Bluetooth module arranged on the vehicle.

In a possible implementation manner, the number of the bluetooth devices of the preset identifier is one.

In a possible implementation manner, the number of the bluetooth devices with the preset identifier is multiple;

the establishing of the Bluetooth pairing connection between the Bluetooth module and the Bluetooth device with the preset identifier in response to the searching of the Bluetooth device with the preset identifier comprises:

and responding to the searched Bluetooth device with any preset identification, and establishing Bluetooth pairing connection between the Bluetooth module and the Bluetooth device with the preset identification.

In a possible implementation manner, the waking up and controlling an image capturing module disposed in the vehicle to capture a first image of a target object includes:

awakening a face recognition module arranged on the vehicle;

and the awakened face recognition module controls the image acquisition module to acquire a first image of the target object.

In one possible implementation manner, after the waking up the face recognition module set in the car, the method further includes:

and if the face image is not acquired within the preset time, controlling the face recognition module to enter a dormant state.

In one possible implementation manner, the sending a door unlocking instruction to at least one door lock of the vehicle in response to successful face recognition includes:

responding to the success of face recognition, and determining the vehicle door of which the target object has the door opening authority;

and sending a door unlocking instruction to at least one door lock of the vehicle according to the vehicle door of which the target object has the door opening authority.

and if the face recognition module fails to pass the face recognition within the preset time, controlling the face recognition module to enter a dormant state.

In one possible implementation, the face recognition includes: living body detection and face authentication;

the face recognition based on the first image comprises:

acquiring the first image through an image sensor in the image acquisition module, and performing face authentication based on the first image and pre-registered face features;

and acquiring a first depth map corresponding to the first image through a depth sensor in the image acquisition module, and carrying out in-vivo detection on the basis of the first image and the first depth map.

In one possible implementation, the performing living body detection based on the first image and the first depth map includes:

updating the first depth map based on the first image to obtain a second depth map;

determining a live body detection result of the target object based on the first image and the second depth map.

In one possible implementation, the image sensor includes an RGB image sensor or an infrared sensor;

the depth sensor comprises a binocular infrared sensor or a time of flight TOF sensor.

In one possible implementation, the TOF sensor employs a TOF module based on an infrared band.

In a possible implementation manner, the updating the first depth map based on the first image to obtain a second depth map includes:

and updating the depth value of the depth failure pixel in the first depth map based on the first image to obtain the second depth map.

determining depth prediction values and associated information of a plurality of pixels in the first image based on the first image, wherein the associated information of the plurality of pixels indicates a degree of association between the plurality of pixels;

and updating the first depth map based on the depth predicted values and the associated information of the plurality of pixels to obtain a second depth map.

In a possible implementation manner, the updating the first depth map based on the depth prediction values and the associated information of the plurality of pixels to obtain a second depth map includes:

determining depth failure pixels in the first depth map;

obtaining a depth predicted value of the depth failure pixel and depth predicted values of a plurality of surrounding pixels of the depth failure pixel from the depth predicted values of the plurality of pixels;

acquiring association degrees between the depth failure pixel and a plurality of surrounding pixels of the depth failure pixel from association information of the plurality of pixels;

determining an updated depth value of the depth failure pixel based on the depth prediction value of the depth failure pixel, the depth prediction values of a plurality of surrounding pixels of the depth failure pixel, and the correlation between the depth failure pixel and the surrounding pixels of the depth failure pixel.

In one possible implementation, the determining an updated depth value of the depth failure pixel based on the depth prediction value of the depth failure pixel, the depth prediction values of a plurality of surrounding pixels of the depth failure pixel, and the association between the depth failure pixel and the plurality of surrounding pixels of the depth failure pixel includes:

determining a depth correlation value of the depth failure pixel based on depth predicted values of surrounding pixels of the depth failure pixel and correlation degrees between the depth failure pixel and a plurality of surrounding pixels of the depth failure pixel;

and determining an updated depth value of the depth failure pixel based on the depth predicted value and the depth related value of the depth failure pixel.

In a possible implementation manner, the determining a depth-related value of the depth-failed pixel based on the depth prediction values of the surrounding pixels of the depth-failed pixel and the association degree between the depth-failed pixel and a plurality of surrounding pixels of the depth-failed pixel includes:

and taking the association degree between the depth failure pixel and each surrounding pixel as the weight of each surrounding pixel, and performing weighted summation processing on the depth predicted values of a plurality of surrounding pixels of the depth failure pixel to obtain the depth association value of the depth failure pixel.

In one possible implementation, the determining depth prediction values of a plurality of pixels in the first image based on the first image includes:

based on the first image and the first depth map, depth predictors for a plurality of pixels in the first image are determined.

In one possible implementation, the determining depth prediction values of a plurality of pixels in the first image based on the first image and the first depth map includes:

and inputting the first image and the first depth map into a depth prediction neural network for processing to obtain depth prediction values of a plurality of pixels in the first image.

performing fusion processing on the first image and the first depth map to obtain a fusion result;

based on the fusion result, depth predicted values of a plurality of pixels in the first image are determined.

In a possible implementation manner, the determining, based on the first image, associated information of a plurality of pixels in the first image includes:

and inputting the first image into a correlation degree detection neural network for processing to obtain correlation information of a plurality of pixels in the first image.

In one possible implementation, the updating the first depth map based on the first image includes:

acquiring an image of the target object from the first image;

updating the first depth map based on the image of the target object.

In one possible implementation, the acquiring the image of the target object from the first image includes:

acquiring key point information of the target object in the first image;

and acquiring an image of the target object from the first image based on the key point information of the target object.

In a possible implementation manner, the acquiring of the key point information of the target object in the first image includes:

carrying out target detection on the first image to obtain a region where the target object is located;

and detecting key points of the image of the area where the target object is located to obtain key point information of the target object in the first image.

acquiring a depth map of the target object from the first depth map;

and updating the depth map of the target object based on the first image to obtain the second depth map.

In one possible implementation, the determining a living body detection result of the target object based on the first image and the second depth map includes:

and inputting the first image and the second depth map into a living body detection neural network for processing to obtain a living body detection result of the target object.

performing feature extraction processing on the first image to obtain first feature information;

performing feature extraction processing on the second depth map to obtain second feature information;

determining a living body detection result of the target object based on the first feature information and the second feature information.

In one possible implementation manner, the determining a living body detection result of the target object based on the first feature information and the second feature information includes:

performing fusion processing on the first characteristic information and the second characteristic information to obtain third characteristic information;

determining a living body detection result of the target object based on the third feature information.

In one possible implementation manner, the determining a living body detection result of the target object based on the third feature information includes:

obtaining the probability that the target object is a living body based on the third characteristic information;

and determining the living body detection result of the target object according to the probability that the target object is a living body.

In one possible implementation, after the face recognition based on the first image, the method further includes:

and responding to the failure of face recognition, activating a password unlocking module arranged on the vehicle to start a password unlocking process.

In one possible implementation, the method further includes one or both of:

registering the car owner according to the face image of the car owner collected by the image collection module;

and carrying out remote registration according to the face image of the vehicle owner collected by the terminal equipment of the vehicle owner, and sending registration information to the vehicle, wherein the registration information comprises the face image of the vehicle owner.

According to a second aspect of the present disclosure, there is provided a vehicle door unlocking method including:

responding to the Bluetooth equipment with the preset identification, and awakening and controlling an image acquisition module arranged on the vehicle to acquire a first image of a target object;

performing face recognition based on the first image;

the response is searched the bluetooth equipment of preset sign, awakens and control set up in the first image of the image acquisition module collection target object of car, includes:

and responding to the Bluetooth equipment which is searched to any one preset identification, and awakening and controlling an image acquisition module arranged on the vehicle to acquire a first image of the target object.

awakening a face recognition module arranged on the vehicle;

the face recognition based on the first image comprises:

determining depth failure pixels in the first depth map;

acquiring an image of the target object from the first image;

updating the first depth map based on the image of the target object.

acquiring key point information of the target object in the first image;

acquiring a depth map of the target object from the first depth map;

In one possible implementation, the method further includes one or both of:

According to a third aspect of the disclosure, there is provided a vehicle door unlocking device including:

the searching module is used for searching the Bluetooth equipment with the preset identification through a Bluetooth module arranged on the vehicle;

the awakening module is used for responding to the Bluetooth device with the preset identifier, establishing Bluetooth pairing connection between the Bluetooth module and the Bluetooth device with the preset identifier, awakening and controlling an image acquisition module arranged on the vehicle to acquire a first image of a target object in response to successful Bluetooth pairing connection, or awakening and controlling the image acquisition module arranged on the vehicle to acquire the first image of the target object in response to the Bluetooth device with the preset identifier;

the face recognition module is used for carrying out face recognition based on the first image;

and the unlocking module is used for responding to the success of the face recognition and sending a vehicle door unlocking instruction to at least one vehicle door lock of the vehicle.

In one possible implementation, the search module is configured to:

the wake-up module is configured to:

and responding to the Bluetooth device searched for any preset identifier, establishing Bluetooth pairing connection between the Bluetooth module and the Bluetooth device of the preset identifier, or responding to the Bluetooth device searched for any preset identifier, awakening and controlling an image acquisition module arranged on the vehicle to acquire a first image of a target object.

In one possible implementation, the wake-up module includes:

the awakening sub-module is used for awakening a face recognition module arranged on the vehicle;

and the control sub-module is used for controlling the image acquisition module to acquire a first image of the target object through the awakened face recognition module.

In one possible implementation, the apparatus further includes:

the first control module is used for controlling the face recognition module to enter a dormant state if the face image is not collected within the preset time.

In one possible implementation, the apparatus further includes:

and the second control module is used for controlling the face recognition module to enter a dormant state if the face recognition module fails within the preset time.

In one possible implementation, the unlocking module is configured to:

the face recognition module includes:

the face authentication module is used for acquiring the first image through an image sensor in the image acquisition module and performing face authentication based on the first image and pre-registered face features;

and the living body detection module is used for acquiring a first depth map corresponding to the first image through a depth sensor in the image acquisition module and carrying out living body detection on the basis of the first image and the first depth map.

In one possible implementation, the liveness detection module includes:

the updating submodule is used for updating the first depth map based on the first image to obtain a second depth map;

a determination sub-module for determining a live detection result of the target object based on the first image and the second depth map.

In one possible implementation, the update submodule is configured to:

determining depth failure pixels in the first depth map;

In one possible implementation, the update submodule is configured to:

acquiring an image of the target object from the first image;

updating the first depth map based on the image of the target object.

In one possible implementation, the update submodule is configured to:

acquiring key point information of the target object in the first image;

In one possible implementation, the update submodule is configured to:

acquiring a depth map of the target object from the first depth map;

In one possible implementation, the determining sub-module is configured to:

In one possible implementation, the apparatus further includes:

and the activation and starting module is used for responding to the failure of face recognition, and activating a password unlocking module arranged on the vehicle to start a password unlocking process.

In one possible implementation, the apparatus further includes a registration module, and the registration module is configured to one or both of:

According to a fourth aspect of the present disclosure, there is provided a vehicle-mounted face unlocking system, including: the system comprises a memory, a face recognition module, an image acquisition module and a Bluetooth module; the face recognition module is respectively connected with the memory, the image acquisition module and the Bluetooth module; the Bluetooth module comprises a microprocessor and a Bluetooth sensor, wherein the microprocessor wakes up the face recognition module when the Bluetooth pairing connection with the Bluetooth equipment with a preset identifier is successful or the Bluetooth equipment with the preset identifier is searched; the face recognition module is also provided with a communication interface used for being connected with the vehicle door domain controller, and if face recognition is successful, control information used for unlocking the vehicle door is sent to the vehicle door domain controller based on the communication interface.

In one possible implementation, the image acquisition module comprises an image sensor and a depth sensor.

In one possible implementation, the depth sensor includes a binocular infrared sensor, and two infrared cameras of the binocular infrared sensor are disposed at two sides of a camera of the image sensor.

In a possible implementation manner, the image acquisition module further comprises at least one light supplement lamp, the at least one light supplement lamp is arranged between the infrared camera of the binocular infrared sensor and the camera of the image sensor, and the at least one light supplement lamp comprises at least one of the light supplement lamp of the image sensor and the light supplement lamp of the depth sensor.

In a possible implementation manner, the image acquisition module further comprises a laser, and the laser is arranged between the camera of the depth sensor and the camera of the image sensor.

In a possible implementation manner, the vehicle-mounted human face unlocking system further includes: and the password unlocking module is used for unlocking the vehicle door and is connected with the face recognition module.

In one possible implementation manner, the password unlocking module includes one or both of a touch screen and a keyboard.

In a possible implementation manner, the vehicle-mounted human face unlocking system further includes: and the battery module is respectively connected with the microprocessor and the face recognition module.

According to a fifth aspect of the present disclosure, there is provided a vehicle comprising a vehicle face unlocking system according to any one of claims 91 to 98, connected to a door domain controller of the vehicle.

In one possible implementation, the image acquisition module is disposed outside the vehicle.

In a possible implementation manner, the image acquisition module is arranged in at least one of the following positions: the B post of car, at least one door, at least one rear-view mirror.

In a possible implementation manner, the face recognition module is arranged in the vehicle and is connected with the vehicle door domain controller through a CAN bus.

According to a sixth aspect of the present disclosure, there is provided an electronic device comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: the method of the first aspect is performed.

According to a seventh aspect of the present disclosure, there is provided an electronic apparatus comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: the method of the second aspect described above is performed.

According to an eighth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method of the first aspect described above.

According to a ninth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method of the second aspect described above.

In the embodiment of the disclosure, the bluetooth pairing connection between the bluetooth module and the bluetooth device with the preset identifier is established in response to the search of the bluetooth device with the preset identifier, and the face recognition module is awakened and the image acquisition module is controlled to acquire the first image of the target object in response to the successful bluetooth pairing connection, so that the face recognition module is awakened based on the successful bluetooth pairing connection, the probability of awakening the face recognition module by mistake can be effectively reduced, and even the face recognition module can be prevented from being awakened by mistake, thereby improving the user experience and effectively reducing the power consumption of the face recognition module. In addition, compared with the technologies of ultrasonic wave, infrared and other short-distance sensors, the Bluetooth-based pairing connection mode has the advantages of high safety and large distance support. Practice shows that the time when a user of the Bluetooth device carrying the preset identifier arrives at the vehicle through the distance (the distance between the user and the vehicle when the Bluetooth pairing connection is successful) is approximately matched with the time when the face recognition module is awakened by the vehicle and is converted from a dormant state to a working state, so that when the user arrives at the vehicle door, the vehicle door can be immediately opened through face recognition by the awakened face recognition module without waiting for the face recognition module to be awakened after the user arrives at the vehicle door, the face recognition efficiency can be improved, and the user experience is improved. In addition, in the process of Bluetooth pairing connection, the user does not feel, so that the user experience can be further improved. Therefore, the embodiment of the disclosure provides a solution capable of better balancing various aspects such as power consumption saving, user experience and security of the face recognition module through a mode of successfully awakening the face recognition module based on Bluetooth pairing connection.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

Fig. 1 shows a flow chart of a vehicle door unlocking method according to an embodiment of the present disclosure.

Fig. 2 shows a schematic view of the B-pillar of the vehicle.

Fig. 3 is a schematic view showing the installation height and the recognizable height range of the door unlocking device in the door unlocking method according to the embodiment of the present disclosure.

Fig. 4a shows a schematic diagram of an image sensor and a depth sensor in a door unlocking method according to an embodiment of the present disclosure.

Fig. 4b shows another schematic diagram of an image sensor and a depth sensor in a vehicle door unlocking method according to an embodiment of the present disclosure.

Fig. 5 shows a schematic diagram of one example of a living body detection method according to an embodiment of the present disclosure.

Fig. 6 is a schematic diagram illustrating an example of determining a live body detection result of a target object in a first image based on the first image and a second depth map in the live body detection method according to the embodiment of the present disclosure.

Fig. 7 shows a schematic diagram of a depth-prediction neural network in a door unlocking method according to an embodiment of the present disclosure.

Fig. 8 shows a schematic diagram of a correlation detection neural network in a vehicle door unlocking method according to an embodiment of the present disclosure.

Fig. 9 illustrates an exemplary schematic diagram of depth map updating in a vehicle door unlocking method according to an embodiment of the disclosure.

Fig. 10 shows a schematic diagram of surrounding pixels in a door unlocking method according to an embodiment of the present disclosure.

Fig. 11 shows another schematic diagram of surrounding pixels in a door unlocking method according to an embodiment of the present disclosure.

FIG. 12 shows another flow chart of a method of unlocking a vehicle door according to an embodiment of the present disclosure.

Fig. 13 shows a block diagram of a vehicle door unlocking apparatus according to an embodiment of the present disclosure.

Fig. 14 shows a block diagram of a vehicle-mounted face unlocking system according to an embodiment of the present disclosure.

Fig. 15 shows a schematic diagram of a vehicle-mounted face unlocking system according to an embodiment of the present disclosure.

FIG. 16 shows a schematic view of a cart according to an embodiment of the present disclosure.

Fig. 17 is a block diagram illustrating an electronic device 800 in accordance with an example embodiment.

Detailed Description

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.

Fig. 1 shows a flow chart of a vehicle door unlocking method according to an embodiment of the present disclosure. The execution subject of the vehicle door unlocking method may be a vehicle door unlocking device. For example, the door unlocking device may be installed in at least one of the following positions: the automobile comprises a B column, at least one automobile door and at least one rearview mirror. Fig. 2 shows a schematic view of the B-pillar of the vehicle. For example, the door unlocking device may be installed at 130cm to 160cm from the B-pillar, and the horizontal recognition distance of the door unlocking device may be 30cm to 100cm, which is not limited herein. Fig. 3 is a schematic view showing the installation height and the recognizable height range of the door unlocking device in the door unlocking method according to the embodiment of the present disclosure. In the example shown in fig. 3, the door unlocking device is mounted at a height of 160cm, and a recognizable height range of 140cm to 190cm is provided.

In one possible implementation, the door unlocking method may be implemented by a processor calling computer readable instructions stored in a memory.

As shown in fig. 1, the door unlocking method includes steps S11 to S15.

In step S11, a bluetooth device of a preset identification is searched for via a bluetooth module provided in the vehicle.

In one possible implementation manner, a bluetooth device that searches for a preset identifier via a bluetooth module provided in a vehicle includes: when the vehicle is in a flameout state or in a flameout and vehicle door locking state, the Bluetooth device with the preset identifier is searched through the Bluetooth module arranged on the vehicle. In this implementation, need not to search for the bluetooth equipment of predetermineeing the sign through bluetooth module before the car is flame-out, perhaps, need not to search for the bluetooth equipment of predetermineeing the sign through bluetooth module before the car is flame-out and when the car is in flame-out state but the door is not in the lock state, can further reduce the consumption from this.

In one possible implementation, the Bluetooth module may be a Bluetooth Low Energy (BLE) module. In this implementation, when the vehicle is in a flameout state or in a flameout and door locked state, the bluetooth module may be in a broadcast mode, broadcasting one broadcast packet to the surroundings at regular intervals (e.g., 100 milliseconds). When the peripheral bluetooth devices execute the scanning action, if receiving the broadcast data packet broadcasted by the bluetooth module, the peripheral bluetooth devices send a scanning request to the bluetooth module, and the bluetooth module can respond to the scanning request and return a scanning response data packet to the bluetooth device sending the scanning request. In this implementation manner, if a scanning request sent by a bluetooth device with a preset identifier is received, it is determined that the bluetooth device with the preset identifier is searched.

In another possible implementation manner, when the vehicle is in a flameout state or in a flameout and vehicle door locking state, the bluetooth module may be in a scanning state, and if the bluetooth device with the preset identifier is scanned, the bluetooth device with the preset identifier is determined to be searched.

In one possible implementation, the bluetooth module and the face recognition module may be integrated into a face recognition system.

In another possible implementation, the bluetooth module may be independent of the face recognition system. That is, the bluetooth module may be disposed outside the face recognition system.

The embodiment of the present disclosure does not limit the maximum search distance of the bluetooth module, and in one example, the maximum search distance may be about 30 m.

In embodiments of the present disclosure, the identification of a bluetooth device may refer to a unique identifier of the bluetooth device. For example, the identification of the bluetooth device may be an ID, name, or address of the bluetooth device, etc.

In the embodiment of the present disclosure, the preset identifier may be an identifier of a device that is successfully paired with the bluetooth module of the vehicle in advance based on the bluetooth secure connection technology.

In the embodiment of the present disclosure, the number of the bluetooth devices of the preset identifier may be one or more. For example, if the identification of the bluetooth device is the ID of the bluetooth device, one or more bluetooth IDs authorized to open the vehicle door may be preset. For example, in the case that the number of the bluetooth devices with the preset identifier is one, the bluetooth device with the preset identifier may be a bluetooth device of an owner; under the condition that the number of the bluetooth devices with the preset identifiers is multiple, the bluetooth devices with the preset identifiers can comprise bluetooth devices of car owners, and bluetooth devices of family members, friends and pre-registered contacts of the car owners. The pre-registered contact may be a pre-registered courier or property worker, etc.

In the embodiments of the present disclosure, the bluetooth device may be any mobile device having bluetooth function, for example, the bluetooth device may be a mobile phone, a wearable device, or an electronic key. Wherein, wearable equipment can be intelligent bracelet or intelligent glasses etc..

In step S12, in response to the bluetooth device with the preset identifier being searched, a bluetooth pairing connection between the bluetooth module and the bluetooth device with the preset identifier is established.

In a possible implementation manner, if the number of the bluetooth devices with the preset identifier is multiple, in response to searching for any bluetooth device with the preset identifier, a bluetooth pairing connection between the bluetooth module and the bluetooth device with the preset identifier is established.

In a possible implementation manner, in response to the bluetooth device with the preset identifier being searched, the bluetooth module performs identity authentication on the bluetooth device with the preset identifier, and after the identity authentication is passed, the bluetooth pairing connection between the bluetooth module and the bluetooth device with the preset identifier is established, so that the security of the bluetooth pairing connection can be improved.

In step S13, in response to the bluetooth pairing connection being successful, the image capturing module set in the vehicle is awakened and controlled to capture the first image of the target object.

In a possible implementation manner, waking up and controlling an image capturing module set in a vehicle to capture a first image of a target object includes: awakening a face recognition module arranged on the vehicle; the awakened face recognition module controls the image acquisition module to acquire a first image of the target object.

In the embodiment of the present disclosure, if the bluetooth device with the preset identifier is searched, it can be indicated to a great extent that a user (e.g., a car owner) carrying the bluetooth device with the preset identifier enters a search range of the bluetooth module. At the moment, the Bluetooth pairing connection between the Bluetooth module and the Bluetooth equipment with the preset identifier is established in response to the Bluetooth equipment with the preset identifier being searched, the face recognition module is awakened and the image acquisition module is controlled to acquire the first image of the target object in response to the successful Bluetooth pairing connection, and therefore the face recognition module is awakened based on the successful Bluetooth pairing connection, the probability of awakening the face recognition module by mistake can be effectively reduced, even the face recognition module can be prevented from being awakened by mistake, the user experience can be improved, and the power consumption of the face recognition module can be effectively reduced. In addition, compared with the technologies of ultrasonic wave, infrared and other short-distance sensors, the Bluetooth-based pairing connection mode has the advantages of high safety and large distance support. Practice shows that the time when a user of the Bluetooth device carrying the preset identifier arrives at the vehicle through the distance (the distance between the user and the vehicle when the Bluetooth pairing connection is successful) is approximately matched with the time when the face recognition module is awakened by the vehicle and is converted from a dormant state to a working state, so that when the user arrives at the vehicle door, the vehicle door can be immediately opened through face recognition by the awakened face recognition module without waiting for the face recognition module to be awakened after the user arrives at the vehicle door, the face recognition efficiency can be improved, and the user experience is improved. In addition, in the process of Bluetooth pairing connection, the user does not feel, so that the user experience can be further improved. Therefore, the embodiment of the disclosure provides a solution capable of better balancing various aspects such as power consumption saving, user experience and security of the face recognition module through a mode of successfully awakening the face recognition module based on Bluetooth pairing connection.

In a possible implementation manner, after waking up the face recognition module set in the vehicle, the method further includes: and if the face image is not collected within the preset time, controlling the face recognition module to enter a dormant state. According to the implementation mode, when the face image is not collected within the preset time after the face recognition module is awakened, the face recognition module is controlled to enter the dormant state, and therefore power consumption can be reduced.

In a possible implementation manner, after waking up the face recognition module set in the vehicle, the method further includes: and if the face recognition module fails to pass the face recognition within the preset time, controlling the face recognition module to enter a dormant state. This implementation mode is through when not passing through face identification in the time of predetermineeing after awakening up face identification module, and control face identification module gets into dormant state, can reduce the consumption from this.

In step S14, face recognition is performed based on the first image.

In one possible implementation, the face recognition includes: living body detection and face authentication; carrying out face recognition based on the first image, comprising: acquiring a first image through an image sensor in an image acquisition module, and performing face authentication based on the first image and pre-registered face features; and acquiring a first depth map corresponding to the first image through a depth sensor in the image acquisition module, and carrying out in-vivo detection based on the first image and the first depth map.

In an embodiment of the disclosure, the first image contains a target object. The target object may be a human face or at least a part of a human body, which is not limited in this disclosure.

The first image may be a still image or a video frame image. For example, the first image may be an image selected from a video sequence, wherein the image may be selected from the video sequence in a variety of ways. In a specific example, the first image is an image selected from a video sequence, and the predetermined quality condition is satisfied, and the predetermined quality condition may include one or any combination of the following: whether the target object is included, whether the target object is located in the central area of the image, whether the target object is completely included in the image, a ratio of the target object in the image, a state of the target object (e.g., a face angle), an image sharpness, an image exposure, and the like, which are not limited in this disclosure.

In one example, the living body detection may be performed first and then the face authentication may be performed. For example, if the living body detection result of the target object is that the target object is a living body, triggering a face authentication process; and if the living body detection result of the target object is that the target object is a prosthesis, the human face authentication process is not triggered.

In another example, face authentication may be performed first followed by liveness detection. For example, if the face authentication passes, a living body detection process is triggered; and if the face authentication fails, not triggering the living body detection process.

In another example, the living body detection and the face authentication may be performed simultaneously.

In this implementation, the living body detection is used to verify whether or not the target object is a living body, and may be used to verify whether or not the target object is a human body, for example. The face authentication is used for extracting face features in the acquired image, comparing the face features in the acquired image with pre-registered face features, and judging whether the face features belong to the face features of the same person, for example, whether the face features in the acquired image belong to the face features of a vehicle owner can be judged.

In the disclosed embodiments, the depth sensor represents a sensor for collecting depth information. The working principle and the working wave band of the depth sensor are not limited by the embodiment of the disclosure.

In the embodiment of the disclosure, the image sensor and the depth sensor of the image acquisition module can be separately arranged or can be arranged together. For example, the image sensor and the depth sensor of the image acquisition module may be separately arranged, the image sensor may be an RGB (Red, Red; Green, Green; Blue, Blue) sensor or an infrared sensor, and the depth sensor may be a binocular infrared sensor or a TOF (Time of Flight) sensor; the image sensor and the depth sensor of the image acquisition module are arranged together, and the image acquisition module adopts an RGBD (Red, Green, Blue, Deep) sensor to realize the functions of the image sensor and the depth sensor.

As one example, the image sensor is an RGB sensor. If the image sensor is an RGB sensor, the image acquired by the image sensor is an RGB image.

As another example, the image sensor is an infrared sensor. If the image sensor is an infrared sensor, the image acquired by the image sensor is an infrared image. The infrared image can be an infrared image with light spots or an infrared image without light spots.

In other examples, the image sensor may be other types of sensors, which are not limited by the embodiments of the present disclosure.

Alternatively, the door unlocking device may acquire the first image in various ways. For example, in some embodiments, a camera is disposed on the vehicle door unlocking device, and the vehicle door unlocking device acquires a still image or a video stream through the camera to obtain a first image.

As one example, the depth sensor is a three-dimensional sensor. For example, the depth sensor is a binocular infrared sensor, which includes two infrared cameras, a time of flight TOF sensor, or a structured light sensor. The structured light sensor may be an encoded structured light sensor or a speckle structured light sensor. The depth sensor acquires the depth map of the target object, and the high-precision depth map can be obtained. The depth map containing the target object is utilized for live body detection, the depth information of the target object can be fully mined, and therefore the accuracy of the live body detection can be improved. For example, when the target object is a human face, the depth map including the human face is used for live body detection, so that the depth information of the human face data can be sufficiently mined, and the accuracy of live body human face detection can be improved.

In one example, the TOF sensor employs a TOF module based on the infrared band. In this example, by employing a TOF module based on an infrared band, the influence of external light on depth map shooting can be reduced.

In an embodiment of the disclosure, the first depth map corresponds to the first image. For example, the first depth map and the first image are acquired by a depth sensor and an image sensor, respectively, for the same scene, or the first depth map and the first image are acquired by a depth sensor and an image sensor for the same target area at the same time, but the embodiment of the present disclosure does not limit this.

Fig. 4a shows a schematic diagram of an image sensor and a depth sensor in a door unlocking method according to an embodiment of the present disclosure. In the example shown in fig. 4a, the image sensor is an RGB sensor, the camera of the image sensor is an RGB camera, and the depth sensor is a binocular infrared sensor, and the depth sensor includes two Infrared (IR) cameras disposed at both sides of the RGB camera of the image sensor. Wherein, two infrared cameras gather the depth information based on binocular parallax principle.

In one example, the image acquisition module still includes at least one light filling lamp, and this at least one light filling lamp sets up between binocular infrared sensor's infrared camera and image sensor's camera, and this at least one light filling lamp is including at least one in the light filling lamp that is used for image sensor and the light filling lamp that is used for depth sensor. For example, if the image sensor is an RGB sensor, the fill-in light for the image sensor may be a white light; if the image sensor is an infrared sensor, the light supplement lamp for the image sensor can be an infrared lamp; if the depth sensor is a binocular infrared sensor, the light supplement lamp for the depth sensor may be an infrared lamp. In the example shown in fig. 4a, an infrared lamp is provided between the infrared camera of the binocular infrared sensor and the camera of the image sensor. For example, the infrared lamp may use 940nm infrared light.

In one example, the fill light may be in a normally open mode. In this example, when the camera of image acquisition module was in operating condition, the light filling lamp was in the on-state.

In another example, the fill light may be turned on when light is insufficient. For example, the ambient light intensity can be obtained through an ambient light sensor, and when the ambient light intensity is lower than a light intensity threshold value, it is determined that the light is insufficient, and a light supplement lamp is turned on.

Fig. 4b shows another schematic diagram of an image sensor and a depth sensor in a vehicle door unlocking method according to an embodiment of the present disclosure. In the example shown in fig. 4b, the image sensor is an RGB sensor, the camera of the image sensor is an RGB camera, and the depth sensor is a TOF sensor.

In one example, the image capture module further comprises a laser disposed between the camera of the depth sensor and the camera of the image sensor. For example, the laser is disposed between the camera of the TOF sensor and the camera of the RGB sensor. For example, the Laser may be a VCSEL (Vertical Cavity Emitting Laser), and the TOF sensor may acquire a depth map based on Laser light emitted from the VCSEL.

In an embodiment of the disclosure, a depth sensor is used to acquire a depth map and an image sensor is used to acquire a two-dimensional image. It should be noted that, although the image sensor is described by taking an RGB sensor and an infrared sensor as examples, and the depth sensor is described by taking a binocular infrared sensor, a TOF sensor and a structured light sensor as examples, those skilled in the art can understand that the disclosed embodiments should not be limited thereto. The type of the image sensor and the type of the depth sensor can be selected by those skilled in the art according to the actual application requirements, as long as the two-dimensional image and the depth map can be acquired respectively.

In step S15, in response to the face recognition being successful, a door unlock instruction is sent to at least one door lock of the vehicle.

The vehicle door in the embodiment of the present disclosure may include a vehicle door (e.g., a left front door, a right front door, a left rear door, and a right rear door) through which a person enters and exits, and may also include a trunk door of a vehicle.

In one possible implementation, in response to the face recognition being successful, sending a door unlock instruction to at least one door lock of the vehicle, including: responding to the success of the face recognition, and determining the vehicle door of which the target object has the door opening authority; and sending a door unlocking instruction to at least one door lock of the vehicle according to the door of the target object with the door opening authority. For example, the doors of the target object having the door opening authority may be all doors, or may be trunk doors.

As an example, if the target object is a courier, the door of the target object having the door opening authority may be a trunk door. If the vehicle door of the target object with the door opening authority is the trunk door, a vehicle door unlocking instruction can be sent to the trunk door lock.

In one example, if the vehicle door of the target object having the door opening authority only includes a trunk door, the vehicle door closing instruction may be sent to the trunk door lock after the preset time of the vehicle door unlocking instruction is sent to the trunk door lock, for example, the preset time may be 3 minutes. For example, the door of the courier with the door opening authority only comprises a trunk door, and then the door opening instruction can be sent to the trunk door lock after 3 minutes of sending the door unlocking instruction, so that the requirement that the courier places the express in the trunk can be met, and the safety of the vehicle can be improved.

In one example, the SoC of the door unlocking device may send a door unlocking instruction to the door domain controller to control the door to unlock.

In one possible implementation, the live body detection based on the first image and the first depth map includes: updating the first depth map based on the first image to obtain a second depth map; based on the first image and the second depth map, a living body detection result of the target object is determined.

Specifically, based on the first image, the depth values of one or more pixels in the first depth map are updated to obtain the second depth map.

In some embodiments, the depth values of the depth-failing pixels in the first depth map are updated based on the first image, resulting in the second depth map.

The depth failure pixel in the depth map may refer to a pixel included in the depth map, in which the depth value is invalid, that is, a pixel whose depth value is inaccurate or significantly inconsistent with the actual situation. The number of depth failure pixels may be one or more. By updating the depth value of at least one depth failure pixel in the depth map, the depth value of the depth failure pixel is more accurate, and the accuracy of in-vivo detection is improved.

In some embodiments, the first depth map is a depth map with missing values, and the second depth map is obtained by repairing the first depth map based on the first image, wherein optionally the repairing of the first depth map comprises determining or supplementing depth values of pixels with missing values, but the disclosed embodiments are not limited thereto.

In the disclosed embodiments, the first depth map may be updated or repaired in a variety of ways. In some embodiments, the liveness detection is performed directly with the first image, for example updating the first depth map directly with the first image. In other embodiments, the first image is pre-processed and the in vivo test is performed based on the pre-processed first image. For example, an image of the target object is acquired from the first image, and the first depth map is updated based on the image of the target object.

The image of the target object may be intercepted from the first image in a number of ways. As an example, object detection is performed on a first image, position information of an object, for example, position information of a bounding box (bounding box) of the object, is obtained, and an image of the object is cut out from the first image based on the position information of the object. For example, an image of an area where a bounding box of the target object is located is cut out from the first image as an image of the target object, and for example, the bounding box of the target object is enlarged by a certain factor and the enlarged image of the area where the bounding box is located is cut out from the first image as an image of the target object. As another example, keypoint information of a target object in a first image is acquired, and an image of the target object is acquired from the first image based on the keypoint information of the target object.

Optionally, performing target detection on the first image to obtain position information of a region where the target object is located; and detecting key points of the image of the area where the target object is located to obtain key point information of the target object in the first image.

Alternatively, the key point information of the target object may include position information of a plurality of key points of the target object. If the target object is a human face, the key points of the target object may include one or more of eye key points, eyebrow key points, nose key points, mouth key points, human face contour key points, and the like. Wherein, the eye key points may comprise one or more of eye contour key points, eye corner key points, pupil key points and the like.

In one example, a contour of the target object is determined based on the keypoint information of the target object, and an image of the target object is cut from the first image according to the contour of the target object. Compared with the position information of the target object obtained through target detection, the position of the target object obtained through the key point information is more accurate, and therefore the accuracy of subsequent living body detection is improved.

Alternatively, the contour of the target object in the first image may be determined based on the key point of the target object in the first image, and an image of a region where the contour of the target object in the first image is located or an image of a region enlarged by a certain multiple may be determined as the image of the target object. For example, an elliptical region determined based on the key point of the target object in the first image may be determined as the image of the target object, or a minimum circumscribed rectangular region of the elliptical region determined based on the key point of the target object in the first image may be determined as the image of the target object, but the embodiment of the present disclosure does not limit this.

In this way, by acquiring the image of the target object from the first image and performing the live body detection based on the image of the target object, it is possible to reduce interference of the background information in the first image with respect to the live body detection.

In this embodiment of the present disclosure, the obtained original depth map may be updated, or in some embodiments, a depth map of the target object is obtained from the first depth map, and the depth map of the target object is updated based on the first image, so as to obtain the second depth map.

As an example, position information of a target object in a first image is acquired, and a depth map of the target object is acquired from a first depth map based on the position information of the target object. Optionally, the first depth map and the first image may be registered or aligned in advance, but this is not limited by the embodiment of the present disclosure.

In this way, the second depth map is obtained by acquiring the depth map of the target object from the first depth map and updating the depth map of the target object based on the first image, so that the interference of the background information in the first depth map on the living body detection can be reduced.

In some embodiments, after the first image and the first depth map corresponding to the first image are acquired, the first image and the first depth map are aligned according to the parameters of the image sensor and the parameters of the depth sensor.

As an example, the first depth map may be subjected to a conversion process such that the first depth map after the conversion process and the first image are aligned. For example, a first conversion matrix may be determined based on parameters of the depth sensor and parameters of the image sensor, and the first depth map may be subjected to conversion processing based on the first conversion matrix. Accordingly, at least a part of the first depth map after the conversion processing may be updated based on at least a part of the first image, resulting in a second depth map. For example, the first depth map after the conversion process is updated based on the first image, and a second depth map is obtained. For another example, based on the image of the target object captured from the first image, the depth map of the target object captured from the first depth map is updated to obtain a second depth map, and so on.

As another example, the first image may be subjected to a conversion process such that the converted first image is aligned with the first depth map. For example, the second conversion matrix may be determined based on parameters of the depth sensor and parameters of the image sensor, and the conversion process may be performed on the first image based on the second conversion matrix. Accordingly, at least a portion of the first depth map may be updated based on at least a portion of the first image after the conversion process, resulting in a second depth map.

Optionally, the parameters of the depth sensor may include intrinsic and/or extrinsic parameters of the depth sensor, and the parameters of the image sensor may include intrinsic and/or extrinsic parameters of the image sensor. By aligning the first depth map and the first image, the positions of the corresponding parts in the first depth map and the first image can be made the same in both images.

In the above example, the first image is an original image (e.g., an RGB or infrared image), and in other embodiments, the first image may also refer to an image of a target object captured from the original image, and similarly, the first depth map may also refer to a depth map of the target object captured from the original depth map, which is not limited by the embodiment of the present disclosure.

Fig. 5 shows a schematic diagram of one example of a living body detection method according to an embodiment of the present disclosure. In the example shown in fig. 5, the first image is an RGB image, the target object is a face, the RGB image and the first depth map are subjected to alignment correction, the processed image is input into the face keypoint model for processing, an RGB face map (image of the target object) and a depth face map (depth map of the target object) are obtained, and the depth face map is updated or repaired based on the RGB face map. Therefore, subsequent data processing amount can be reduced, and the living body detection efficiency and accuracy can be improved.

In the embodiment of the present disclosure, the living body detection result of the target object may be that the target object is a living body or that the target object is a prosthesis.

In some embodiments, the first image and the second depth map are input to a living body detection neural network for processing, and a living body detection result of the target object in the first image is obtained. Or processing the first image and the second depth map through other living body detection algorithms to obtain a living body detection result.

In some embodiments, the first image is subjected to feature extraction processing to obtain first feature information; performing feature extraction processing on the second depth map to obtain second feature information; based on the first feature information and the second feature information, a living body detection result of the target object in the first image is determined.

Optionally, the feature extraction processing may be implemented by a neural network or other machine learning algorithms, and the type of the extracted feature information may optionally be obtained by learning the sample, which is not limited in this embodiment of the disclosure.

In some specific scenarios (such as outdoor strong light scenarios), the acquired depth map (for example, the depth map acquired by the depth sensor) may have a partial area failure. In addition, under normal illumination, the depth map is also randomly disabled due to factors such as glasses reflection, black hair or black glasses frame. And certain special paper can cause the printed face photo to generate similar effects of large-area failure or partial failure of the depth map. In addition, the depth map can be partially disabled by shielding the active light source of the depth sensor, and the image of the prosthesis on the image sensor is normal. Therefore, in the event of partial or total failure of some depth maps, using the depth maps to distinguish between living and prosthetic bodies can cause errors. Therefore, in the embodiment of the disclosure, the first depth map is repaired or updated, and the repaired or updated depth map is used for performing the in vivo detection, which is beneficial to improving the accuracy of the in vivo detection.

In this example, the first image and the second depth map are input into a living body detection network to perform living body detection processing, resulting in a living body detection result.

As shown in fig. 6, the living body detection network includes two branches, namely a first sub-network and a second sub-network, wherein the first sub-network is used for performing feature extraction processing on the first image to obtain first feature information, and the second sub-network is used for performing feature extraction processing on the second depth map to obtain second feature information.

In one optional example, the first sub-network may include a convolutional layer, a downsampling layer, and a fully-connected layer.

For example, the first sub-network may include one level of convolutional layers, one level of downsampling layers, and one level of fully-connected layers. Wherein the level of convolutional layers may comprise one or more convolutional layers, the level of downsampling layers may comprise one or more downsampling layers, and the level of fully-connected layers may comprise one or more fully-connected layers.

As another example, the first subnetwork may include a multi-level convolutional layer, a multi-level downsampling layer, and a one-level fully-connected layer. Wherein each level of convolutional layer may comprise one or more convolutional layers, each level of downsampling layer may comprise one or more downsampling layers, and the level of fully-connected layers may comprise one or more fully-connected layers. The depth prediction neural network comprises an i-th stage convolutional layer, an i-th stage downsampling layer, an i + 1-th stage convolutional layer and a full-connection layer, wherein the i-th stage convolutional layer is cascaded with the i-th stage downsampling layer, the i-th stage downsampling layer is cascaded with the i + 1-th stage convolutional layer, the n-th stage downsampling layer is cascaded with the full-connection layer, i and n are positive integers, i is more than or equal to 1 and less than or equal to n, and n represents the stage number of the convolutional layer and the downsampling layer in the depth prediction neural network.

Alternatively, the first sub-network may include a convolutional layer, a downsampling layer, a normalization layer, and a fully-connected layer.

For example, the first sub-network may include one convolution layer, one normalization layer, one down-sampling layer, and one fully-connected layer. Wherein the level of convolutional layers may comprise one or more convolutional layers, the level of downsampling layers may comprise one or more downsampling layers, and the level of fully-connected layers may comprise one or more fully-connected layers.

As another example, the first subnetwork may include a multi-level convolutional layer, a plurality of normalization layers, and a multi-level downsampling layer and a one-level fully-connected layer. Wherein each level of convolutional layer may comprise one or more convolutional layers, each level of downsampling layer may comprise one or more downsampling layers, and the level of fully-connected layers may comprise one or more fully-connected layers. The i-th normalization layer is cascaded after the i-th convolution layer, the i-th down-sampling layer is cascaded after the i-th normalization layer, the i + 1-th convolution layer is cascaded after the i-th down-sampling layer, and the full-connection layer is cascaded after the n-th down-sampling layer, wherein i and n are positive integers, i is more than or equal to 1 and less than or equal to n, and n represents the number of stages of the convolution layers and the down-sampling layers in the first sub-network and the number of the normalization layers.

As an example, performing convolution processing on a first image to obtain a first convolution result; performing downsampling processing on the first convolution result to obtain a first downsampling result; and obtaining first characteristic information based on the first downsampling result.

For example, the first image may be subjected to convolution processing and downsampling processing by one convolution layer and one downsampling layer. Wherein the level of convolutional layers may comprise one or more convolutional layers, and the level of downsampling layers may comprise one or more downsampling layers.

As another example, the first image may be convolved and downsampled by multiple convolutional layers and multiple downsampling layers. Wherein each level of convolutional layer may comprise one or more convolutional layers, and each level of downsampling layer may comprise one or more downsampling layers.

For example, the downsampling the first convolution result to obtain the first downsampled result may include: carrying out normalization processing on the first convolution result to obtain a first normalization result; and performing downsampling processing on the first normalization result to obtain a first downsampling result.

For example, the first downsampling result may be input into the full-link layer, and the first downsampling result is fused by the full-link layer to obtain the first feature information.

Optionally, the second sub-network and the first sub-network have the same network structure but different parameters. Alternatively, the second sub-network has a different network structure from the first sub-network, which is not limited in this disclosure.

As shown in fig. 6, the living body detection network further includes a third sub-network, which is used for processing the first feature information obtained by the first sub-network and the second feature information obtained by the second sub-network to obtain a living body detection result of the target object in the first image. Optionally, the third sub-network may comprise a fully connected layer and an output layer. For example, the output layer uses a softmax function, and if the output of the output layer is 1, the target object is a living body, and if the output of the output layer is 0, the target object is a prosthetic body.

As an example, the first feature information and the second feature information are subjected to fusion processing to obtain third feature information; based on the third feature information, a living body detection result of the target object in the first image is determined.

For example, the first feature information and the second feature information are fused by the full connection layer to obtain third feature information.

In some embodiments, based on the third feature information, a probability that the target object in the first image is a living body is obtained, and a living body detection result of the target object is determined according to the probability that the target object is a living body.

For example, if the probability that the target object is a living body is greater than the second threshold value, the living body detection result of the target object is determined as that the target object is a living body. For another example, if the probability that the target object is a living body is less than or equal to the second threshold, the living body detection result of the target object is determined to be a prosthesis.

In other embodiments, based on the third characteristic information, a probability that the target object is a prosthesis is obtained, and the in-vivo detection result of the target object is determined according to the probability that the target object is the prosthesis. For example, if the probability that the target object is a prosthesis is greater than the third threshold, it is determined that the living body detection result of the target object is that the target object is a prosthesis. For another example, if the probability that the target object is a prosthesis is less than or equal to the third threshold, the living body detection result of the target object is determined to be a living body.

In one example, the third feature information may be input into a Softmax layer, and the probability that the target object is a living body or a prosthesis is obtained through the Softmax layer. For example, the output of the Softmax layer includes two neurons, where one neuron represents the probability that the target object is a living body and the other neuron represents the probability that the target object is a prosthesis, but the embodiments of the present disclosure are not limited thereto.

In the embodiment of the disclosure, the first depth map corresponding to the first image and the first image is acquired, the first depth map is updated based on the first image to obtain the second depth map, and the living body detection result of the target object in the first image is determined based on the first image and the second depth map, so that the depth map can be improved, and the accuracy of the living body detection is improved.

In a possible implementation manner, updating the first depth map based on the first image to obtain the second depth map includes: determining depth prediction values and associated information of a plurality of pixels in the first image based on the first image, wherein the associated information of the plurality of pixels indicates a degree of association among the plurality of pixels; and updating the first depth map based on the depth predicted values and the associated information of the plurality of pixels to obtain a second depth map.

Specifically, depth prediction values of a plurality of pixels in the first image are determined based on the first image, and the first depth map is repaired based on the depth prediction values of the plurality of pixels.

Specifically, depth predicted values of a plurality of pixels in the first image are obtained by processing the first image. For example, the first image is input into the depth prediction depth network to be processed, so as to obtain depth prediction results of a plurality of pixels, for example, a depth prediction map corresponding to the first image is obtained, but this is not limited by the embodiment of the present disclosure.

In some embodiments, based on the first image and the first depth map, depth predictors for a plurality of pixels in the first image are determined.

As an example, the first image and the first depth map are input to a depth prediction neural network for processing, so as to obtain depth prediction values of a plurality of pixels in the first image. Or, the first image and the first depth map are processed in other manners to obtain depth prediction values of a plurality of pixels, which is not limited in the embodiment of the present disclosure.

Fig. 7 shows a schematic diagram of a depth-prediction neural network in a door unlocking method according to an embodiment of the present disclosure. As shown in fig. 7, the first image and the first depth map may be input to a depth prediction neural network for processing, so as to obtain an initial depth estimation map. Based on the initial depth estimation map, depth predictors for a plurality of pixels in the first image may be determined. For example, the pixel values of the initial depth estimation map are depth prediction values of corresponding pixels in the first image.

The deep predictive neural network may be implemented by various network structures. In one example, a depth-predictive neural network includes an encoding portion and a decoding portion. Wherein, optionally, the encoding portion may comprise a convolutional layer and a downsampling layer, and the decoding portion comprises an inverse convolutional layer and/or an upsampling layer. In addition, the encoding portion and/or the decoding portion may further include a normalization layer, and the embodiment of the present disclosure does not limit specific implementations of the encoding portion and the decoding portion. In the encoding part, along with the increase of the number of network layers, the resolution ratio of the feature maps is gradually reduced, and the number of the feature maps is gradually increased, so that abundant semantic features and image space features can be obtained; in the decoding part, the resolution of the feature map is gradually increased, and the resolution of the feature map finally output by the decoding part is the same as the resolution of the first depth map.

In some embodiments, the first image and the first depth map are fused to obtain a fusion result, and depth prediction values of a plurality of pixels in the first image are determined based on the fusion result.

In one example, the first image and the first depth map may be concatenated (concat), resulting in a fused result.

In one example, the fusion result is subjected to convolution processing to obtain a second convolution result; performing downsampling processing based on the second convolution result to obtain a first coding result; based on the first encoding result, depth predictors for a plurality of pixels in the first image are determined.

For example, the fusion result may be convolved by a convolution layer to obtain a second convolution result.

For example, the second convolution result is normalized to obtain a second normalized result; and performing downsampling processing on the second normalization result to obtain a first coding result. Here, the normalization processing may be performed on the second convolution result by a normalization layer to obtain a second normalization result; and performing downsampling processing on the second normalization result through a downsampling layer to obtain a first coding result. Alternatively, the second convolution result may be downsampled by a downsampling layer to obtain the first encoding result.

For example, performing deconvolution processing on the first encoding result to obtain a first deconvolution result; and carrying out normalization processing on the first deconvolution result to obtain a depth predicted value. Here, the first encoding result may be deconvoluted by the deconvolution layer to obtain a first deconvolution result; and carrying out normalization processing on the first deconvolution result through a normalization layer to obtain a depth prediction value. Alternatively, the first coding result may be deconvoluted by a deconvolution layer to obtain a depth prediction value.

For example, the first encoding result is subjected to upsampling processing to obtain a first upsampling result; and carrying out normalization processing on the first up-sampling result to obtain a depth prediction value. Here, the first encoding result may be upsampled by an upsampling layer to obtain a first upsampling result; and carrying out normalization processing on the first up-sampling result through a normalization layer to obtain a depth prediction value. Or, the first coding result may be upsampled by an upsampling layer to obtain a depth prediction value.

Furthermore, by processing the first image, the associated information of the plurality of pixels in the first image is obtained. Wherein the association information of the plurality of pixels in the first image may include the degree of association between each of the plurality of pixels of the first image and its surrounding pixels. Wherein the surrounding pixels of the pixel may include at least one adjacent pixel of the pixel, or a plurality of pixels spaced from the pixel by no more than a certain value. For example, as shown in fig. 10, the surrounding pixels of the pixel 5 include a pixel 1, a pixel 2, a pixel 3, a pixel 4, a pixel 6, a pixel 7, a pixel 8, and a pixel 9 adjacent thereto, and accordingly, the association information of the plurality of pixels in the first image includes the association degrees between the pixel 1, the pixel 2, the pixel 3, the pixel 4, the pixel 6, the pixel 7, the pixel 8, and the pixel 9 and the pixel 5. As an example, the association degree between the first pixel and the second pixel may be measured by using a correlation between the first pixel and the second pixel, wherein the correlation between the pixels may be determined by using a correlation technique in the embodiments of the present disclosure, and details thereof are not repeated herein.

In the embodiments of the present disclosure, the association information of the plurality of pixels may be determined in various ways. In some embodiments, the first image is input to a relevance detection neural network for processing, and relevance information of a plurality of pixels in the first image is obtained. For example, a corresponding associated feature map of the first image is obtained. Alternatively, the associated information of the plurality of pixels may also be obtained through other algorithms, which is not limited in this disclosure.

Fig. 8 shows a schematic diagram of a correlation detection neural network in a vehicle door unlocking method according to an embodiment of the present disclosure. As shown in fig. 8, the first image is input to the relevance degree detecting neural network and processed to obtain a plurality of relevance feature maps. Based on the plurality of associated feature maps, associated information of a plurality of pixels in the first image may be determined. For example, if a pixel surrounding a certain pixel refers to a pixel whose distance from the certain pixel is equal to 0, that is, the pixel surrounding the certain pixel refers to a pixel adjacent to the certain pixel, the relevance detecting neural network may output 8 relevant feature maps. For example, in the first associated feature map, pixel P_i,jIs equal to the pixel value of the pixel P in the first image_i-1,j-1And a pixel P_i,jA degree of correlation between, wherein P_i,jA pixel representing the ith row and the jth column; in the second associated feature map, pixel P_i,jIs equal to the pixel value of the pixel P in the first image_i-1,jAnd a pixel P_i,jThe degree of association between; in the third associated feature map, pixel P_i,jIs equal to the pixel value of the pixel P in the first image_i-1,j+1And a pixel P_i,jThe degree of association between; in the fourth associated feature map, pixel P_i,jIs equal to the pixel value of the pixel P in the first image_i,j-1And a pixel P_i,jThe degree of association between; in the fifth associated feature map, pixel P_i,jIs equal to the pixel value of the pixel P in the first image_i,j+1And a pixel P_i,jThe degree of association between; in the sixth associated feature map, pixel P_i,jIs equal to the pixel value of the pixel P in the first image_i+1,j-1And a pixel P_i,jThe degree of association between; in the seventh associated feature map, pixel P_i,jIs equal to the pixel value of the pixel P in the first image_i+1,jAnd a pixel P_i,jThe degree of association between; in the eighth associated feature map, pixel P_i,jIs equal to the pixel value of the pixel P in the first image_i+1,j+1And a pixel P_i,jThe degree of association between them.

The relevancy detection neural network may be implemented by various network structures. As one example, the relevance detecting neural network may include an encoding portion and a decoding portion. Wherein the encoding portion may include a convolutional layer and a downsampling layer, and the decoding portion may include an inverse convolutional layer and/or an upsampling layer. The encoding portion may also include a normalization layer, and the decoding portion may also include a normalization layer. In the encoding part, the resolution of the feature maps is gradually reduced, and the number of the feature maps is gradually increased, so that abundant semantic features and image space features are obtained; in the decoding section, the resolution of the feature map is gradually increased, and the resolution of the feature map finally output by the decoding section is the same as the resolution of the first image. In the embodiment of the present disclosure, the related information may be an image, or may be in other data forms, such as a matrix.

As an example, inputting the first image into a neural network for relevance detection to process, and obtaining relevance information of a plurality of pixels in the first image may include: performing convolution processing on the first image to obtain a third convolution result; performing downsampling processing based on the third convolution result to obtain a second coding result; and obtaining the associated information of a plurality of pixels in the first image based on the second encoding result.

In one example, the first image may be convolved by the convolution layer, resulting in a third convolution result.

In one example, the downsampling process based on the third convolution result to obtain the second encoding result may include: carrying out normalization processing on the third convolution result to obtain a third normalization result; and performing downsampling processing on the third normalization result to obtain a second coding result. In this example, the third convolution result may be normalized by a normalization layer to obtain a third normalized result; and performing downsampling processing on the third normalization result through a downsampling layer to obtain a second coding result. Alternatively, the third convolution result may be downsampled by a downsampling layer to obtain the second encoding result.

In one example, determining the association information based on the second encoding result may include: performing deconvolution processing on the second coding result to obtain a second deconvolution result; and carrying out normalization processing on the second deconvolution result to obtain associated information. In this example, the second encoding result may be deconvoluted by the deconvolution layer to obtain a second deconvolution result; and carrying out normalization processing on the second deconvolution result through a normalization layer to obtain associated information. Alternatively, the second encoding result may be deconvoluted by a deconvolution layer to obtain the related information.

In one example, determining the association information based on the second encoding result may include: performing upsampling processing on the second coding result to obtain a second upsampling result; and carrying out normalization processing on the second up-sampling result to obtain the associated information. In an example, the second encoding result may be upsampled by an upsampling layer to obtain a second upsampled result; and carrying out normalization processing on the second up-sampling result through a normalization layer to obtain associated information. Alternatively, the second encoding result may be upsampled by an upsampling layer to obtain the associated information.

The current 3D sensors such as TOF and structured light are easily affected by sunlight outdoors, so that a depth map has large-area void loss, and the performance of a 3D living body detection algorithm is affected. The depth map self-perfection-based 3D in-vivo detection algorithm provided by the embodiment of the disclosure improves the performance of the 3D in-vivo detection algorithm by perfecting and repairing the depth map detected by the 3D sensor.

In some embodiments, after the depth prediction values and the associated information of the plurality of pixels are obtained, the first depth map is updated based on the depth prediction values and the associated information of the plurality of pixels, and the second depth map is obtained. Fig. 9 illustrates an exemplary schematic diagram of depth map updating in a vehicle door unlocking method according to an embodiment of the disclosure. In the example shown in fig. 9, the first depth map is a depth map with missing values, the obtained depth prediction values and associated information of a plurality of pixels are an initial depth estimation map and an associated feature map, respectively, and at this time, the depth map with missing values, the initial depth estimation map and the associated feature map are input to a depth map updating module (for example, a depth updating neural network) to be processed, so as to obtain a final depth map, that is, a second depth map.

In some embodiments, a depth prediction value of a depth failure pixel and depth prediction values of a plurality of surrounding pixels of the depth failure pixel are obtained from the depth prediction values of the plurality of pixels; acquiring association degrees between the depth failure pixel and a plurality of surrounding pixels of the depth failure pixel from the association information of the plurality of pixels; and determining an updated depth value of the depth failure pixel based on the depth predicted value of the depth failure pixel, the depth predicted values of a plurality of surrounding pixels of the depth failure pixel, and the correlation degree between the depth failure pixel and the surrounding pixels of the depth failure pixel.

In embodiments of the present disclosure, the depth failing pixels in the depth map may be determined in a variety of ways. As an example, a pixel in the first depth map having a depth value equal to 0 is determined as a depth failure pixel, or a pixel in the first depth map having no depth value is determined as a depth failure pixel.

In this example, for a part of the first depth map with missing values that has values (i.e., a depth value that is not 0), we consider its depth value to be correct and trustworthy, and do not update this part, leaving the original depth value. And updating the depth value of the pixel with the depth value of 0 in the first depth map.

As another example, the depth sensor may set the depth values of the depth failure pixels to one or more preset values or preset ranges. In an example, pixels in the first depth map having a depth value equal to a preset numerical value or belonging to a preset range may be determined as depth failure pixels.

The depth failure pixel in the first depth map may also be determined based on other statistical manners, which is not limited in the embodiment of the present disclosure.

In this implementation, depth values of pixels in the first image at the same location as the depth failure pixel may be determined as depth predictors for the depth failure pixel, and similarly, depth values of pixels in the first image at the same location as surrounding pixels of the depth failure pixel may be determined as depth predictors for surrounding pixels of the depth failure pixel.

As one example, the distance between surrounding pixels of the depth failure pixel and the depth failure pixel is less than or equal to the first threshold.

Fig. 10 shows a schematic diagram of surrounding pixels in a door unlocking method according to an embodiment of the present disclosure. For example, if the first threshold is 0, only the neighboring pixels are regarded as the surrounding pixels. For example, the neighboring pixels of the pixel 5 include the pixel 1, the pixel 2, the pixel 3, the pixel 4, the pixel 6, the pixel 7, the pixel 8, and the pixel 9, and only the pixel 1, the pixel 2, the pixel 3, the pixel 4, the pixel 6, the pixel 7, the pixel 8, and the pixel 9 are taken as surrounding pixels of the pixel 5.

Fig. 11 shows another schematic diagram of surrounding pixels in a door unlocking method according to an embodiment of the present disclosure. For example, if the first threshold is 1, the neighboring pixels of the neighboring pixels are regarded as the surrounding pixels in addition to the neighboring pixels. That is, in addition to the pixel 1, the pixel 2, the pixel 3, the pixel 4, the pixel 6, the pixel 7, the pixel 8, and the pixel 9 as the peripheral pixels of the pixel 5, the pixels 10 to 25 are also used as the peripheral pixels of the pixel 5.

As one example, a depth correlation value of the depth failure pixel is determined based on depth predicted values of surrounding pixels of the depth failure pixel and a correlation degree between the depth failure pixel and a plurality of surrounding pixels of the depth failure pixel; and determining the updated depth value of the depth failure pixel based on the depth predicted value and the depth correlation value of the depth failure pixel.

As another example, based on depth prediction values of surrounding pixels of the depth failure pixel and a degree of association between the depth failure pixel and the surrounding pixels, effective depth values of the surrounding pixels for the depth failure pixel are determined; and determining the updated depth value of the depth failure pixel based on the effective depth value of each surrounding pixel of the depth failure pixel to the depth failure pixel and the depth predicted value of the depth failure pixel. For example, the product of the depth prediction value of a certain peripheral pixel of the depth failure pixel and the corresponding association degree of the peripheral pixel, which refers to the association degree between the peripheral pixel and the depth failure pixel, may be determined as the effective depth value of the peripheral pixel for the depth failure pixel. For example, a product of a sum of effective depth values of respective surrounding pixels of the depth failure pixel with respect to the depth failure pixel and a first preset coefficient may be determined to obtain a first product; determining the product of the depth predicted value of the depth failure pixel and a second preset coefficient to obtain a second product; the sum of the first product and the second product is determined as the updated depth value of the depth failure pixel. In some embodiments, the sum of the first predetermined coefficient and the second predetermined coefficient is 1.

In one example, the association degree between the depth failure pixel and each surrounding pixel is used as the weight of each surrounding pixel, and the depth predicted values of a plurality of surrounding pixels of the depth failure pixel are subjected to weighted summation processing to obtain the depth association value of the depth failure pixel. For example, if the pixel 5 is a depth-fail pixel, the depth-related value of the depth-fail pixel 5 is

Updated depth value of depth failure pixel 5

Wherein,

w_irepresenting the degree of association, F, between pixel i and pixel 5_iRepresenting the depth prediction value for pixel i.

In another example, a product of a degree of association between each surrounding pixel of a plurality of surrounding pixels of the depth-failed pixel and a depth prediction value of each surrounding pixel is determined; the maximum value of the product is determined as the depth-related value of the depth-failed pixel.

In one example, a sum of the depth prediction value and the depth associated value of the depth failure pixel is determined as an updated depth value of the depth failure pixel.

In another example, a product of the depth prediction value of the depth failure pixel and a third preset coefficient is determined to obtain a third product; determining the product of the depth correlation value and a fourth preset coefficient to obtain a fourth product; the sum of the third product and the fourth product is determined as the updated depth value of the depth failure pixel. In some embodiments, the sum of the third predetermined coefficient and the fourth predetermined coefficient is 1.

In some embodiments, the depth value of the non-depth-failing pixel in the second depth map is equal to the depth value of the non-depth-failing pixel in the first depth map.

In other embodiments, the depth values of the non-depth-failed pixels may also be updated to obtain a more accurate second depth map, so that the accuracy of the in-vivo detection can be further improved.

In the disclosed embodiment, a bluetooth device of a preset logo is searched via a bluetooth module provided in a vehicle, a bluetooth pairing connection of the bluetooth module and the bluetooth device of the preset logo is established in response to the bluetooth device of the preset logo being searched, a first image of a target object is wakened and controlled by an image acquisition module provided in the vehicle in response to the bluetooth pairing connection being successful, face recognition is performed based on the first image, and a door unlock command is transmitted to at least one door lock of the vehicle in response to the face recognition being successful, whereby the face recognition module can be in a sleep state to maintain low power consumption operation when the bluetooth pairing connection with the bluetooth device of the preset logo is not established, so that operation power consumption in a face-brushing and door-opening manner can be reduced, and the face recognition module can be in an operable state before a user carrying the bluetooth device of the preset logo arrives at the vehicle door, when the user who carries the bluetooth equipment of predetermineeing the sign reachs the door, the face image processing can be carried out fast through the face identification module of awakening up behind the image acquisition module gather first image, and then face identification efficiency can be improved, user experience is improved. Therefore, the embodiment of the disclosure can meet the requirements of low power consumption operation and rapid door opening. By adopting the embodiment of the disclosure, when the vehicle owner approaches the vehicle, the processes of the living body detection and the face authentication can be automatically triggered without any deliberate action (such as touching a button or making a gesture), and the vehicle door is automatically opened after the live body detection and the face authentication of the vehicle owner pass.

In one possible implementation, after performing face recognition based on the first image, the method further includes: and responding to the failure of the face recognition, activating a password unlocking module arranged on the vehicle to start a password unlocking process.

In this implementation, password unlocking is an alternative to face recognition unlocking. The reason for the face recognition failure may include that the living body detection result is at least one of that the target object is a prosthesis, that the face authentication fails, that the image acquisition fails (for example, a camera failure), that the number of recognition times exceeds a predetermined number of times, and the like. When the target object does not pass face recognition, a password unlock procedure is expected. For example, the password input by the user may be acquired through a touch screen on the B-pillar. In one example, after M consecutive inputs of the wrong password, the password unlock will expire, e.g., M equals 5.

In one possible implementation, the method further includes one or both of: registering the car owner according to the face image of the car owner collected by the image collecting module; the remote registration is carried out according to the face image of the vehicle owner collected by the terminal equipment of the vehicle owner, and registration information is sent to the vehicle, wherein the registration information comprises the face image of the vehicle owner.

In one example, the car owner registration is carried out according to the face image of the car owner collected by the image collection module, and the method comprises the following steps: when detecting that a registration button on a touch screen is clicked, requesting a user to input a password, starting an RGB (red, green and blue) camera in an image acquisition module to acquire a face image of the user after the password passes verification, registering according to the acquired face image, extracting face features in the face image as pre-registered face features, and comparing faces based on the pre-registered face features during subsequent face authentication.

In one example, remote registration is carried out according to a face image of an owner collected by a terminal device of the owner, and registration information is sent to the automobile, wherein the registration information comprises the face image of the owner. In this example, the car owner may send a registration request to a TSP (Telematics Service Provider) cloud through a mobile phone App (Application), where the registration request may carry a face image of the car owner; the TSP cloud sends the registration request to a vehicle-mounted T-Box (Telematics Box) of the vehicle door unlocking device, the vehicle-mounted T-Box activates a face recognition function according to the registration request, and the face features in the face image carried in the registration request are used as pre-registered face features, so that face comparison is performed based on the pre-registered face features during subsequent face authentication.

FIG. 12 shows another flow chart of a method of unlocking a vehicle door according to an embodiment of the present disclosure. The execution subject of the vehicle door unlocking method may be a vehicle door unlocking device. In one possible implementation, the door unlocking method may be implemented by a processor calling computer readable instructions stored in a memory. For brevity, similar parts to those described above will not be described in detail below. As shown in fig. 12, the door unlocking method includes steps S21 to S24.

In step S21, a bluetooth device of a preset identification is searched for via a bluetooth module provided in the vehicle.

In one possible implementation manner, the bluetooth device that searches for the preset identifier via the bluetooth module provided to the vehicle includes: when the vehicle is in a flameout state or in a flameout and vehicle door locking state, the Bluetooth device with the preset identifier is searched through the Bluetooth module arranged on the vehicle.

In step S22, in response to the bluetooth device with the preset identifier being searched, the image capture module disposed in the vehicle is awakened and controlled to capture a first image of the target object.

In another possible implementation manner, the number of the bluetooth devices with the preset identifier is multiple; the response is searched the bluetooth equipment of preset sign, awakens and control set up in the first image of the image acquisition module collection target object of car, includes: and responding to the Bluetooth equipment which is searched to any one preset identification, and awakening and controlling an image acquisition module arranged on the vehicle to acquire a first image of the target object.

In a possible implementation manner, the waking up and controlling an image capturing module disposed in the vehicle to capture a first image of a target object includes: awakening a face recognition module arranged on the vehicle; and the awakened face recognition module controls the image acquisition module to acquire a first image of the target object.

For short distance sensor technologies such as ultrasonic wave, infrared, this disclosed embodiment can support great distance through the mode that adopts bluetooth. Practice shows that the time when a user carrying the Bluetooth device with the preset identifier arrives at the vehicle through the distance (the distance between the user and the vehicle when the Bluetooth module of the vehicle searches the Bluetooth device with the preset identifier of the user) is approximately matched with the time when the face recognition module is awakened by the vehicle and is converted from the dormant state into the working state, so that when the user arrives at the vehicle door, the user can immediately perform face recognition to open the vehicle door through the awakened face recognition module, the user does not need to wait for the face recognition module to be awakened after the user arrives at the vehicle door, the face recognition efficiency can be improved, and the user experience is improved. In addition, in the process of Bluetooth searching, the user has no perception, so that the user experience can be further improved. Therefore, the embodiment of the disclosure provides a solution capable of better balancing various aspects such as power consumption saving, user experience and security of the face recognition module in a manner of responding to the fact that the face recognition module is awakened by the Bluetooth device which searches the preset identifier.

In one possible implementation manner, after the waking up the face recognition module set in the car, the method further includes: and if the face image is not acquired within the preset time, controlling the face recognition module to enter a dormant state.

In one possible implementation manner, after the waking up the face recognition module set in the car, the method further includes: and if the face recognition module fails to pass the face recognition within the preset time, controlling the face recognition module to enter a dormant state.

In step S23, face recognition is performed based on the first image.

In step S24, in response to the face recognition being successful, a door unlock instruction is sent to at least one door lock of the vehicle.

In one possible implementation manner, the sending a door unlocking instruction to at least one door lock of the vehicle in response to successful face recognition includes: responding to the success of face recognition, and determining the vehicle door of which the target object has the door opening authority; and sending a door unlocking instruction to at least one door lock of the vehicle according to the vehicle door of which the target object has the door opening authority.

In one possible implementation, the face recognition includes: living body detection and face authentication; the face recognition based on the first image comprises: acquiring the first image through an image sensor in the image acquisition module, and performing face authentication based on the first image and pre-registered face features; and acquiring a first depth map corresponding to the first image through a depth sensor in the image acquisition module, and carrying out in-vivo detection on the basis of the first image and the first depth map.

In one possible implementation, the performing living body detection based on the first image and the first depth map includes: updating the first depth map based on the first image to obtain a second depth map; determining a live body detection result of the target object based on the first image and the second depth map.

In one possible implementation, the image sensor includes an RGB image sensor or an infrared sensor; the depth sensor comprises a binocular infrared sensor or a time of flight TOF sensor.

In a possible implementation manner, the updating the first depth map based on the first image to obtain a second depth map includes: and updating the depth value of the depth failure pixel in the first depth map based on the first image to obtain the second depth map.

In a possible implementation manner, the updating the first depth map based on the first image to obtain a second depth map includes: determining depth prediction values and associated information of a plurality of pixels in the first image based on the first image, wherein the associated information of the plurality of pixels indicates a degree of association between the plurality of pixels; and updating the first depth map based on the depth predicted values and the associated information of the plurality of pixels to obtain a second depth map.

In a possible implementation manner, the updating the first depth map based on the depth prediction values and the associated information of the plurality of pixels to obtain a second depth map includes: determining depth failure pixels in the first depth map; obtaining a depth predicted value of the depth failure pixel and depth predicted values of a plurality of surrounding pixels of the depth failure pixel from the depth predicted values of the plurality of pixels; acquiring association degrees between the depth failure pixel and a plurality of surrounding pixels of the depth failure pixel from association information of the plurality of pixels; determining an updated depth value of the depth failure pixel based on the depth prediction value of the depth failure pixel, the depth prediction values of a plurality of surrounding pixels of the depth failure pixel, and the correlation between the depth failure pixel and the surrounding pixels of the depth failure pixel.

In one possible implementation, the determining an updated depth value of the depth failure pixel based on the depth prediction value of the depth failure pixel, the depth prediction values of a plurality of surrounding pixels of the depth failure pixel, and the association between the depth failure pixel and the plurality of surrounding pixels of the depth failure pixel includes: determining a depth correlation value of the depth failure pixel based on depth predicted values of surrounding pixels of the depth failure pixel and correlation degrees between the depth failure pixel and a plurality of surrounding pixels of the depth failure pixel; and determining an updated depth value of the depth failure pixel based on the depth predicted value and the depth related value of the depth failure pixel.

In a possible implementation manner, the determining a depth-related value of the depth-failed pixel based on the depth prediction values of the surrounding pixels of the depth-failed pixel and the association degree between the depth-failed pixel and a plurality of surrounding pixels of the depth-failed pixel includes: and taking the association degree between the depth failure pixel and each surrounding pixel as the weight of each surrounding pixel, and performing weighted summation processing on the depth predicted values of a plurality of surrounding pixels of the depth failure pixel to obtain the depth association value of the depth failure pixel.

In one possible implementation, the determining depth prediction values of a plurality of pixels in the first image based on the first image includes: based on the first image and the first depth map, depth predictors for a plurality of pixels in the first image are determined.

In one possible implementation, the determining depth prediction values of a plurality of pixels in the first image based on the first image and the first depth map includes: and inputting the first image and the first depth map into a depth prediction neural network for processing to obtain depth prediction values of a plurality of pixels in the first image.

In one possible implementation, the determining depth prediction values of a plurality of pixels in the first image based on the first image and the first depth map includes: performing fusion processing on the first image and the first depth map to obtain a fusion result; based on the fusion result, depth predicted values of a plurality of pixels in the first image are determined.

In a possible implementation manner, the determining, based on the first image, associated information of a plurality of pixels in the first image includes: and inputting the first image into a correlation degree detection neural network for processing to obtain correlation information of a plurality of pixels in the first image.

In one possible implementation, the updating the first depth map based on the first image includes: acquiring an image of the target object from the first image; updating the first depth map based on the image of the target object.

In one possible implementation, the acquiring the image of the target object from the first image includes: acquiring key point information of the target object in the first image; and acquiring an image of the target object from the first image based on the key point information of the target object.

In a possible implementation manner, the acquiring of the key point information of the target object in the first image includes: carrying out target detection on the first image to obtain a region where the target object is located; and detecting key points of the image of the area where the target object is located to obtain key point information of the target object in the first image.

In a possible implementation manner, the updating the first depth map based on the first image to obtain a second depth map includes: acquiring a depth map of the target object from the first depth map; and updating the depth map of the target object based on the first image to obtain the second depth map.

In one possible implementation, the determining a living body detection result of the target object based on the first image and the second depth map includes: and inputting the first image and the second depth map into a living body detection neural network for processing to obtain a living body detection result of the target object.

In one possible implementation, the determining a living body detection result of the target object based on the first image and the second depth map includes: performing feature extraction processing on the first image to obtain first feature information; performing feature extraction processing on the second depth map to obtain second feature information; determining a living body detection result of the target object based on the first feature information and the second feature information.

In one possible implementation manner, the determining a living body detection result of the target object based on the first feature information and the second feature information includes: performing fusion processing on the first characteristic information and the second characteristic information to obtain third characteristic information; determining a living body detection result of the target object based on the third feature information.

In one possible implementation manner, the determining a living body detection result of the target object based on the third feature information includes: obtaining the probability that the target object is a living body based on the third characteristic information; and determining the living body detection result of the target object according to the probability that the target object is a living body.

In one possible implementation, after the face recognition based on the first image, the method further includes: and responding to the failure of face recognition, activating a password unlocking module arranged on the vehicle to start a password unlocking process.

In one possible implementation, the method further includes one or both of: registering the car owner according to the face image of the car owner collected by the image collection module; and carrying out remote registration according to the face image of the vehicle owner collected by the terminal equipment of the vehicle owner, and sending registration information to the vehicle, wherein the registration information comprises the face image of the vehicle owner.

It is understood that the above-mentioned method embodiments of the present disclosure can be combined with each other to form a combined embodiment without departing from the logic of the principle, which is limited by the space, and the detailed description of the present disclosure is omitted.

It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.

In addition, the present disclosure also provides a vehicle door unlocking device, an electronic apparatus, a computer-readable storage medium, and a program, which can be used to implement any one of the vehicle door unlocking methods provided by the present disclosure, and the corresponding technical solutions and descriptions and corresponding descriptions in the method sections are not repeated.

Fig. 13 shows a block diagram of a vehicle door unlocking apparatus according to an embodiment of the present disclosure. As shown in fig. 13, the door unlocking device includes: a searching module 31 for searching for a bluetooth device of a preset identification through a bluetooth module provided in the vehicle; the awakening module 32 is used for responding to the Bluetooth device with the preset identifier, establishing Bluetooth pairing connection between the Bluetooth module and the Bluetooth device with the preset identifier, awakening and controlling an image acquisition module arranged on the vehicle to acquire a first image of a target object in response to successful Bluetooth pairing connection, or awakening and controlling the image acquisition module arranged on the vehicle to acquire the first image of the target object in response to the Bluetooth device with the preset identifier; a face recognition module 33, configured to perform face recognition based on the first image; and the unlocking module 34 is used for responding to the success of the face recognition and sending a vehicle door unlocking instruction to at least one vehicle door lock of the vehicle.

In a possible implementation manner, the search module 31 is configured to: when the vehicle is in a flameout state or in a flameout and vehicle door locking state, the Bluetooth device with the preset identifier is searched through the Bluetooth module arranged on the vehicle.

the wake-up module 32 is configured to: and responding to the Bluetooth device searched for any preset identifier, establishing Bluetooth pairing connection between the Bluetooth module and the Bluetooth device of the preset identifier, or responding to the Bluetooth device searched for any preset identifier, awakening and controlling an image acquisition module arranged on the vehicle to acquire a first image of a target object.

In one possible implementation, the wake-up module 32 includes: the awakening sub-module is used for awakening a face recognition module arranged on the vehicle; and the control sub-module is used for controlling the image acquisition module to acquire a first image of the target object through the awakened face recognition module.

In one possible implementation, the apparatus further includes: the first control module is used for controlling the face recognition module to enter a dormant state if the face image is not collected within the preset time.

In one possible implementation, the apparatus further includes: and the second control module is used for controlling the face recognition module to enter a dormant state if the face recognition module fails within the preset time.

In one possible implementation, the unlocking module 34 is configured to: responding to the success of face recognition, and determining the vehicle door of which the target object has the door opening authority; and sending a door unlocking instruction to at least one door lock of the vehicle according to the vehicle door of which the target object has the door opening authority.

In one possible implementation, the face recognition includes: living body detection and face authentication; the face recognition module 33 includes: the face authentication module is used for acquiring the first image through an image sensor in the image acquisition module and performing face authentication based on the first image and pre-registered face features; and the living body detection module is used for acquiring a first depth map corresponding to the first image through a depth sensor in the image acquisition module and carrying out living body detection on the basis of the first image and the first depth map.

In one possible implementation, the liveness detection module includes: the updating submodule is used for updating the first depth map based on the first image to obtain a second depth map; a determination sub-module for determining a live detection result of the target object based on the first image and the second depth map.

In one possible implementation, the update submodule is configured to: and updating the depth value of the depth failure pixel in the first depth map based on the first image to obtain the second depth map.

In one possible implementation, the update submodule is configured to: determining depth prediction values and associated information of a plurality of pixels in the first image based on the first image, wherein the associated information of the plurality of pixels indicates a degree of association between the plurality of pixels; and updating the first depth map based on the depth predicted values and the associated information of the plurality of pixels to obtain a second depth map.

In one possible implementation, the update submodule is configured to: determining depth failure pixels in the first depth map; obtaining a depth predicted value of the depth failure pixel and depth predicted values of a plurality of surrounding pixels of the depth failure pixel from the depth predicted values of the plurality of pixels; acquiring association degrees between the depth failure pixel and a plurality of surrounding pixels of the depth failure pixel from association information of the plurality of pixels; determining an updated depth value of the depth failure pixel based on the depth prediction value of the depth failure pixel, the depth prediction values of a plurality of surrounding pixels of the depth failure pixel, and the correlation between the depth failure pixel and the surrounding pixels of the depth failure pixel.

In one possible implementation, the update submodule is configured to: determining a depth correlation value of the depth failure pixel based on depth predicted values of surrounding pixels of the depth failure pixel and correlation degrees between the depth failure pixel and a plurality of surrounding pixels of the depth failure pixel; and determining an updated depth value of the depth failure pixel based on the depth predicted value and the depth related value of the depth failure pixel.

In one possible implementation, the update submodule is configured to: and taking the association degree between the depth failure pixel and each surrounding pixel as the weight of each surrounding pixel, and performing weighted summation processing on the depth predicted values of a plurality of surrounding pixels of the depth failure pixel to obtain the depth association value of the depth failure pixel.

In one possible implementation, the update submodule is configured to: based on the first image and the first depth map, depth predictors for a plurality of pixels in the first image are determined.

In one possible implementation, the update submodule is configured to: and inputting the first image and the first depth map into a depth prediction neural network for processing to obtain depth prediction values of a plurality of pixels in the first image.

In one possible implementation, the update submodule is configured to: performing fusion processing on the first image and the first depth map to obtain a fusion result; based on the fusion result, depth predicted values of a plurality of pixels in the first image are determined.

In one possible implementation, the update submodule is configured to: and inputting the first image into a correlation degree detection neural network for processing to obtain correlation information of a plurality of pixels in the first image.

In one possible implementation, the update submodule is configured to: acquiring an image of the target object from the first image; updating the first depth map based on the image of the target object.

In one possible implementation, the update submodule is configured to: acquiring key point information of the target object in the first image; and acquiring an image of the target object from the first image based on the key point information of the target object.

In one possible implementation, the update submodule is configured to: carrying out target detection on the first image to obtain a region where the target object is located; and detecting key points of the image of the area where the target object is located to obtain key point information of the target object in the first image.

In one possible implementation, the update submodule is configured to: acquiring a depth map of the target object from the first depth map; and updating the depth map of the target object based on the first image to obtain the second depth map.

In one possible implementation, the determining sub-module is configured to: and inputting the first image and the second depth map into a living body detection neural network for processing to obtain a living body detection result of the target object.

In one possible implementation, the determining sub-module is configured to: performing feature extraction processing on the first image to obtain first feature information; performing feature extraction processing on the second depth map to obtain second feature information; determining a living body detection result of the target object based on the first feature information and the second feature information.

In one possible implementation, the determining sub-module is configured to: performing fusion processing on the first characteristic information and the second characteristic information to obtain third characteristic information; determining a living body detection result of the target object based on the third feature information.

In one possible implementation, the determining sub-module is configured to: obtaining the probability that the target object is a living body based on the third characteristic information; and determining the living body detection result of the target object according to the probability that the target object is a living body.

In one possible implementation, the apparatus further includes: and the activation and starting module is used for responding to the failure of face recognition, and activating a password unlocking module arranged on the vehicle to start a password unlocking process.

In one possible implementation, the apparatus further includes a registration module, and the registration module is configured to one or both of: registering the car owner according to the face image of the car owner collected by the image collection module; and carrying out remote registration according to the face image of the vehicle owner collected by the terminal equipment of the vehicle owner, and sending registration information to the vehicle, wherein the registration information comprises the face image of the vehicle owner.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

Fig. 14 shows a block diagram of a vehicle-mounted face unlocking system according to an embodiment of the present disclosure. As shown in fig. 14, the vehicle-mounted face unlocking system includes: the device comprises a memory 41, a face recognition module 42, an image acquisition module 43 and a Bluetooth module 44; the face recognition module 42 is respectively connected with the memory 41, the image acquisition module 43 and the bluetooth module 44; the bluetooth module 44 includes a microprocessor 441 and a bluetooth sensor 442, wherein the microprocessor 441 wakes up the face recognition module 42 when a bluetooth pairing connection with a bluetooth device with a preset identifier is successful or the bluetooth device with the preset identifier is searched; the face recognition module 42 is further provided with a communication interface for connecting with a vehicle door domain controller, and if the face recognition is successful, control information for unlocking the vehicle door is sent to the vehicle door domain controller based on the communication interface.

In one example, the memory 41 may include at least one of Flash memory (Flash) and DDR3(Double data Rate 3, third generation Double data Rate) memory.

In one example, the face recognition module 42 may be implemented in a SoC (System on Chip).

In one example, the face recognition module 42 is connected to the door domain Controller via a CAN (Controller Area Network) bus.

In one example, the image capture module 43 includes an image sensor and a depth sensor.

In one example, the depth sensor includes at least one of a binocular infrared sensor and a time of flight TOF sensor.

In one possible implementation, the depth sensor includes a binocular infrared sensor, and two infrared cameras of the binocular infrared sensor are disposed at both sides of a camera of the image sensor. For example, in the example shown in fig. 4a, the image sensor is an RGB sensor, the camera of the image sensor is an RGB camera, and the depth sensor is a binocular infrared sensor, the depth sensor including two IR (infrared) cameras, the two infrared cameras of the binocular infrared sensor being disposed at both sides of the RGB camera of the image sensor.

In one example, the image capturing module 43 further includes at least one light supplement lamp, the at least one light supplement lamp is disposed between the infrared camera of the binocular infrared sensor and the camera of the image sensor, and the at least one light supplement lamp includes at least one of the light supplement lamp for the image sensor and the light supplement lamp for the depth sensor. For example, if the image sensor is an RGB sensor, the fill-in light for the image sensor may be a white light; if the image sensor is an infrared sensor, the light supplement lamp for the image sensor can be an infrared lamp; if the depth sensor is a binocular infrared sensor, the light supplement lamp for the depth sensor may be an infrared lamp. In the example shown in fig. 4a, an infrared lamp is provided between the infrared camera of the binocular infrared sensor and the camera of the image sensor. For example, the infrared lamp may use 940nm infrared light.

In a possible implementation manner, the image capturing module 43 further includes a laser, and the laser is disposed between the camera of the depth sensor and the camera of the image sensor. For example, in the example shown in fig. 4b, the image sensor is an RGB sensor, the camera of the image sensor is an RGB camera, the depth sensor is a TOF sensor, and the laser is disposed between the camera of the TOF sensor and the camera of the RGB sensor. For example, the laser may be a VCSEL, and the TOF sensor may acquire a depth map based on laser light emitted by the VCSEL.

In one example, the depth sensor is connected to the face recognition system 42 via an LVDS (Low-Voltage Differential Signaling) interface.

In a possible implementation manner, the vehicle-mounted human face unlocking system further includes: and the password unlocking module 45 is used for unlocking the vehicle door, and the password unlocking module 45 is connected with the face recognition module 42.

In a possible implementation manner, the password unlocking module 45 includes one or both of a touch screen and a keyboard.

In one example, the touch screen is connected to the face recognition module 42 via a FPD-Link (Flat Panel Display Link).

In a possible implementation manner, the vehicle-mounted human face unlocking system further includes: and the battery module 46, wherein the battery module 46 is respectively connected with the microprocessor 441 and the face recognition module 42.

In one possible implementation, the memory 41, the face recognition module 42, the bluetooth module 44 and the battery module 46 may be built on an ECU (Electronic Control Unit).

Fig. 15 shows a schematic diagram of a vehicle-mounted face unlocking system according to an embodiment of the present disclosure. In the example shown in fig. 15, the face recognition module is implemented by SoC101, the memory includes Flash memory (Flash)102 and DDR3 memory 103, the Bluetooth module includes Bluetooth sensor (Bluetooth)104 and Microprocessor (MCU) 105, SoC101, Flash memory 102, DDR3 memory 103, Bluetooth sensor 104, microprocessor 105 and battery module (Power Management)106 are built on ECU100, the image acquisition module includes depth sensor (3D Camera)200, depth sensor 200 is connected to SoC101 through LVDS interface, the password unlocking module includes Touch Screen (Touch Screen)300, Touch Screen 300 is connected to SoC101 through FPD-Link, and SoC101 is connected to door domain 400 through CAN bus.

FIG. 16 shows a schematic view of a cart according to an embodiment of the present disclosure. As shown in fig. 16, the vehicle includes a vehicle-mounted human face unlocking system 51, and the vehicle-mounted human face unlocking system 51 is connected to a door domain controller 52 of the vehicle.

Embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the above-mentioned method. The computer readable storage medium may be a non-volatile computer readable storage medium.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured as the above method.

The electronic device may be provided as a terminal, server, or other form of device.

Fig. 17 is a block diagram illustrating an electronic device 800 in accordance with an example embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like terminal.

Referring to fig. 17, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 804, is also provided that includes computer program instructions executable by the processor 820 of the electronic device 800 to perform the above-described methods.

The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method of unlocking a vehicle door, comprising:

responding to the Bluetooth pairing connection success, waking up a face recognition module arranged on the vehicle, and controlling an image acquisition module arranged on the vehicle to acquire a first image of a target object by the woken-up face recognition module, wherein the face recognition module is in a dormant state before being woken up;

performing face recognition based on the first image, wherein the face recognition comprises: detecting a living body;

responding to the success of the face recognition, and sending a vehicle door unlocking instruction to at least one vehicle door lock of the vehicle;

the face recognition based on the first image comprises:

acquiring a first depth map corresponding to the first image through a depth sensor in the image acquisition module, and performing in-vivo detection based on the first image and the first depth map;

the performing living body detection based on the first image and the first depth map comprises:

updating the first depth map based on the depth predicted values and the associated information of the plurality of pixels to obtain a second depth map;

2. The method of claim 1, wherein the searching for the bluetooth device of the preset identity via the bluetooth module provided to the vehicle comprises:

3. The method according to claim 1 or 2, wherein the number of the bluetooth devices of the preset identification is one.

4. The method according to claim 1 or 2, wherein the number of the bluetooth devices with the preset identification is multiple;

5. The method of any of claims 1 to 4, wherein after said waking up a face recognition module disposed on the vehicle, the method further comprises:

6. The method of any of claims 1 to 4, wherein after said waking up a face recognition module disposed on the vehicle, the method further comprises:

7. The method of any one of claims 1 to 6, wherein sending a door unlock instruction to at least one door lock of the vehicle in response to successful face recognition comprises:

8. The method of any one of claims 1 to 7, wherein the face recognition further comprises: authenticating the human face;

the face recognition based on the first image further comprises:

and acquiring the first image through an image sensor in the image acquisition module, and performing face authentication based on the first image and the pre-registered face features.

9. The method of claim 8, wherein the image sensor comprises an RGB image sensor or an infrared sensor;

10. The method of claim 9, wherein the TOF sensor employs an infrared band based TOF module.

11. The method according to any one of claims 1 to 10, wherein updating the first depth map based on the first image to obtain a second depth map comprises:

12. The method according to any one of claims 1 to 11, wherein the updating the first depth map based on the depth prediction values and the associated information of the plurality of pixels to obtain a second depth map comprises:

determining depth failure pixels in the first depth map;

13. The method of claim 12, wherein determining the updated depth value for the depth failure pixel based on the depth prediction value for the depth failure pixel, the depth prediction values for a plurality of surrounding pixels of the depth failure pixel, and the degree of correlation between the depth failure pixel and the plurality of surrounding pixels of the depth failure pixel comprises:

14. The method of claim 13, wherein determining the depth correlation value for the depth failure pixel based on depth predicted values for surrounding pixels of the depth failure pixel and a degree of correlation between the depth failure pixel and a plurality of surrounding pixels of the depth failure pixel comprises:

15. The method of any one of claims 1 to 14, wherein determining depth predictors for a plurality of pixels in the first image based on the first image comprises:

16. The method of claim 15, wherein determining depth predictors for a plurality of pixels in the first image based on the first image and the first depth map comprises:

17. The method of claim 15 or 16, wherein determining depth predictors for a plurality of pixels in the first image based on the first image and the first depth map comprises:

18. The method according to any one of claims 1 to 17, wherein the determining, based on the first image, the association information of the plurality of pixels in the first image comprises:

19. The method of any of claims 1-18, wherein updating the first depth map based on the first image comprises:

acquiring an image of the target object from the first image;

updating the first depth map based on the image of the target object.

20. The method of claim 19, wherein said obtaining an image of the target object from the first image comprises:

acquiring key point information of the target object in the first image;

21. The method of claim 20, wherein the obtaining of the keypoint information of the target object in the first image comprises:

22. The method according to any one of claims 1 to 21, wherein updating the first depth map based on the first image to obtain a second depth map comprises:

acquiring a depth map of the target object from the first depth map;

23. The method of any one of claims 1 to 22, wherein determining the in-vivo detection result of the target object based on the first image and the second depth map comprises:

24. The method of any one of claims 1 to 23, wherein determining the in-vivo detection result of the target object based on the first image and the second depth map comprises:

25. The method according to claim 24, wherein the determining a live detection result of the target object based on the first feature information and the second feature information comprises:

26. The method according to claim 25, wherein the determining a living body detection result of the target object based on the third feature information comprises:

27. The method of any one of claims 1 to 26, wherein after the performing face recognition based on the first image, the method further comprises:

28. The method of any one of claims 1 to 27, further comprising one or both of:

29. A method of unlocking a vehicle door, comprising:

responding to the Bluetooth equipment with the preset identification, awakening a face recognition module arranged on the vehicle, and controlling an image acquisition module arranged on the vehicle to acquire a first image of a target object by the awakened face recognition module, wherein the face recognition module is in a dormant state before being awakened;

the face recognition based on the first image comprises:

30. The method of claim 29, wherein the searching for the bluetooth device of the preset identity via the bluetooth module provided to the vehicle comprises:

31. The method according to claim 29 or 30, wherein the number of bluetooth devices of the preset identification is one.

32. The method according to claim 29 or 30, wherein the number of the bluetooth devices with the preset identifier is multiple;

33. The method of any one of claims 29 to 32, wherein after said waking up a face recognition module provided to the vehicle, the method further comprises:

34. The method of any one of claims 29 to 32, wherein after said waking up a face recognition module provided to the vehicle, the method further comprises:

35. The method of any one of claims 29 to 34, wherein sending a door unlock command to at least one door lock of the vehicle in response to successful face recognition comprises:

36. The method according to any one of claims 29 to 35, wherein the face recognition further comprises: authenticating the human face;

the face recognition based on the first image further comprises:

37. The method of claim 36, wherein the image sensor comprises an RGB image sensor or an infrared sensor;

38. The method of claim 37, wherein said TOF sensor employs an infrared band based TOF module.

39. The method according to any one of claims 29 to 38, wherein updating the first depth map based on the first image to obtain a second depth map comprises:

40. The method according to any one of claims 29 to 39, wherein the updating the first depth map based on the depth prediction values and the associated information of the plurality of pixels to obtain a second depth map comprises:

determining depth failure pixels in the first depth map;

41. The method of claim 40, wherein determining the updated depth value for the depth failure pixel based on the depth prediction value for the depth failure pixel, the depth prediction values for a plurality of surrounding pixels of the depth failure pixel, and the degree of correlation between the depth failure pixel and the plurality of surrounding pixels of the depth failure pixel comprises:

42. The method of claim 41, wherein determining the depth correlation value of the depth failure pixel based on the depth prediction values of the surrounding pixels of the depth failure pixel and the degree of correlation between the depth failure pixel and the plurality of surrounding pixels of the depth failure pixel comprises:

43. A method as claimed in any one of claims 29 to 42, wherein determining depth predictors for a plurality of pixels in the first image based on the first image comprises:

44. The method of claim 43, wherein determining depth predictors for a plurality of pixels in the first image based on the first image and the first depth map comprises:

45. The method of claim 43 or 44, wherein determining depth predictors for a plurality of pixels in the first image based on the first image and the first depth map comprises:

46. The method according to any one of claims 29 to 45, wherein the determining the association information of the plurality of pixels in the first image based on the first image comprises:

47. The method of any one of claims 29 to 46, wherein updating the first depth map based on the first image comprises:

acquiring an image of the target object from the first image;

updating the first depth map based on the image of the target object.

48. The method of claim 47, wherein said obtaining an image of the target object from the first image comprises:

acquiring key point information of the target object in the first image;

49. The method of claim 48, wherein the obtaining of the keypoint information of the target object in the first image comprises:

50. The method according to any one of claims 29 to 49, wherein updating the first depth map based on the first image to obtain a second depth map comprises:

acquiring a depth map of the target object from the first depth map;

51. The method of any one of claims 29 to 50, wherein determining the in-vivo detection result of the target object based on the first image and the second depth map comprises:

52. The method of any one of claims 29 to 51, wherein determining the in-vivo detection result of the target object based on the first image and the second depth map comprises:

53. The method of claim 52, wherein the determining a live detection result of the target object based on the first characteristic information and the second characteristic information comprises:

54. The method according to claim 53, wherein the determining a living body detection result of the target object based on the third feature information comprises:

55. The method of any one of claims 29 to 54, wherein after said performing face recognition based on said first image, the method further comprises:

56. The method of any one of claims 29 to 55, further comprising one or both of:

57. A vehicle door unlocking device, comprising:

the awakening module is used for responding to the Bluetooth device with the preset identifier, establishing Bluetooth pairing connection between the Bluetooth module and the Bluetooth device with the preset identifier, awakening a face recognition module arranged on the vehicle in response to successful Bluetooth pairing connection, and controlling the image acquisition module arranged on the vehicle to acquire a first image of a target object through the awakened face recognition module, or awakening the face recognition module arranged on the vehicle in response to the Bluetooth device with the preset identifier, and controlling the image acquisition module arranged on the vehicle to acquire the first image of the target object through the awakened face recognition module, wherein the face recognition module is in a dormant state before being awakened;

a face recognition module, configured to perform face recognition based on the first image, where the face recognition includes: detecting a living body;

the unlocking module is used for responding to the success of the face recognition and sending a vehicle door unlocking instruction to at least one vehicle door lock of the vehicle;

the face recognition module includes:

the living body detection module is used for acquiring a first depth map corresponding to the first image through a depth sensor in the image acquisition module and carrying out living body detection on the basis of the first image and the first depth map;

the living body detecting module includes:

the updating sub-module is used for determining depth predicted values and associated information of a plurality of pixels in the first image based on the first image, wherein the associated information of the plurality of pixels indicates the degree of association among the plurality of pixels, and updating the first depth map based on the depth predicted values and the associated information of the plurality of pixels to obtain a second depth map;

58. The apparatus of claim 57, wherein the search module is configured to:

59. The apparatus according to claim 57 or 58, wherein the number of the bluetooth devices of the preset identifier is one.

60. The apparatus according to claim 57 or 58, wherein the number of the bluetooth devices of the preset identifier is plural;

the wake-up module is configured to:

61. The apparatus of any one of claims 57 to 60, further comprising:

62. The apparatus of any one of claims 57 to 60, further comprising:

63. The device of any one of claims 57-62, wherein the unlocking module is configured to:

64. The apparatus according to any one of claims 57 to 63, wherein the face recognition further comprises face authentication;

the face recognition module further comprises:

and the face authentication module is used for acquiring the first image through an image sensor in the image acquisition module and performing face authentication based on the first image and the pre-registered face features.

65. The apparatus of claim 64, wherein the image sensor comprises an RGB image sensor or an infrared sensor;

66. The apparatus of claim 65, wherein said TOF sensor employs an infrared band based TOF module.

67. The apparatus according to any one of claims 57 to 66, wherein the update submodule is configured to:

68. The apparatus according to any one of claims 57 to 67, wherein the update submodule is configured to:

determining depth failure pixels in the first depth map;

69. The apparatus of claim 68, wherein the update submodule is configured to:

70. The apparatus of claim 69, wherein the update submodule is configured to:

71. The apparatus according to any one of claims 57 to 70, wherein the update submodule is configured to:

72. The apparatus of claim 71, wherein the update submodule is configured to:

73. The apparatus of claim 71 or 72, wherein the update submodule is configured to:

74. The apparatus of any one of claims 57 to 73, wherein the update submodule is configured to:

75. The apparatus according to any one of claims 57 to 74, wherein the update submodule is configured to:

acquiring an image of the target object from the first image;

updating the first depth map based on the image of the target object.

76. The apparatus of claim 75, wherein the update submodule is configured to:

acquiring key point information of the target object in the first image;

77. The apparatus according to claim 76, wherein the update submodule is configured to:

78. The apparatus according to any one of claims 57 to 77, wherein the update submodule is configured to:

acquiring a depth map of the target object from the first depth map;

79. The apparatus according to any one of claims 57 to 78, wherein the determination submodule is configured to:

80. The apparatus of any one of claims 57 to 79, wherein the determination submodule is configured to:

81. The apparatus of claim 80, wherein the determination submodule is configured to:

82. The apparatus of claim 81, wherein the determination submodule is configured to:

83. The apparatus of any one of claims 57 to 82, further comprising:

84. The apparatus according to any one of claims 57 to 83, further comprising a registration module for one or both of:

85. An on-vehicle people face unblock system which characterized in that includes: the system comprises a memory, a face recognition module, an image acquisition module and a Bluetooth module; the face recognition module is respectively connected with the memory, the image acquisition module and the Bluetooth module; the Bluetooth module comprises a microprocessor and a Bluetooth sensor, wherein the microprocessor wakes up the face recognition module when the Bluetooth pairing connection with the Bluetooth equipment with a preset identifier is successful or the Bluetooth equipment with the preset identifier is searched; the awakened face recognition module is used for controlling the image acquisition module to acquire a first image of a target object and a first depth map corresponding to the first image, and carrying out living body detection based on the first image and the first depth map, wherein the face recognition module is in a dormant state before being awakened; the performing living body detection based on the first image and the first depth map comprises: determining depth predicted values and associated information of a plurality of pixels in the first image based on the first image, wherein the associated information of the plurality of pixels indicates a degree of association between the plurality of pixels, updating the first depth map based on the depth predicted values and associated information of the plurality of pixels to obtain a second depth map, and determining a living body detection result of the target object based on the first image and the second depth map; the face recognition module is also provided with a communication interface used for being connected with the vehicle door domain controller, and if face recognition is successful, control information used for unlocking the vehicle door is sent to the vehicle door domain controller based on the communication interface.

86. The vehicle-mounted human face unlocking system of claim 85, wherein the image acquisition module comprises an image sensor and a depth sensor.

87. The vehicle-mounted face unlocking system of claim 86, wherein the depth sensor includes a binocular infrared sensor, and two infrared cameras of the binocular infrared sensor are disposed on both sides of a camera of the image sensor.

88. The vehicle-mounted human face unlocking system of claim 87, wherein the image acquisition module further comprises at least one light supplement lamp, the at least one light supplement lamp is arranged between the infrared camera of the binocular infrared sensor and the camera of the image sensor, and the at least one light supplement lamp comprises at least one of a light supplement lamp for the image sensor and a light supplement lamp for the depth sensor.

89. The vehicle-mounted human face unlocking system of claim 86, wherein the image acquisition module further comprises a laser, and the laser is disposed between the camera of the depth sensor and the camera of the image sensor.

90. The vehicle-mounted face unlocking system according to any one of claims 85 to 89, further comprising: and the password unlocking module is used for unlocking the vehicle door and is connected with the face recognition module.

91. The vehicle-mounted human face unlocking system of claim 90, wherein the password unlocking module comprises one or both of a touch screen and a keyboard.

92. The vehicle-mounted face unlocking system according to any one of claims 85 to 91, further comprising: and the battery module is respectively connected with the microprocessor and the face recognition module.

93. A vehicle comprising a vehicle face unlocking system according to any of claims 85 to 92, the vehicle face unlocking system being connected to a door domain controller of the vehicle.

94. The cart of claim 93, wherein the image capture module is disposed outside of a compartment of the cart.

95. The cart of claim 94, wherein the image capture module is disposed in at least one of: the B post of car, at least one door, at least one rear-view mirror.

96. The vehicle of any of claims 93-95, wherein the face recognition module is disposed within the vehicle, the face recognition module being coupled to the door domain controller via a CAN bus.

97. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: performing the method of any one of claims 1 to 28.

98. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: performing the method of any one of claims 29 to 56.

99. A computer readable storage medium having computer program instructions stored thereon, which when executed by a processor implement the method of any one of claims 1 to 28.

100. A computer readable storage medium having computer program instructions stored thereon which, when executed by a processor, implement the method of any one of claims 29 to 56.