WO2024087982A1 - 一种图像处理方法及电子设备 - Google Patents

一种图像处理方法及电子设备 Download PDF

Info

Publication number
WO2024087982A1
WO2024087982A1 PCT/CN2023/121062 CN2023121062W WO2024087982A1 WO 2024087982 A1 WO2024087982 A1 WO 2024087982A1 CN 2023121062 W CN2023121062 W CN 2023121062W WO 2024087982 A1 WO2024087982 A1 WO 2024087982A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
image
area
roi area
image sensor
Prior art date
Application number
PCT/CN2023/121062
Other languages
English (en)
French (fr)
Inventor
张新功
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2024087982A1 publication Critical patent/WO2024087982A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/04—Context-preserving transformations, e.g. by using an importance map
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]

Definitions

  • the present application relates to the field of electronic devices, and in particular to an image processing method and electronic device.
  • smart door locks can provide users with a variety of different unlocking methods, smart cat-eye functions, voice prompts, and low-battery reminders. Users can also interact with smart door locks through terminal devices. For example, users can view the cat-eye screen of the smart door lock in real time or review the cat-eye video of the smart door lock through the application (Application, APP) in the terminal device. Users can also communicate with the smart door lock through the application in the terminal.
  • application Application
  • the smart door locks only collect images and videos of the user's door. In this way, other users near the door can be seen from the images and videos. Similarly, when users are near other users' doors, they will also be collected by other users' smart door locks. This can easily infringe on the privacy of other users and leak the user's own privacy, resulting in poor privacy security.
  • the embodiments of the present application provide an image processing method and an electronic device, which blurs the privacy information corresponding to the privacy object in the first scene image based on the target depth data, thereby avoiding infringement of the privacy information of the privacy object and improving the security effect of the electronic device and the user experience.
  • an image processing method is provided, which is applied to an electronic device, and the method includes: obtaining a first scene image and a first depth image corresponding to the first scene image. Afterwards, if it is determined that the first scene image includes a target ROI region of interest, target depth data corresponding to the target ROI region is obtained based on the first depth image; the target ROI region includes privacy information corresponding to the privacy object. Finally, according to the target depth data corresponding to the target ROI region, blurring is performed in the target ROI region on the first scene image. If it is determined that the first scene image does not include the target ROI region, the first scene image is directly displayed.
  • the possibility of interaction between the object outside the door and the user is related to the distance corresponding to the target depth data. For example, if the object outside the door is far away from the door, the possibility of the object outside the door interacting with the user (such as the object outside the door visiting) is low, so the privacy information corresponding to the object outside the door can be blurred. For another example, if the object outside the door is close to the door, the possibility of the object outside the door interacting with the user is high, so the privacy information corresponding to the object outside the door does not need to be blurred. Moreover, if the object outside the door is close to the door, the object outside the door is less safe for the user and there are potential security issues, so there is no need to blur it.
  • the user can clearly view the object outside the door. If the object outside the door is far away from the door, the object outside the door is more safe for the user and there are no potential security issues, so it needs to be blurred. Thus, the privacy information of the object outside the door is avoided from being violated. At the same time, the privacy information of the object outside the door can also be protected to improve the security of the electronic device.
  • the process in the process of performing blurring processing in a target ROI area on a first scene image according to target depth data corresponding to the target ROI area, includes: blurring a privacy area in the target ROI area according to the target depth data corresponding to the target ROI area; wherein the privacy area is used to characterize an area corresponding to privacy information, and the privacy area includes a face area or a human shape area corresponding to a privacy object.
  • the method further includes: displaying the blurred first scene image, wherein the first scene image includes the blurred target ROI area. It can be seen that the present application can display the blurred first scene image in the electronic device, and the privacy information of the privacy object in the displayed first scene image is blurred. In this way, even if the The captured first scene image carries private information, and the user cannot see the private information of the private object when using the electronic device, thereby avoiding infringement of the private information of the private object and improving the user experience.
  • the electronic device includes a first image sensor and a second image sensor, the first image sensor is used to capture a first scene image, and the second image sensor is used to capture a first depth image; in the process of acquiring the first scene image and the first depth image corresponding to the scene image, it includes: when a first preset condition is met, controlling the first image sensor and the second image sensor to synchronously capture the first scene image and the first depth image.
  • the first preset condition includes: the electronic device detects that the sound of the privacy object is greater than the preset decibel, and/or the electronic device detects that the privacy object touches the electronic device, and/or the electronic device detects that the privacy object stays for more than a preset time.
  • the electronic device can continuously collect and process the first scene image and the first depth image in real time, in order to reduce the power consumption of the electronic device, the electronic device can collect and process the first scene image and the first depth image when the first preset condition is met. For example, the electronic device detects that the decibel of the privacy object is greater than the preset decibel, and/or detects the touch operation of the privacy object on the electronic device, and/or detects that the privacy object stays for more than the preset time, etc. In this way, after the privacy object is detected, the corresponding image processing is performed. The power consumption of the electronic device can be effectively reduced and the user experience can be improved.
  • the process of blurring the target ROI area according to the target depth data corresponding to the target ROI area includes: when it is determined that the target ROI area satisfies the second preset condition according to the target depth data corresponding to the target ROI area, blurring the target ROI area.
  • the second preset condition includes: the target depth value corresponding to the target depth data corresponding to the target ROI area is within the first preset range; or, the target depth value corresponding to the target depth data corresponding to the target ROI area is within the second preset range, and the confidence of the target depth data corresponding to the target ROI area is greater than the first preset threshold.
  • the second preset condition also includes: the privacy information in the target ROI area includes a face.
  • multiple different second preset conditions can be set to blur the target ROI area. For example, if the target depth value corresponding to the target depth data corresponding to the target ROI area is within the first preset range and the privacy information in the target ROI area includes a face, the target ROI area is blurred. For another example, if the target depth value corresponding to the target depth data corresponding to the target ROI area is within the second preset range, and the confidence of the target depth data corresponding to the target ROI area is greater than the first preset threshold. The target ROI area is blurred.
  • the privacy information of the privacy object can be effectively blurred, and the privacy information of the target visitor, that is, the non-privacy object, can also be obtained.
  • the security effect of the electronic device is guaranteed and the user experience is improved.
  • the process in a process of obtaining target depth data corresponding to a target ROI area based on a first depth image, includes: obtaining target depth data corresponding to the target ROI area based on the first depth image and a distance correction parameter; the distance correction parameter is obtained according to a calibration result of the first image sensor and the second image sensor, and the distance correction parameter is used to correct the depth data on the first depth image.
  • the calibration process includes: controlling the first image sensor and the second image sensor to synchronously capture the second scene image and the second depth image. Then, the calibration parameters corresponding to the first image sensor are calculated based on the second scene image. Finally, the calibration result is determined according to the calibration parameters corresponding to the first image sensor, and the calibration result includes the distance correction parameter corresponding to the second image sensor; the distance correction parameter is used to establish a mapping relationship between the first scene image and the first depth image.
  • the present application jointly calibrates the second image sensor and the first image sensor to determine the calibration result, and the calibration result includes a distance correction parameter to ensure the matching of visible light and infrared light wave images, and has the characteristics of high precision and good stability.
  • the depth data on the first depth image can be corrected using the distance correction parameter to obtain more accurate target depth data corresponding to the target ROI area. It is convenient to perform blur processing on the target ROI area according to the target depth data corresponding to the target ROI area.
  • the process of determining the target ROI area in the first scene image includes: identifying an initial ROI area including a target feature in the first scene image; wherein the target feature is used to characterize the privacy information corresponding to the privacy object, and the privacy information includes a face or a human shape corresponding to the privacy object. Afterwards, based on the initial ROI area, the target ROI area is determined.
  • the method includes: obtaining pixel data corresponding to the initial ROI region based on the first scene image. Then, obtaining initial depth data corresponding to the initial ROI region based on the first depth image and the pixel data. Then, correcting the initial depth data corresponding to the initial ROI region based on the distance correction parameter to obtain the initial ROI region. Finally, according to the target depth data corresponding to the initial ROI area, the initial ROI area is screened, and the screened initial ROI area is determined as the target ROI area.
  • the initial ROI area including the target feature can be pre-identified, and the target feature is used to characterize the privacy information corresponding to the privacy object, and the privacy information may include the face or human shape corresponding to the privacy object.
  • the initial depth data corresponding to the initial ROI area is obtained and corrected.
  • the initial ROI area is screened according to the target depth data corresponding to the initial ROI area, and the screened initial ROI area is determined as the target ROI area.
  • the process of screening the initial ROI area according to the target depth data corresponding to the initial ROI area includes: obtaining the confidence of the target depth data corresponding to the initial ROI area. Afterwards, the initial ROI area is screened according to the confidence and the effective distance range of the second image sensor; the effective distance range is set according to the hardware conditions of the second image sensor.
  • the present application can screen the initial ROI area according to the target depth data and the corresponding confidence level.
  • the purpose is to filter out invalid ROI areas.
  • a smart door lock which includes a memory, one or more processors and a camera; the memory is coupled to the processor; wherein the camera is used to collect a first scene image and a first depth image; a computer program code is stored in the memory, and the computer program code includes computer instructions.
  • the processor executes the following steps: the smart door lock obtains the first scene image and the first depth image corresponding to the first scene image. If it is determined that the first scene image includes a target ROI area of interest, the target depth data corresponding to the target ROI area is obtained based on the first depth image; the target ROI area includes privacy information corresponding to the privacy object.
  • the smart door lock performs blur processing in the target ROI area on the first scene image according to the target depth data corresponding to the target ROI area.
  • the privacy information of the privacy object can be avoided from being violated.
  • the privacy information of the privacy object can also be protected, which improves the security effect of the electronic device and the user experience.
  • the processor performs blurring in the target ROI area on the first scene image according to the target depth data corresponding to the target ROI area, including: the smart door lock blurs the privacy area in the target ROI area according to the target depth data corresponding to the target ROI area; wherein the privacy area is used to characterize the area corresponding to the privacy information, and the privacy area includes the face area or human shape area corresponding to the privacy object. It can be seen that when the smart door lock blurs the target ROI area, it can only blur the privacy area in the target ROI area, so as to more accurately protect the privacy information of the privacy object and improve the security of the smart door lock.
  • the processor further performs the following steps: displaying the blurred first scene image, the first scene image including the blurred target ROI area.
  • the smart door lock can display the blurred first scene image on the display screen, the first scene image including the blurred target ROI area. In this way, even if the captured first scene image carries privacy information, the user cannot see the privacy information of the privacy object when using the smart door lock. Avoiding the infringement of the privacy information of the privacy object improves the security effect of the electronic device and the user experience.
  • an electronic device comprising a memory and one or more processors; the memory is coupled to the processor; wherein computer program code is stored in the memory, and the computer program code comprises computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the image processing method as described in the first aspect.
  • a computer-readable storage medium in which instructions are stored.
  • the computer can execute the image processing method as described in the first aspect.
  • FIG1 is a schematic diagram of a scene of an image processing method provided in an embodiment of the present application.
  • FIG2 is a schematic diagram of the hardware structure of a smart door lock provided in an embodiment of the present application.
  • FIG3 is a schematic diagram of the software structure of a smart door lock provided in an embodiment of the present application.
  • FIG4 is a schematic diagram of a calibration process provided in an embodiment of the present application.
  • FIG5 is a two-dimensional plane schematic diagram of a calibration process of a first image sensor and a second image sensor provided in an embodiment of the present application;
  • FIG6 is a three-dimensional schematic diagram of a calibration process of a first image sensor and a second image sensor provided in an embodiment of the present application;
  • FIG7 is a schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application.
  • FIG8 is a schematic diagram of a flow chart of a calibration process of a first image sensor and a second image sensor provided in an embodiment of the present application;
  • FIG9 is a schematic diagram of an initial ROI region and corresponding region types provided in an embodiment of the present application.
  • FIG10 is a schematic diagram of a local area in a blurred target ROI region provided in an embodiment of the present application.
  • FIG11 is a schematic diagram of blurring all target ROI areas provided by an embodiment of the present application.
  • FIG. 12 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
  • At least one of the following or similar expressions refers to any combination of these items, including any combination of single items or plural items.
  • at least one of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple.
  • the words "first", “second” and the like are used to distinguish the same items or similar items with substantially the same functions and effects.
  • the network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
  • smart door locks can provide users with a variety of different unlocking methods, smart cat-eye functions, voice prompts, and low-battery reminders. Users can also interact with smart door locks through terminal devices. For example, users can view the cat-eye screen of the smart door lock in real time or review the cat-eye video of the smart door lock through the application (Application, APP) in the terminal device.
  • application Application, APP
  • the smart door lock collects images and videos outside the user's home. From the images and videos, you can see the users outside the door near the door. Among them, the users outside the door include the target visitors at home and the non-target visitors other than the target visitors. In this way, it is easy to infringe the privacy of non-target visitors. Similarly, if the user himself is not the target visitor, he will be collected by the smart door lock of other users. In this way, the user's own privacy will also be leaked, resulting in poor privacy security.
  • the present application provides an image processing method that can be applied to electronic devices.
  • the electronic device can be an intelligent device with a camera, such as intelligent cat's eye, intelligent door lock, intelligent surveillance camera and other intelligent security equipment, and can also include mobile phones, tablet computers, laptops, ultra-mobile personal computers (ultra-mobile personal computers, UMPC), netbooks, personal digital assistants (personal digital assistant, PDA), wearable devices, artificial intelligence (artificial intelligence, AI) devices or other devices.
  • a camera such as intelligent cat's eye, intelligent door lock, intelligent surveillance camera and other intelligent security equipment
  • mobile phones tablet computers, laptops, ultra-mobile personal computers (ultra-mobile personal computers, UMPC), netbooks, personal digital assistants (personal digital assistant, PDA), wearable devices, artificial intelligence (artificial intelligence, AI) devices or other devices.
  • UMPC ultra-mobile personal computers
  • PDA personal digital assistant
  • wearable devices wearable devices
  • AI artificial intelligence
  • a smart door lock has the function of a smart cat's eye.
  • a smart cat's eye can be installed on a smart door lock.
  • a smart door lock can collect scene images outside the door and perform blurring and other processing, and can also display the processed images on a display screen configured with the smart door lock.
  • the user can decide whether to hide the lock according to the distance between the user outside the door and the smart door lock.
  • Privacy information of users outside the door For example, by using the image sensor (color sensor) and the depth image sensor (TOF sensor) in the smart door lock to work together, more abundant image information about users outside the door can be obtained.
  • the first image sensor is used to collect scene images, and the first image sensor can be, for example, an RGB sensor; the second image sensor is used to collect depth images, and the second image sensor can be, for example, a TOF sensor.
  • the image sensor is referred to as the first image sensor
  • the depth image sensor is referred to as the second image sensor.
  • the distance between the user outside the door and the smart door lock is obtained based on the image information.
  • the smart door lock can hide the privacy information about the user outside the door in the image information.
  • the user outside the door whose privacy information needs to be hidden is a non-target visitor. For example, a non-target visitor who passes by the house by chance. In this way, by hiding their privacy information, the privacy corresponding to the non-target visitor is avoided from being violated.
  • the smart door lock can obtain the key information of the target visitor, such as the face information of the target visitor, the action information of the target visitor, and the information of the target visitor's carried items, etc.
  • the key information can also be processed to hide the privacy information of the non-target visitor. While avoiding the violation of the privacy of non-target visitors, the security effect of the smart door lock is guaranteed, the safety performance of the smart door lock is improved, and the user's experience is improved.
  • the non-target visitor is referred to as a privacy object
  • the target visitor is referred to as a non-privacy object.
  • FIG. 2 shows a schematic diagram of the hardware structure of the smart door lock 500 .
  • the smart door lock 500 may include a processor 510, an internal memory 520, a universal serial bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, an antenna, a wireless communication module 550, an audio module 560, a speaker 560A, a sensor module 570, a button 580, an indicator 581, a camera 582, a display screen 583, etc.
  • the sensor module 570 may include a gyroscope sensor 570A, an acceleration sensor 570B, a fingerprint sensor 570C, a touch sensor 570D, an ambient light sensor 570E, a first image sensor 570F, and a second image sensor 570G, etc.
  • the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the smart door lock 500.
  • the smart door lock 500 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently.
  • the components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
  • the processor 510 may include one or more processing units, for example, the processor 510 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc.
  • AP application processor
  • GPU graphics processor
  • ISP image signal processor
  • controller a memory
  • DSP digital signal processor
  • DSP digital signal processor
  • NPU neural-network processing unit
  • Different processing units may be independent devices or integrated into one or more processors.
  • the controller may be the nerve center and command center of the electronic device 100.
  • the controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
  • the processor 510 may also be provided with a memory for storing instructions and data.
  • the memory in the processor 510 is a cache memory.
  • the memory may store instructions or data that the processor 510 has just used or cyclically used. If the processor 510 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 510, and thus improves the efficiency of the system.
  • the processor 510 may include one or more interfaces.
  • the interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (SIM) interface, and/or a universal serial bus (USB) interface, etc.
  • I2C inter-integrated circuit
  • I2S inter-integrated circuit sound
  • PCM pulse code modulation
  • UART universal asynchronous receiver/transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB universal serial bus
  • the I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL).
  • the processor 510 may include multiple groups of I2C buses.
  • the processor 510 may couple the touch sensor 570D through the I2C interface, so that the processor 510 and the touch sensor 570D communicate through the I2C bus interface to realize the touch function of the smart door lock 500.
  • the I2S interface can be used for audio communication.
  • the processor 510 can include multiple I2S buses. 510 can be coupled to the audio module 560 through the I2S bus to achieve communication between the processor 510 and the audio module 560.
  • the audio module 560 can transmit an audio signal to the wireless communication module 550 through the I2S interface to achieve the function of inputting voice commands through the smart door lock 500.
  • the PCM interface can also be used for audio communication, and the audio module 560 can also transmit audio signals to the wireless communication module 550 through the PCM interface, thereby realizing the function of inputting voice commands through the smart door lock 500.
  • the UART interface is a universal serial data bus for asynchronous communication.
  • the bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • the UART interface is generally used to connect the processor 510 and the wireless communication module 550.
  • the processor 510 communicates with the Bluetooth module in the wireless communication module 550 through the UART interface to implement Bluetooth functions such as sending or scanning Bluetooth broadcasts.
  • the MIPI interface can be used to connect the processor 510 with peripheral devices such as the display screen 583 and the camera 582.
  • the processor 510 and the camera 582 communicate via the CSI interface to realize the shooting function of the smart door lock 500.
  • the processor 510 and the display screen 583 communicate via the DSI interface to realize the display function of the smart door lock 500.
  • the GPIO interface can be configured by software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 510 with the camera 582, the display screen 583, the wireless communication module 550, the audio module 560, the sensor module 570, etc.
  • the USB interface 530 is an interface that complies with USB standard specifications, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.
  • the USB interface 530 may be used to connect a charger to charge the smart door lock 500, and may also be used to transmit data between the smart door lock 500 and peripheral devices.
  • the charging management module 540 is used to receive charging input from a charger.
  • the charger may be a wireless charger or a wired charger.
  • the charging management module 540 may receive charging input from a wired charger through the USB interface 530.
  • the charging management module 540 may receive wireless charging input through a wireless charging coil of the smart door lock 500. While the charging management module 540 is charging the battery 542, it may also power the smart door lock 500 through the power management module 541.
  • the power management module 541 is used to connect the battery 542, the charging management module 540 and the processor 510.
  • the power management module 541 receives input from the battery 542 and/or the charging management module 540, and supplies power to the processor 510, the internal memory 520, the external memory, the display screen 583, the camera 582, and the wireless communication module 550.
  • the power management module 541 can also be used to monitor parameters such as battery capacity, battery cycle number, and battery health status (leakage, impedance).
  • the wireless communication function of the smart door lock 500 can be realized through an antenna and a wireless communication module 550, a modem processor, a baseband processor, and the like.
  • the antenna is used to transmit and receive electromagnetic wave signals.
  • Each antenna in the smart door lock 500 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna.
  • the wireless communication module 550 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network) and Bluetooth (BT) for application in the smart door lock 500.
  • WLAN wireless local area networks
  • BT Bluetooth
  • the wireless communication module 550 can be one or more devices integrating at least one communication processing module.
  • the wireless communication module 550 receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 510.
  • the wireless communication module 550 can also receive the signal to be sent from the processor 510, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna.
  • the smart door lock 500 implements the display function through a GPU, a display screen 583, and an application processor.
  • the GPU is a microprocessor for image processing, which connects the display screen 583 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations for graphics rendering.
  • the processor 510 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 583 is used to display images, videos, etc.
  • the display screen 583 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (QLED), etc.
  • the smart door lock 500 may include 1 or N display screens 583, where N is a positive integer greater than 1.
  • the smart door lock 500 can be connected to the ISP, the camera 582, the video codec, the GPU, the display 583 and the application processor. etc. to realize the shooting function.
  • the ISP is used to process the data fed back by the camera 582. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye.
  • the ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 582.
  • the camera 582 is used to capture still images or videos.
  • the object generates an optical image through the lens and projects it onto the photosensitive element.
  • the photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor.
  • CMOS complementary metal oxide semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal.
  • the ISP outputs the digital image signal to the DSP for processing.
  • the DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.
  • the smart door lock 500 may include 1 or N cameras 582, where N is a positive integer greater than 1.
  • NPU is a neural network (NN) computing processor.
  • NN neural network
  • the intelligent door lock 500 can realize intelligent cognition and other applications, such as image recognition, face recognition, voice recognition, text understanding, etc.
  • the internal memory 520 can be used to store computer executable program codes, which include instructions.
  • the processor 510 executes various functional applications and data processing of the smart door lock 500 by running the instructions stored in the internal memory 520.
  • the internal memory 520 may include a program storage area and a data storage area.
  • the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.
  • the data storage area may store data (such as password data, audio data, etc.) created during the use of the smart door lock 500, etc.
  • the internal memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
  • UFS universal flash storage
  • the smart door lock 500 can implement audio functions through the audio module 560, the speaker 560A, and the application processor.
  • the speaker 560A also called a "speaker", is used to convert audio electrical signals into sound signals.
  • the smart door lock 500 can listen to the prompt tone through the speaker 560A.
  • the gyroscope sensor 570A can be used to determine the motion posture of the smart door lock 500. In some embodiments, the gyroscope sensor 570A can be used to determine whether the smart door lock 500 is in a moving state.
  • the distance sensor 570B is used to measure the distance.
  • the smart door lock 500 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the smart door lock 500 can use the distance sensor 570B to measure the distance to achieve fast focusing.
  • the ambient light sensor 570E is used to sense the brightness of the ambient light.
  • the smart door lock 500 can adaptively adjust the brightness of the display screen 583 according to the perceived brightness of the ambient light.
  • the ambient light sensor 570E can also be used to automatically adjust the white balance when taking pictures.
  • the fingerprint sensor 570C is used to collect fingerprints.
  • the smart door lock 500 can use the collected fingerprint characteristics to realize fingerprint unlocking.
  • the touch sensor 570D is also called a "touch panel”.
  • the touch sensor 570D can be set on the display screen 583.
  • the touch sensor 570D and the display screen 583 form a touch screen, also called a "touch screen”.
  • the touch sensor 570D is used to detect a touch operation acting on or near it.
  • the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
  • Visual output related to the touch operation can be provided through the display screen 583.
  • the first image sensor 570F is used to convert the visible light signal in the received reflected light into visible light information, which exists in the form of an electrical signal, and then generate a visible light image based on the visible light information.
  • the visible light image can also be called a scene image.
  • the first image sensor 570F can be used to collect a scene image in a scene outside the door.
  • the second image sensor 570G usually measures the depth data of the measured object (or measured target) by the time-of-flight method.
  • the time-of-flight method measures the time interval T from the emission to the reception of the pulse signal actively emitted by the measuring instrument (often referred to as the pulse ranging method) or the phase difference (phase difference ranging method) generated by the laser traveling back and forth to the measured object once, and converts it into the distance of the photographed scene, which is used to generate depth data to achieve the measurement of the three-dimensional structure or three-dimensional contour of the measured object (or the detection area of the measured object), and then obtain the grayscale image and depth data of the measured object.
  • TOF time-of-flight method
  • the second image sensor converts the infrared light signal in the received reflected light into depth information, which exists in the form of an electrical signal, and then generates a grayscale image and depth data based on the depth information.
  • the second image sensor 570G can be used to collect depth data between the user outside the door and the smart door lock.
  • the first image sensor 570F and the second image sensor 570G may also be disposed in one or more Among the several cameras 582.
  • the embodiment of the present application does not specifically limit the arrangement of components.
  • the button 580 includes a power button, a volume button, etc.
  • the button 580 may be a mechanical button or a touch button.
  • the smart door lock 500 may receive a button input and generate a key signal input related to the user settings and function control of the smart door lock 500.
  • Indicator 581 can be an indicator light, which can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, etc.
  • FIG3 shows a schematic diagram of the software structure of the smart door lock 500 .
  • the software system of the smart door lock 500 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the smart door lock 500 is illustrated.
  • the layered architecture divides the software into several layers, each with a clear role and division of labor.
  • the layers communicate with each other through software interfaces.
  • the system is divided into three layers, from top to bottom: application layer, hardware abstraction layer and kernel layer.
  • the application layer can include a series of application packages.
  • the application package may include a first application, a second application, and a third application.
  • the first application is used for wandering detection.
  • the first application can detect whether a target stays in a specified area for more than a certain period of time.
  • the second application is used to talk with visitors.
  • the third application is used to intelligently capture the specified area.
  • the Hardware Abstraction Layer is located between the application layer and the kernel layer, and is used to connect the application layer and the kernel layer, so that the upper-layer business software is basically unaware of the hardware changes.
  • the hardware abstraction layer can realize the functions of the application framework layer, for example, providing an application programming interface (API) and a programming framework for the application of the application layer. Therefore, the hardware abstraction layer can include some pre-defined functions.
  • the hardware abstraction layer further includes a plurality of modules, each module corresponding to a type of hardware, such as a video input (VI) module, a video processing subsystem (VPSS) module, a video encoding (VENC) module, a region of interest (ROI) identification module, a ROI region processing module, a calibration module, and a video output (VO) module, etc.
  • the video processing subsystem module can be electrically connected to the video input module, the video output module, and the video encoding module, etc.
  • the calibration module is used to calibrate the first image sensor and the second image sensor.
  • the video input module is used to receive multiple video signals and transmit them to the video processing subsystem module.
  • the video processing subsystem module is used to receive decoded data and perform video processing, and transmit the processed video to the ROI area identification module.
  • the VPSS module can also be used to implement video processing functions. For example, it supports video cropping, setting border size and color configuration, block processing, video occlusion, video overlay, and video data compression and other video processing functions.
  • the ROI region recognition module is used to identify the ROI region in the video.
  • the ROI region processing module is used to process the ROI region in the video and transmit the processed video to the video encoding module.
  • the ROI region processing module can screen the identified ROI region and eliminate invalid ROI regions.
  • the identified ROI region can also be divided into regions to obtain ROI foreground regions and ROI background regions.
  • the video encoding module is used to receive the processed video sent by the ROI area processing module and encode the video.
  • the video encoding module can be used to blur the ROI area in the video.
  • the video output module is used to output the blurred video.
  • the kernel layer is a layer between hardware and software.
  • the kernel layer at least includes a first image sensor driver and a second image sensor driver, and so on.
  • the electronic device may include a camera, a processor, and a display screen
  • the camera is used to capture scene images
  • the processor is used to process the scene images captured by the camera, such as performing module processing, etc.
  • the display screen is used to display the scene images processed by the processor.
  • the electronic device may be a smart cat's eye, a smart door lock, or a smart door lock with a smart cat's eye function.
  • the camera, the processor, and the display screen may be arranged together or separately.
  • the electronic device is a smart door lock
  • the camera may be located on the outside of the door
  • the display screen may be located on the inside of the door.
  • the electronic device may include only a camera and a processor, but not a display screen.
  • the processor may perform blurring and other processing on the scene image captured by the camera.
  • the electronic device may send the processed scene image to other devices for display.
  • the electronic device may be a smart door lock without a display screen.
  • the smart door lock may send the processed scene image to other devices for display. Display to the mobile phone.
  • multiple image sensors may be provided in the same camera.
  • the first image sensor and the second image sensor are both provided in one camera.
  • these sensors are used to image the same photographed object.
  • a spectrometer may be provided between the lens and the sensor to decompose the light entering from one lens onto multiple sensors to ensure that each sensor receives the light.
  • the number of processors may be one or more. Multiple image sensors may cooperate to capture images of photographed objects and obtain the distance of the photographed objects.
  • a first image sensor and a second image sensor are provided in the smart door lock 500, and the second image sensor and the first image sensor can be respectively provided in their respective depth image cameras and image cameras, or can be provided in one camera.
  • the depth image camera can also be referred to as a depth image camera, and the image camera can also be referred to as an image camera.
  • the photosensitive surfaces of the second image sensor and the first image sensor can be arranged in parallel. Since the second image sensor collects infrared light wave images, and the first image sensor collects visible light images. Therefore, arranging the two in parallel can ensure the matching of visible light and infrared light wave images, so as to obtain richer image information.
  • other types of image sensors and distance sensors may also be provided in the smart door lock 500.
  • Other types of image sensors may also include color mark sensors.
  • Other types of distance sensors may include laser distance sensors, ultrasonic sensors, infrared ranging sensors, etc. It is understandable that other types of image sensors and other types of distance sensors may also be provided in the smart door lock in an embodiment of the present application to facilitate obtaining richer image information.
  • camera calibration is usually performed to make the image information collected by the first image sensor and the depth information collected by the second image sensor more accurate.
  • calibration refers to the process of determining the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and the corresponding point in the image captured by the first image sensor and the second image sensor.
  • these geometric model parameters can be summarized as calibration parameters.
  • the calibration parameters include internal parameters, external parameters, and distortion parameters.
  • the calibration parameters can be solved through experiments and calculations, and the process of solving the calibration parameters through experiments and calculations is called the calibration process.
  • the calibration process is an important link in fields such as machine vision/computer vision. Therefore, doing a good job in the calibration process is a prerequisite for doing a good job in subsequent work, and completing the calibration process quickly and accurately is the basis for improving the efficiency of subsequent work.
  • the smart door lock of the embodiment of the present application completes the calibration process of the first image sensor and the second image sensor by determining the conversion relationship between the three-dimensional space point and the pixel plane pixel point in the world coordinate system and the distortion coefficient in the imaging process, so as to facilitate subsequent image processing. And the smart door lock can register and fuse the image data and depth data collected by the first image sensor and the second image sensor to obtain an information-rich image.
  • the basic principle involved in the calibration process is that objects in real space are three-dimensional, while objects in their corresponding images are two-dimensional. Therefore, it can be considered that there is a three-dimensional to two-dimensional geometric model between objects in a three-dimensional scene and their corresponding two-dimensional images, and this geometric model enables the objects in the three-dimensional scene to form a conversion from three-dimensional to two-dimensional or two-dimensional to three-dimensional. Then, it is easy to understand that when the first image sensor and the second image sensor capture images of objects in a three-dimensional scene, the first image sensor and the second image sensor can be considered as this geometric model, and the calibration parameters are the parameters of this geometric model. Therefore, as long as the calibration parameters are obtained, the world coordinates of the object in space can be inferred from the pixel coordinates of the object in the image, thereby realizing functions such as visual detection, biometric recognition, distance measurement, and three-dimensional reconstruction.
  • An embodiment of the present application provides an image processing method that can be applied to an electronic device, and the method includes: first, calibrating a first image sensor and a second image sensor in the electronic device to obtain a calibration result. Afterwards, synchronously acquiring a first scene image captured by the first image sensor and a first depth image captured by the second sensor. In the case where it is determined that the first scene image includes a target ROI region of interest, target depth data corresponding to the target ROI region is acquired based on the first depth image. Among them, the target ROI region includes privacy information corresponding to a privacy object; then, according to the target depth data corresponding to the target ROI region, blurring is performed in the target ROI region on the first scene image.
  • the privacy information corresponding to the privacy object in the first scene image can be hidden to avoid infringing the privacy information of the privacy object, and the privacy information of the privacy object can also be protected.
  • the following will take the smart door lock as an example to specifically illustrate the image processing method provided by the embodiment of the present application.
  • the embodiment shows a flow chart of an image processing method, as shown in FIG7 .
  • the method may include the following steps S701 - S708 .
  • Step S701 The smart door lock calibrates the first image sensor and the second image sensor.
  • the first image sensor and the second image sensor are usually installed on the same plane, and the installation positions cannot overlap. Therefore, the first image sensor and the second image sensor will form two conical fields of view according to their own visual angles, and the two fields of view are projected on the plane as shown in Figure 6. It should be noted that the field of view angle of the receiving lens of the first image sensor can be greater than the field of view angle of the receiving lens of the second image sensor. In other words, the field of view ⁇ of the first image sensor can cover the field of view ⁇ of the second image sensor.
  • the purpose is to intercept the field of view of the infrared light wave within the field of view of the visible light, so that the visible light image collected by the first image sensor and the depth data in the infrared light wave image collected by the second image sensor can correspond one to one, which is convenient for subsequent image processing.
  • FIG 5 is a top view of the user outside the door shown in the embodiment of the present application.
  • position 1 is the position where the user stands outside the door and is a certain distance away from the smart door lock.
  • the second image sensor captures and analyzes the depth image, and it can be obtained that the depth value L2 between the smart door lock and the user is 3 meters.
  • the first image sensor captures and analyzes the scene image, and it can be obtained that the distance between the smart door lock and the user in the scene image is L1. It can be seen that L1 is equal to L2 and is also 3 meters.
  • the distance between the smart door lock and the user in the parsed scene image is the vertical distance between the user and the smart door lock in the space, that is, 3 meters.
  • the depth image collected by the second image sensor is analyzed, and the distance between the smart door lock and the user is L3. It can be seen that L3 is greater than L2 and should be equal to L2/sin( ⁇ ). But in fact, even if the user translates to the left to position 2, the distance between the smart door lock and the user should be 3 meters. Therefore, in order to make the depth data collected by the second image sensor the vertical distance between the user and the smart door lock.
  • the embodiment of the present application can jointly calibrate the second image sensor and the first image sensor to ensure the matching of visible light and infrared light wave images, and has the characteristics of high precision and good stability.
  • the calibration process provided by the embodiment of the present application may include: first, controlling the first image sensor and the second image sensor to synchronously capture the second scene image and the second depth image. Then, the calibration parameters corresponding to the first image sensor are calculated based on the second scene image. Finally, the calibration result is determined according to the calibration parameters corresponding to the first image sensor, and the calibration result includes the distance correction parameters corresponding to the second image sensor; wherein the distance correction parameters are used to establish a mapping relationship between the first scene image captured by the first image sensor and the first depth image captured by the second image sensor.
  • the process of calibrating the first image sensor and the second image sensor includes the following steps:
  • Step S801 The smart door lock controls the first image sensor and the second image sensor to synchronously capture a second scene image and a second depth image.
  • the second scene image may include a red, green, blue (RGB) image.
  • the second scene image and the second depth image are the same image synchronously collected on the same plane along the same shooting direction.
  • the second depth image includes depth data from the shooting object corresponding to each pixel point on the second scene image to the above-mentioned plane.
  • a calibration plate is usually configured in a common field of view of the first image sensor and the second image sensor.
  • the purpose of configuring the calibration plate is to enable the first image sensor and the second image sensor to simultaneously shoot the calibration plate, thereby obtaining a second scene image and a second depth image corresponding to the calibration plate, so as to facilitate subsequent calibration of the first image sensor and the second image sensor.
  • the calibration plate can be placed without constraints.
  • the position of the calibration plate can be changed arbitrarily.
  • the calibration plate is translated left and right along the same plane, and the second scene image and the second depth image can be repeatedly collected after each change of the position of the calibration plate.
  • the subsequent calibration parameters of the first image sensor and the second image sensor need to be calculated based on the second scene image and the second depth image. Therefore, in an embodiment of the present application, multiple groups of calibration plates at different positions can be collected at the same time. In this way, the first image sensor and the second image sensor can be calibrated based on multiple groups of second scene images and second depth images to improve the calibration accuracy.
  • a preset pattern can be set on the calibration plate, such as a printed checkerboard pattern, a dot pattern, a QR code pattern or other specific patterns.
  • any pattern unit therein for example, any square of a checkerboard pattern, any dot of a dot pattern or an area of any shape located in the center can be called a landmark point for subsequent calculation of calibration parameters.
  • the first image sensor and the second image sensor synchronously capture the second scene image and the second depth image, and it is necessary to keep the same image and the size equal.
  • the resolution i.e., pixels
  • the second scene image captured by the first image sensor can be pre-processed by downsampling or other processing methods to obtain a second scene image with the same resolution as the second depth image.
  • Step S802 The smart door lock calculates calibration parameters corresponding to the first image sensor based on the second scene image.
  • Step S803 The smart door lock determines a calibration result based on the calibration parameters corresponding to the first image sensor, where the calibration result includes a distance correction parameter corresponding to the second image sensor.
  • the dot located in the center of each group of second scene images is selected as a marker point.
  • the calibration parameters corresponding to the first image sensor can be calculated through the marker points in the collected multiple groups of second scene images and the conversion matrix between each coordinate system.
  • the calibration parameters may include the intrinsic parameter matrix corresponding to the first image sensor, the extrinsic parameter matrix corresponding to the first image sensor, and the distortion coefficient corresponding to the first image sensor, etc. It should be noted that the embodiment of the present application does not make specific limitations on the marker points and the method for calculating the calibration parameters. Those skilled in the art can choose the marker points and calculation methods according to actual needs. These designs do not exceed the protection scope of the embodiments of the present application.
  • the embodiment of the present application can determine the distance correction parameters corresponding to the second image sensor according to the calibration parameters corresponding to the first image sensor.
  • the captured second scene image is distorted, such as when the marker point in the second scene image is translated to the left by 20 pixels relative to the marker point in the calibration plate image.
  • the depth value corresponding to the marker point in the depth image will also be different from the depth value corresponding to the marker point in the calibration plate image. Therefore, the second image sensor can be calibrated according to the calibration parameters corresponding to the second scene image to complete the joint calibration between the first image sensor and the second image sensor.
  • the landmark point in the captured second scene image can be defined as the user in the image.
  • the second scene image is parsed. It can be concluded that no matter the user is in a non-moving state or a moving state, the distance between the user and the first image sensor within the spatial range is constant, which is the vertical distance, i.e., L4.
  • the distance measurement principle of the second image sensor is to determine the distance between the second image sensor and the object or the surrounding environment, that is, the time taken by the emitted light to reach the object and reflect back to the second image sensor is measured by the second image sensor, and converted into distance. Therefore, after parsing the depth image collected by the second image sensor, it can be concluded that the distance between the user and the second image sensor is L6. In the actual space range, the distance between the second image sensor and the user should be L5. And L5 should also be the distance in the vertical direction, and is equal to L4.
  • the distance correction parameter can be calculated based on the coordinates of the marker point in the second scene image and the calibration parameters corresponding to the first image sensor, so as to correct the distance L6 between the user in the moved state and the second image sensor to the distance L5 between the user in the non-moved state and the second image sensor.
  • the distance correction parameter may include the sine value of the offset angle ⁇ of the marker point relative to the second image sensor.
  • Step S804 The smart door lock corrects the depth data in the second depth image according to the distance correction parameter to complete the calibration process of the first image sensor and the second image sensor.
  • the depth data corresponding to the second depth image output by the second image sensor can be corrected according to the distance correction parameter, thereby completing the joint calibration of the first image sensor and the second image sensor.
  • the angle ⁇ of the marker point relative to the second image sensor is 30°
  • the depth value in the vertical direction between the second image sensor and the user is 0.5 meters
  • the depth data output by the second image sensor that is, the depth value
  • the sine value of the offset angle 30° by the depth value of 1 meter to obtain the corrected depth value of 0.5 meters.
  • the depth data output by the second image sensor after correction is the same as the depth value in the vertical direction between the second image sensor and the user within the actual space range, completing the calibration process.
  • the embodiment of the present application can map the depth value in the second depth image to the second scene image, and then establish a mapping relationship between the visible light image and the infrared light wave image. And establish a corresponding relationship between the depth data (such as the first depth image) subsequently collected by the second image sensor and the image data (such as the first scene image) collected by the first image sensor.
  • the depth data corresponding to the second scene image in the same field of view can be synchronously acquired.
  • rapid calibration can be achieved. Accurately align scene image data and depth data to improve calibration accuracy and simplify the calibration process.
  • the images captured by the first image sensor and the second image sensor during the calibration process or during actual use are usually different.
  • the first image sensor and the second image sensor capture images including the calibration plate during the calibration process, while the first image sensor and the second image sensor capture images outside the door during actual use. Therefore, for ease of description, the first image sensor and the second image sensor capture the second scene image and the second depth image during the calibration process. In the subsequent actual use process, the first image sensor and the second image sensor capture the first scene image and the first depth image.
  • Step S702 If the smart door lock meets the first preset condition, the first image sensor and the second image sensor are controlled to synchronously capture the first scene image and the first depth image.
  • smart door locks can continuously collect scene images outside the user's home. However, when there is no user outside the door, the smart door lock does not need to process key information of the collected images and hide privacy information.
  • the embodiment of the present application can set a first preset condition. If the smart door lock meets the first preset condition, it can be determined that there is a user outside the door. Then, the first image sensor and the second image sensor are controlled to synchronously capture the first scene image and the first depth image, and key information processing is performed based on the first scene image and the first depth image to hide the privacy information.
  • the first preset condition includes: the smart door lock detects that the decibel of sound of the privacy object is greater than the preset decibel, and/or the smart door lock detects that the privacy object touches the smart door lock, and/or the smart door lock detects that the privacy object stays for a time greater than a preset time.
  • the voice of the user outside the door exceeds the preset decibel.
  • Another example is the user outside the door touching the smart door lock.
  • the smart door lock detects the user outside the door touching the smart door lock, or detects that the voice of the user outside the door exceeds the preset decibel, it can be determined that there is a user outside the door, that is, the first preset condition is met.
  • the smart door lock does not need to hide the privacy information in the image.
  • the user outside the door stays in front of the smart door lock for more than or equal to 3 seconds.
  • the smart door lock detects that the user outside the door stays in front of the smart door lock for more than or equal to 3 seconds, it can be determined that the user outside the door is not passing by the user's home for a short time, that is, the first preset condition is met.
  • a distance sensor and a single-chip microcomputer may be provided in the smart door lock.
  • the distance sensor can be used to measure the distance
  • the single-chip microcomputer is used for timing.
  • the smart door lock can measure the distance through infrared or laser in the distance sensor. For example, when the distance sensor detects a user outside the door, the distance sensor continuously sends a first signal to the single-chip microcomputer, and the first signal is used to trigger the single-chip microcomputer to start timing.
  • the single-chip microcomputer starts timing after receiving the first signal. If the single-chip microcomputer timing exceeds the preset threshold, and can still continue to receive the first signal sent by the distance sensor, it can be determined that the user outside the door is staying outside the door.
  • the smart door lock can collect images and perform key information processing on the images to hide privacy information, wherein the key information includes face information, action information, and information about items carried by the user outside the door.
  • the smart door lock in the process of continuously collecting scene images, can use a camera with lower power consumption to collect images, such as using a camera with poor clarity other than the first image sensor and the second image sensor.
  • the first image sensor and the second image sensor are controlled to collect the first scene image and the first depth image to achieve the subsequent hiding of the privacy information in the image.
  • the smart door lock can also periodically collect scene images outside the user's home at preset intervals to reduce power consumption.
  • the smart door lock can always use the first image sensor and the second image sensor to collect images.
  • the smart door lock meets the first preset condition, it is necessary to perform key information processing on the collected first scene image and the second depth image, such as hiding privacy information.
  • the smart door lock does not meet the first preset condition, it is not necessary to perform key information processing on the collected first scene image and the second depth image.
  • the smart door lock sets a first preset condition and collects the first scene image and the second depth image after the first preset condition is met to perform subsequent operations/tasks to hide the privacy information of the user outside the door.
  • the key information of the user outside the door can be processed to achieve the purpose of protecting the user's privacy information.
  • the smart door lock does not meet the first preset condition, there is no need to collect the first scene image and the second depth image and perform subsequent processing. In this way, the scene recognition of the user outside the door can be improved, and the power consumption of the smart door lock can be reduced.
  • Step S703 The smart door lock identifies an initial region of interest (ROI) including target features in the first scene image.
  • ROI initial region of interest
  • the initial ROI region including the target features represents the region corresponding to the privacy information of the user outside the door.
  • the target features include face features and human shape features.
  • the first scene image and the first depth image are obtained. Based on the first scene image, an initial ROI region including the target feature is identified, and the initial ROI region represents the region corresponding to the privacy information of the user outside the door. Therefore, by identifying and processing the initial ROI region including the target feature, the privacy information of the user outside the door can be avoided from being leaked while protecting the user's own privacy information. At the same time, there is no need to process other parts of the first scene image except the initial ROI region, thereby improving processing efficiency.
  • the first scene image can be input into the target model for detection to obtain a detection frame containing target features such as facial features or human-shaped features and a corresponding region type.
  • the region type is used to characterize that the initial ROI region includes a facial region or a human-shaped region.
  • the region in the detection frame is determined as the initial ROI region.
  • the detection frame in the present application can be a rectangle of appropriate size, and the initial ROI region can be a rectangular region of different sizes.
  • the initial ROI area refers to an image area selected from the image, which is the focus of image analysis.
  • the embodiment of the present application can reduce image processing time and improve image processing accuracy by delineating an area including target features as a prerequisite for further processing of the image.
  • the initial ROI area can be a rectangular area, a circular area or an area of any shape. In the embodiment of the present application, it can be a rectangular area.
  • the inclusion of target features in the initial ROI area means that the initial ROI area includes a face area and a human shape area.
  • the smart door lock can identify the initial ROI area containing facial features or human-shaped features in the first scene image and the corresponding area type based on the collected first scene image.
  • the area type is used to characterize that the initial ROI area includes a facial area or a human-shaped area.
  • the facial area is used to characterize the facial information corresponding to the user outside the door.
  • the human-shaped area is used to characterize the appearance contour information of the user outside the door. It can be understood that the initial ROI area containing the target features identified by the present application can all characterize the privacy information corresponding to the user outside the door. So as to facilitate the subsequent operation/task of hiding or not hiding the privacy information of the user outside the door on the initial ROI area.
  • the privacy information corresponding to the user outside the door includes but is not limited to the face area and the human shape area in the above-mentioned initial ROI area.
  • the process of identifying the initial ROI area includes but is not limited to the target features such as the above-mentioned face features and human shape features.
  • the target features may also include specific items carried by the user, the identification of specific items, and the features corresponding to the pets carried.
  • the specific item may be an ID card.
  • the specific item may also be a backpack, and the identification of the specific item may be the brand corresponding to the backpack.
  • the specific item may also be a smart device in use. Therefore, the above-mentioned target features corresponding to the privacy information may all be used as objects for identification.
  • the first scene image may include one or more initial ROI areas.
  • Each initial ROI area corresponds to an area type.
  • initial ROI area A is a face
  • initial ROI area B is a face
  • initial ROI area C is a human shape, etc.
  • the area type corresponding to each initial ROI area may be the same or different.
  • the area type corresponding to initial ROI area A includes a face area
  • the area type corresponding to initial ROI area B includes a face area
  • the area type corresponding to initial ROI area C includes a human shape area.
  • the area type corresponding to initial ROI area A may be the same as the area type corresponding to initial ROI area B, and different from the area type corresponding to initial ROI area C.
  • the smart door lock can determine whether the initial ROI area includes a face area or a human shape area by the area type corresponding to each initial ROI area.
  • a main initial ROI region may include a sub-initial ROI region.
  • the region type of the initial ROI region C is a human-shaped region.
  • the human-shaped region is likely to include a face region. That is to say, the initial ROI region C serves as the main initial ROI region, and may also include a sub-initial ROI region.
  • the region type of the sub-initial ROI region is a face.
  • the sub-initial ROI region may also correspond to a detection frame. It is understandable that in the subsequent processing of the initial ROI region, the hiding or non-hiding operation may be performed with the region type corresponding to the main initial ROI region.
  • the smart door lock can identify the initial ROI area based on the characteristic parameters corresponding to the target features during the process of identifying the initial ROI area and the corresponding area type. For example, a face has a target face feature and a human figure has a target human figure feature. Both the target face feature and the target human figure feature have corresponding unique characteristic parameters.
  • the smart door lock can extract feature data of each initial ROI area. Perform similarity matching based on the feature parameters corresponding to the target facial features and the target human shape features according to the extracted feature data. For example, if the feature data has the highest similarity after matching the feature parameters of the target facial features. Then, the area type corresponding to the initial ROI area includes the face area. For another example, search in the face database based on the extracted feature data to match data samples similar to the feature data. For example, the face sample with the highest similarity is matched. Then, the area type corresponding to the initial ROI area includes the face area. Furthermore, after the smart door lock identifies each initial ROI area, it generates a corresponding area type. And it can be determined what type of area the initial ROI area includes through the corresponding area type.
  • the region type can also be used to indicate the position and area occupied by the face region or human-shaped region in the initial ROI region.
  • the smart door lock can also mark the position and area occupied by the face region or human-shaped region in the initial ROI region in the detection frame and record them in the local memory, so as to determine which specific regions need to be hidden more accurately based on the initial ROI region and its corresponding region type.
  • CNN convolutional neural networks
  • RNN region proposal networks
  • RCNN regions with CNN features
  • Faster-RCNN fast RCNN networks
  • MobileV2 networks and residual networks etc.
  • the target model provided in the embodiment of the present application may include one or more of the YOLO algorithm, the SSD algorithm and the DenseBox algorithm. These algorithms all have the advantages of fast speed and high precision.
  • the embodiment of the present application does not limit the specific form of the target model.
  • the embodiment of the present application further trains the target model.
  • the target model a large number of first scene images can be used as training samples for training, so that the target model can learn the ability to recognize the initial ROI region including the target feature in the first scene image and the corresponding region type.
  • the original first scene image includes not only the area corresponding to the key information, but also other unnecessary areas.
  • the initial ROI area and the corresponding area type are extracted. Then, it can be obtained what type of area is included in the initial ROI area.
  • the initial ROI area includes a face area or the initial ROI area includes a human shape area. Compared with the method of directly obtaining the face area and the human shape area from the first scene image, the processing complexity of the overall process can be reduced, the time cost can be reduced, and the operation efficiency of the smart door lock can be improved.
  • the target ROI area can be determined based on the initial ROI area.
  • the process of determining the target ROI area based on the initial ROI area includes: based on the first scene image, obtaining pixel data corresponding to the initial ROI area. Afterwards, based on the first depth image and the pixel data, the initial depth data corresponding to the initial ROI area is obtained. And the initial depth data corresponding to the initial ROI area is corrected based on the distance correction parameter to obtain the target depth data corresponding to the initial ROI area. Finally, according to the target depth data corresponding to the initial ROI area, the initial ROI area is screened, and the screened initial ROI area is determined as the target ROI area.
  • the specific implementation method can be found in the following steps.
  • Step S704 The smart door lock obtains pixel data corresponding to the initial ROI area based on the first scene image.
  • Step S705 The smart door lock obtains initial depth data corresponding to the initial ROI area based on the first depth image and pixel data.
  • the smart door lock can establish a correspondence between the first depth image and the first scene image to determine the three-dimensional spatial information of each pixel in the first scene image. It can be understood that the correspondence between the first depth image and the first scene image refers to the mapping relationship formed by mapping the depth data in the first depth image to the first scene image. In an embodiment of the present application, the depth value in the first depth image can be mapped to the first scene image by coordinate conversion.
  • the first scene image is divided into a plurality of pixel regions, and each depth data in the first depth image is defined as a depth value.
  • each pixel region corresponds to a depth value.
  • the depth value corresponding to each pixel region in the first scene image can be determined to generate three-dimensional spatial information of each pixel region.
  • each pixel in each image area in the first scene image is defined as a pixel point
  • each depth data in the first depth image is defined as a depth value.
  • each pixel point corresponds to a depth value.
  • the depth value corresponding to each pixel point in the first scene image can be determined to generate three-dimensional spatial information of each pixel area.
  • the smart door lock can first obtain the pixel data of the above-mentioned identified initial ROI area based on the first scene image. It can be understood that each initial ROI area includes multiple pixels, and each pixel in the initial ROI area is defined as a pixel point. Then the pixel data corresponding to each initial ROI area includes multiple pixel points and corresponding pixel values. Afterwards, the smart door lock obtains the initial depth data corresponding to the initial ROI area based on the correspondence between the first depth image and the first scene image. Since the acquisition position and acquisition time of the first depth image and the first scene image are the same. Therefore, each pixel point in the pixel data corresponds to a depth value in the first depth image. Then, the multiple depth values corresponding to the multiple pixel points constitute the initial depth data corresponding to the initial ROI area.
  • each pixel in each initial ROI region may correspond to a depth value. It should be noted that the embodiment of the present application only takes one pixel corresponding to one depth value as an example, and those skilled in the art may design it according to the actual situation.
  • Step S706 The smart door lock corrects the initial depth data corresponding to the initial ROI area to generate target depth data corresponding to the initial ROI area.
  • the initial depth data corresponding to the initial ROI area includes multiple depth values.
  • the depth values corresponding to the pixels of the left end face and the right end face of the initial ROI area photographed by the second image sensor are also different.
  • the pixels at any position in the initial ROI area are located in the same plane. Therefore, in order to improve the accuracy of the data, the smart door lock needs to calibrate the initial depth data.
  • the target depth data includes the corrected target depth value.
  • the target depth data corresponding to the central area can represent the target depth data of the initial ROI area corresponding to the central area.
  • the central area is the area surrounding the center point of the initial ROI area.
  • the initial depth data corresponding to the central area also includes multiple depth values.
  • the smart door lock can obtain the target depth value by calculating the arithmetic mean of the initial depth data corresponding to the central area.
  • the target depth value is corrected based on the distance correction parameter to complete the correction process.
  • the distance correction parameter is obtained in the above-mentioned joint calibration process of the second image sensor and the first image sensor. Since an initial ROI area corresponds to a corrected target depth value. Therefore, the corresponding corrected target depth values in one or several initial ROI areas can constitute the target depth data.
  • the central area is a 3 ⁇ 3 square area.
  • the square area includes 9 pixels and corresponding 9 depth values.
  • the 9 depth values are: 1.5 meters, 1.6 meters, 1.5 meters, 1.2 meters, 0.8 meters, 0.7 meters, 0.5 meters, 0.6 meters and 0.6.
  • the distance correction parameter includes the sine value of the offset angle ⁇ of the marker point relative to the second image sensor, and ⁇ is 30°.
  • the arithmetic mean of the above 9 depth values is calculated to be equal to 1 meter, that is, the target depth value is 1 meter.
  • the sine value of the offset angle 30° is multiplied by the target depth value of 1 meter to obtain the corrected target depth value of 0.5 meters.
  • a target depth value of 0.5 meters in the vertical direction between the second image sensor and the initial ROI area within the actual space range can be generated.
  • the smart door lock can also divide the initial ROI area into multiple sub-areas and calculate the target depth value of each sub-area respectively.
  • the target depth value of each sub-area can be considered as the depth value between each sub-area and the second image sensor. Then, by calculating the arithmetic mean of the target depth values corresponding to the multiple sub-areas, the target depth value corresponding to the initial ROI area can be obtained. Finally, the target depth value is corrected based on the distance correction parameter to complete the correction process.
  • the target depth value corresponding to the initial ROI area can be calculated using a variety of methods or formulas.
  • the target depth value of each sub-area is calculated by calculating the median of the initial depth data corresponding to all pixels in the area.
  • it can also be obtained by calculating the arithmetic mean of the initial depth data corresponding to all pixels in all sub-areas.
  • it can also be determined by determining the pixel point at the center of the plane in each sub-area and using the initial depth data corresponding to the pixel point as the target depth value of the area.
  • Step S707 The smart door lock screens the initial ROI area based on the target depth data and the corresponding confidence level, and determines the screened initial ROI area as the target ROI area.
  • the smart door lock in order to further improve the data processing speed and data accuracy, can obtain the confidence of the target depth data corresponding to the initial ROI area; and filter the initial ROI area according to the confidence and the effective distance range of the second image sensor; the effective distance range is set according to the hardware conditions of the second image sensor.
  • the purpose is to filter out invalid initial ROI areas.
  • the second image sensor itself usually has a corresponding effective distance range when collecting depth data.
  • the smart door lock can set the proximal threshold and the distal threshold according to the set position of the second image sensor.
  • the effective distance range is determined based on the proximal threshold and the distal threshold. For example, the closest acquisition distance of the second image sensor is 0.02 meters, that is, the proximal threshold is 0.02 meters.
  • the farthest acquisition distance of the second image sensor is 8 meters, that is, the distal threshold is 8 meters.
  • the effective distance range is 0.02 meters-8 meters. So, that is to say, if the depth data collected by the second image sensor is outside the effective distance range, it can be determined that the depth data is invalid depth data.
  • the data processing speed can be improved by filtering the initial ROI area that does not have a spatial position.
  • the confidence level can be used to indicate the reliability of the depth data, and the confidence level is proportional to the intensity of the received reflected light pulse, that is, the greater the intensity of the received reflected light pulse, the greater the confidence level, and the greater the reliability of the depth data.
  • the confidence level is between 0 and 1 (including 0 and 1).
  • the initial ROI area can also be screened by obtaining the confidence of the target depth data corresponding to the initial ROI area to eliminate invalid initial ROI areas.
  • the confidence of the target depth data corresponding to the initial ROI area can be compared with a third preset threshold. If the confidence of the target depth data corresponding to the initial ROI area is greater than or equal to the third preset threshold, the initial ROI area can be determined to be a valid initial ROI area. If the confidence of the target depth data corresponding to the initial ROI area is less than the third preset threshold, the initial ROI area can be determined to be an invalid initial ROI area. Finally, the initial ROI area that has been screened is determined to be the target ROI area.
  • the specific threshold is 20. If the confidence level of the initial ROI region corresponding to the target depth data is greater than or equal to 20, the initial ROI region can be determined to be a valid initial ROI region. If the confidence level of the initial ROI region corresponding to the target depth data is less than 20, the initial ROI region can be determined to be an invalid initial ROI region, and the initial ROI region is discarded.
  • the embodiment of the present application further screens the identified initial ROI area, eliminates the initial ROI area with low confidence and the initial ROI area whose depth data is outside the effective distance range, and obtains the target ROI area. This is to more accurately obtain the area type of the user outside the door and the distance between the user outside the door and the smart door lock.
  • unnecessary processing workload is reduced, and only data processing needs to be performed on the target ROI area. The efficiency of data processing is improved and the power consumption of the smart door lock is reduced.
  • the target depth data corresponding to the target ROI area can be obtained based on the first depth image and the distance correction parameter, wherein the distance correction parameter is obtained according to the calibration results of the first image sensor and the second image sensor, and the distance correction parameter is used to correct the depth data on the first depth image. Furthermore, blurring is performed in the target ROI area according to the target depth data, and the privacy area of the privacy object can be accurately hidden according to the target depth data.
  • Step S708 When the smart door lock determines that the target ROI area meets the second preset condition based on the target depth data corresponding to the target ROI area, the target ROI area is blurred.
  • multiple parallel second preset conditions can be set according to the distance between the user outside the door and the smart door lock, so as to determine whether to hide the privacy information of the user outside the door. It can be understood that the user outside the door whose privacy information needs to be hidden should not be a real visitor to the home, that is, a non-target visitor. At the same time, the non-target visitor is usually a certain distance away from the smart door lock.
  • the second preset condition includes: the target depth corresponding to the target depth data corresponding to the target ROI area The value is within the first preset range; or, the target depth value corresponding to the target depth data corresponding to the target ROI area is within the second preset range, and the confidence of the target depth data corresponding to the target ROI area is greater than the first preset threshold.
  • the second preset condition includes: a target depth value corresponding to the target depth data corresponding to the target ROI area is within a first preset range, and the private information in the target ROI area includes a face.
  • the fuzzy processing task is performed on the target ROI area.
  • the first preset range is greater than 1.5 meters and less than 3 meters.
  • the target ROI area B is blurred.
  • the target ROI area B is blurred. In other words, in order to obtain the image information of the target visitor outside the door and hide the privacy information of the non-target visitor. It can be determined whether the user outside the door is a target visitor or a non-target visitor based on the target depth data corresponding to the target ROI area.
  • the target depth value corresponding to the target depth data corresponding to the target ROI area is within the first preset range, the user represented in the target ROI area can be considered to be a non-target visitor. Therefore, it is necessary to hide the key information of the non-target visitor, such as hiding the face area included in the target ROI area, to avoid infringing the privacy information of the non-target visitor.
  • the smart door lock may further perform blur processing on the target ROI region according to the region information of the target ROI region, wherein the region information may include the region type corresponding to the target ROI region.
  • the second preset condition may further include that the confidence level of the target depth data corresponding to the target ROI area is greater than a second preset threshold.
  • the second preset threshold is 50. Then, when the target depth data corresponding to the target ROI area B is greater than 1.5 meters and less than 3 meters and the corresponding confidence level is greater than 50, and the target ROI area B includes a face area, the target ROI area B is blurred.
  • the second preset condition includes that a target depth value corresponding to the target depth data corresponding to the target ROI area is within a second preset range and the confidence level is greater than a first preset threshold.
  • the fuzzy processing task is performed on the target ROI area.
  • the second preset range is greater than and equal to 3 meters.
  • the target depth data corresponding to the target ROI area C is greater than and equal to 3 meters, and the confidence corresponding to the target depth data is greater than 80, the target ROI area C is fuzzy processed. It can be understood that when the target depth data corresponding to the target ROI area is within the second preset range, it can be characterized that the distance between the user outside the door and the smart door lock is farther, and it is more accurately determined that the target ROI area needs to be fuzzy.
  • the target depth data can also be screened in combination with the confidence.
  • the confidence of the target depth data is greater than the first preset threshold, it can be said that the reliability of the target depth data is higher and the target depth data is valid depth data. Therefore, it is necessary to hide the target ROI area to avoid infringing the privacy information of non-target visitors.
  • the smart door lock when the target depth data corresponding to the target ROI area meets the third preset condition, the smart door lock does not perform the blur processing task on the target ROI area.
  • the third preset condition may include that the target depth value corresponding to the target depth data corresponding to the target ROI area is within a third preset range.
  • the third preset range is less than or equal to 1.5 meters.
  • the ROI area is not blurred.
  • the embodiment of the present application does not specifically limit the second preset condition and the third preset condition.
  • the smart door lock blurs the target ROI area
  • the blurred first scene image is displayed; the first scene image includes the blurred target ROI area.
  • the blurred first scene image is stored locally.
  • the first scene image shown in Figure 10 can be seen. The user can obtain the key information of the target visitor in the first scene image, but cannot obtain the key information of the non-target visitor.
  • the smart door lock can also blur the target ROI area in multiple frames of the first scene image to generate multiple blurred first scene images. And multiple blurred first scene images are superimposed to generate a target video and stored locally.
  • the key information of the target visitor can be obtained in the target video, but the key information of the non-target visitor cannot be obtained. Avoid infringing the privacy of non-target users and improve the security effect of smart door locks.
  • the blurring process may be to change the pixel value of the target ROI region, or to perform a mosaic algorithm on the target ROI region.
  • the pixel value change may be to replace the pixel points in the target ROI region with the same pixel value, or to perform a Gaussian transformation on the pixel values corresponding to the pixel points in the target ROI region to change the corresponding pixel values.
  • the mosaic algorithm can be to divide the target ROI area into multiple pixel blocks, each pixel block contains multiple pixels, and randomly select the pixel value of a pixel in each pixel block, and use the pixel value to replace the pixel values of other pixels in the corresponding pixel block.
  • the above-mentioned blur processing can also use a preset pattern to block the target ROI area.
  • the preset pattern can be a cartoon avatar, an object pattern, an emoticon package, etc.
  • a cartoon avatar is used to block the face area in the target ROI area.
  • an emoticon package is used to block the human-shaped area in the target ROI area.
  • the blurring process may be to blur the entire target ROI region.
  • the entire region in the detection frame corresponding to the target ROI region B and the target ROI region C may be blurred.
  • the privacy area in the target ROI area is blurred according to the target depth data corresponding to the target ROI area; wherein the privacy area is used to characterize the area corresponding to the privacy information, and the privacy area includes the face area or the human shape area corresponding to the privacy object.
  • the local area in the target ROI area can also be blurred.
  • the local area in the detection frame corresponding to the target ROI area can be blurred. For example, only the face area in the target ROI area B is blurred. For another example, only the human shape area in the target ROI area C is blurred.
  • the smart door lock of the embodiment of the present application can jointly calibrate the first image sensor and the second image sensor. After jointly calibrating the first image sensor and the second image sensor, the first scene image and the first depth image are collected in real time, and the initial ROI area including the face area or the human shape area is identified. Afterwards, the initial ROI area is screened to obtain the target ROI area. Finally, if the depth data corresponding to the target ROI area meets the preset conditions, the target ROI area is blurred.
  • the embodiment of the present application can obtain the area type of the target visitor, and can also hide the privacy information corresponding to the non-target visitor. While avoiding infringement of the privacy of non-target visitors, the security effect of the smart door lock is guaranteed, the safety performance of the smart door lock is improved, and the user experience is improved.
  • the image processing method provided in the embodiment of the present application is not limited to the application in the smart door locks in the user's home. It can also be applied to other electronic devices. For example, it is applied to smart door locks installed on car doors and smart surveillance cameras in various places. For another example, it can also be applied to mobile phones. When a user uses a mobile phone to make a video call outdoors, the camera in the mobile phone is likely to capture other users besides the user. For another example, it can also be applied to a laptop computer. When a user uses a laptop computer to make a video conference outdoors, the camera of the laptop computer is also likely to capture other users besides the user. Then, whether the user is making a video call, a video conference, or recording the screen of the above process, the image processing method provided based on the embodiment of the present application can improve privacy security.
  • the smart door lock before blurring the target ROI area, can also identify non-safety elements in the first scene image. If non-safety elements are identified in the first scene image, a safety reminder is directly given to the user. For example, the smart door lock controls the display screen to display reminder information for safety reminder. For another example, the smart door lock can also provide safety reminders to the user by setting an alarm and using voice prompts.
  • the ROI region is not blurred.
  • the smart door lock can also store the unblurred image locally in advance and automatically set the viewing permission of the unblurred image.
  • the viewing permission includes that only specific users can view it.
  • the specific user can be a police officer, etc.
  • the viewing permission involving the privacy information of users outside the door can be effectively managed to ensure information security. It is also possible to hide the privacy information corresponding to non-target visitors on the user side of the smart door lock to improve the user experience.
  • An embodiment of the present application also provides a smart door lock, which includes a memory, one or more processors and a camera; the memory is coupled to the processor; wherein the camera is used to capture scene images and depth images; computer program code is stored in the memory, and the computer program code includes computer instructions.
  • the processor executes the corresponding method provided above.
  • An embodiment of the present application also provides an electronic device, as shown in FIG. 12 , which may include one or more processors 1010 , a memory 1020 , and a communication interface 1030 .
  • the memory 1020 and the communication interface 1030 are coupled to the processor 1010.
  • the memory 1020, the communication interface 1030 and the processor 1010 may be coupled together via a bus 1040.
  • the communication interface 1030 is used for data transmission with other devices.
  • the memory 1020 stores computer program codes.
  • the computer program code includes computer instructions. When the computer instructions are executed by the processor 1010, the electronic device executes the image processing method in the embodiment of the present application.
  • the processor 1010 can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the present disclosure.
  • the processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.
  • the bus 1040 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus.
  • PCI Peripheral Component Interconnect
  • EISA Extended Industry Standard Architecture
  • the bus 1040 may be divided into an address bus, a data bus, a control bus, etc.
  • FIG12 only uses one thick line, but does not mean that there is only one bus or one type of bus.
  • An embodiment of the present application further provides a computer-readable storage medium, in which a computer program code is stored.
  • a computer program code is stored.
  • the electronic device executes the relevant method steps in the method embodiment.
  • the embodiment of the present application also provides a computer program product.
  • the computer program product When the computer program product is run on a computer, it enables the computer to execute the relevant method steps in the above method embodiment.
  • the smart door lock, electronic device, computer storage medium or computer program product provided in this application is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.
  • the disclosed devices and methods can be implemented in other ways.
  • the device embodiments described above are only schematic.
  • the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.
  • Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium.
  • the technical solution of the embodiment of the present application is essentially or the part that makes the contribution or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Studio Devices (AREA)

Abstract

本申请提供一种图像处理方法及电子设备，涉及电子设备领域。本申请提供的方法可以获取第一场景图像和第一场景图像对应的第一深度图像，并确定第一场景图像中的目标感兴趣ROI区域。之后基于第一深度图像获取目标ROI区域对应的目标深度数据。根据目标ROI区域对应的目标深度数据，在第一场景图像上的目标ROI区域内进行模糊处理。从而能够避免侵犯隐私对象的隐私信息，提升了电子设备的安防效果以及用户的使用体验。

Description

一种图像处理方法及电子设备
本申请要求于2022年10月28日提交国家知识产权局、申请号为202211336084.7、申请名称为“一种图像处理方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及电子设备领域，尤其涉及一种图像处理方法及电子设备。
背景技术
随着电子设备的普及，很多场所都会使用智能安防设备。如，智能门锁和智能监控摄像头等。其中，智能门锁能够提供用户多种不同的开锁方式、智能猫眼功能、语音提示以及低电量提醒等。用户还可以通过终端设备与智能门锁进行交互。例如，用户可以通过终端设备中的应用程序(Application，APP)实时查看智能门锁的猫眼画面或者回看智能门锁的猫眼视频，用户还可以通过终端中的应用程序和智能门锁进行通讯。
然而，用户在实时查看猫眼画面或者回看猫眼视频的过程中，智能门锁采集的均为用户家门外的图像和视频。这样，从图像和视频中能够看到门外门口附近的其他用户。同样用户自身在其他用户的家门附近时也会被其他用户的智能门锁采集到。这样，容易侵犯其他用户的隐私，而且也容易泄露用户自身的隐私，导致隐私安全性较差。
发明内容
本申请实施例提供一种图像处理方法及电子设备，通过基于目标深度数据对第一场景图像中隐私对象对应的隐私信息进行模糊处理，避免侵犯隐私对象的隐私信息，提升电子设备的安防效果以及用户的使用体验。
为达到上述目的，本申请的实施例采用如下技术方案：
第一方面，提供了一种图像处理方法，应用于电子设备，该方法包括：获取第一场景图像和第一场景图像对应的第一深度图像。之后，若确定第一场景图像中包括目标感兴趣ROI区域，则基于第一深度图像获取目标ROI区域对应的目标深度数据；目标ROI区域包括隐私对象对应的隐私信息。最后，根据目标ROI区域对应的目标深度数据，在第一场景图像上的目标ROI区域内进行模糊处理。若确定第一场景图像中不包括目标ROI区域，则直接显示第一场景图像。
可见，门外对象与用户的交互可能性与目标深度数据对应的距离相关，比如如果门外对象距离门较远，则门外对象与用户进行交互(比如门外对象来访)的可能性较低，因而可以对该门外对象对应的隐私信息进行模糊处理。再比如，如果门外对象距离门较近，则门外对象与用户进行交互的可能性较高，因而可以不对该门外对象对应的隐私信息进行模糊处理。而且，如果门外对象距离门较近，则门外对象对用户来说安全性较低，存在潜在的安全性问题，因而不需要对其模糊处理。从而使用户可以查看清楚该门外对象。如果门外对象距离门较远，则门外对象对用户来说安全性较高，不存在潜在的安全性问题，因而需要对其模糊处理。从而避免侵犯门外对象的隐私信息。同时也可以对门外对象的隐私信息进行保护，提高电子设备的安全性。
在第一方面的一种可实现方式中，在根据目标ROI区域对应的目标深度数据，在第一场景图像上的目标ROI区域内进行模糊处理的过程中，包括：根据目标ROI区域对应的目标深度数据对目标ROI区域中的隐私区域进行模糊处理；其中，隐私区域用于表征隐私信息对应的区域，隐私区域包括隐私对象对应的人脸区域或人形区域。
这样，在目标ROI区域内进行模糊处理的过程中，也可以仅对目标ROI区域内的隐私区域进行模糊处理。本申请无需将整个目标ROI区域进行模糊处理，可以将隐私对象对应的人脸区域或人形区域进行模糊，更为精确地对隐私对象的隐私信息进行保护，提升电子设备的安防效果。
在第一方面的一种可实现方式中，方法还包括：显示模糊处理后的第一场景图像，第一场景图像上包括模糊处理后的目标ROI区域。可见，本申请可以将模糊处理后的第一场景图像在电子设备中显示出来，并且显示的第一场景图像中针对隐私对象的隐私信息是模糊的。这样，即使所 拍摄的第一场景图像中携带有隐私信息，用户在使用电子设备时也看不到隐私对象的隐私信息。进而，避免侵犯隐私对象的隐私信息，提高了用户的使用体验。
在第一方面的一种可实现方式中，电子设备包括第一图像传感器和第二图像传感器，第一图像传感器用于采集第一场景图像，第二图像传感器用于采集第一深度图像；在获取第一场景图像和场景图像对应的第一深度图像的过程中，包括：在满足第一预设条件时，控制第一图像传感器和第二图像传感器同步采集的第一场景图像和第一深度图像。其中，第一预设条件包括：电子设备检测到隐私对象的声音分贝大于预设分贝，和/或，电子设备检测到隐私对象针对电子设备的触碰操作，和/或，电子设备检测到隐私对象的停留时间大于预设时间。
可见，虽然电子设备可以实时持续采集并处理第一场景图像和第一深度图像，但为了降低电子设备的使用功耗，电子设备可以在满足第一预设条件时，再采集并处理第一场景图像和第一深度图像。例如，电子设备检测到隐私对象的分贝大于预设分贝，和/或，检测到隐私对象针对电子设备的触碰操作，和/或，检测到隐私对象的停留时间大于预设时间等等。这样，在检测到有隐私对象后，执行对应的图像处理。可以有效地降低电子设备的使用功耗，提高用户的使用体验感。
在第一方面的一种可实现方式中，在根据目标ROI区域对应的目标深度数据对目标ROI区域进行模糊处理的过程中，包括：根据目标ROI区域对应的目标深度数据确定目标ROI区域满足第二预设条件时，对目标ROI区域进行模糊处理。其中，第二预设条件包括：目标ROI区域对应的目标深度数据所对应的目标深度值位于第一预设范围内；或，目标ROI区域对应的目标深度数据所对应的目标深度值位于第二预设范围内，且目标ROI区域对应的目标深度数据的置信度大于第一预设阈值。第二预设条件还包括：目标ROI区域中的隐私信息包括人脸。
可见，在根据目标ROI区域对应的目标深度数据对目标ROI区域进行模糊处理的过程中，可以设定多个不同的第二预设条件来对目标ROI区域进行模糊处理。例如，若目标ROI区域对应的目标深度数据所对应的目标深度值位于第一预设范围内且目标ROI区域中的隐私信息包括人脸，则对目标ROI区域进行模糊处理。再例如，若目标ROI区域对应的目标深度数据所对应的目标深度值位于第二预设范围内，且目标ROI区域对应的目标深度数据的置信度大于第一预设阈值。则对目标ROI区域进行模糊处理。这样，通过设定第二预设条件，可以有效地将隐私对象的隐私信息进行模糊，同时还能够获取到目标访客也就是非隐私对象的隐私信息。保证了电子设备的安防效果，提高用户的使用体验。
在第一方面的一种可实现方式中，在基于第一深度图像获取目标ROI区域对应的目标深度数据的过程中，包括：基于第一深度图像和距离修正参数，获取目标ROI区域对应的目标深度数据；距离修正参数根据第一图像传感器和第二图像传感器的标定结果获得，距离修正参数用于修正第一深度图像上的深度数据。
其中，标定的过程包括：控制第一图像传感器和第二图像传感器同步采集第二场景图像和第二深度图像。之后，基于第二场景图像计算得到第一图像传感器对应的标定参数。最后，根据第一图像传感器对应的标定参数确定标定结果，标定结果包括第二图像传感器对应的距离修正参数；距离修正参数用于建立第一场景图像与第一深度图像之间的映射关系。
可见，为了第二图像传感器采集的深度图像中的的深度数据为用户与电子设备之间垂直方向的距离，本申请通过联合标定第二图像传感器和第一图像传感器，确定标定结果，标定结果包括距离修正参数，以便于保证可见光和红外光波图像的匹配，具有精度高、稳定性好的特点。同时，在基于第一深度图像获取目标ROI区域对应的目标深度数据的过程中，可以利用距离修正参数修正第一深度图像上的深度数据，以得到更为准确的目标ROI区域对应的目标深度数据。便于后续根据目标ROI区域对应的目标深度数据对目标ROI区域进行模糊处理。在第一方面的一种可实现方式中，在确定第一场景图像中的目标ROI区域的过程中，包括：识别第一场景图像中包括目标特征的初始ROI区域；其中，目标特征用于表征隐私对象对应的隐私信息，隐私信息包括隐私对象对应的人脸或人形。之后，基于初始ROI区域，确定目标ROI区域。
在基于初始ROI区域，确定目标ROI区域的过程中，包括：基于第一场景图像，获取初始ROI区域对应的像素数据。之后，基于第一深度图像和像素数据，获取初始ROI区域对应的初始深度数据。接着，基于距离修正参数修正初始ROI区域对应的初始深度数据，得到初始ROI区域 对应的目标深度数据。最后，根据初始ROI区域对应的目标深度数据，对初始ROI区域进行筛选，将筛选后的初始ROI区域确定为目标ROI区域。
可见，在确定第一场景图像中的目标ROI区域的过程中，可以预先识别包括目标特征的初始ROI区域，目标特征用于表征隐私对象对应的隐私信息，隐私信息可以包括隐私对象对应的人脸或人形。并且获取并修正初始ROI区域对应的初始深度数据，最后，根据初始ROI区域对应的目标深度数据对初始ROI区域进行筛选，将筛选后的初始ROI区域确定为目标ROI区域。进而，提高数据处理速度以及数据的准确性。在第一方面的一种可实现方式中，在根据初始ROI区域对应的目标深度数据对初始ROI区域进行筛选的过程中，包括：获取初始ROI区域对应的目标深度数据的置信度。之后，根据置信度和第二图像传感器的有效距离范围对初始ROI区域进行筛选；有效距离范围根据第二图像传感器的硬件条件设定。
这样，本申请可以根据目标深度数据以及对应的置信度对初始ROI区域进行筛选。目的是过滤无效ROI区域。通过将不具备空间位置的初始ROI区域进行过滤，提高数据处理速度以及数据的准确性。
第二方面，提供了一种智能门锁，智能门锁包括存储器、一个或多个处理器和摄像头；存储器与处理器耦合；其中，摄像头用于采集第一场景图像和第一深度图像；存储器中存储有计算机程序代码，计算机程序代码包括计算机指令，当计算机指令被处理器执行时，使得处理器执行下述步骤：智能门锁获取第一场景图像和第一场景图像对应的第一深度图像。若确定第一场景图像中包括目标感兴趣ROI区域，则基于第一深度图像获取目标ROI区域对应的目标深度数据；目标ROI区域包括隐私对象对应的隐私信息。之后，智能门锁根据目标ROI区域对应的目标深度数据，在第一场景图像上的目标ROI区域内进行模糊处理。从而能够避免侵犯隐私对象的隐私信息。同时也可以对隐私对象的隐私信息进行保护，提升了电子设备的安防效果以及用户的使用体验。
在第二方面的一种可实现方式中，处理器执行根据目标ROI区域对应的目标深度数据，在第一场景图像上的目标ROI区域内进行模糊处理，包括：智能门锁根据目标ROI区域对应的目标深度数据对目标ROI区域中的隐私区域进行模糊处理；其中，隐私区域用于表征隐私信息对应的区域，隐私区域包括隐私对象对应的人脸区域或人形区域。可见，智能门锁在模糊处理目标ROI区域时，可以仅对目标ROI区域中的隐私区域进行模糊处理，更为精确地对隐私对象的隐私信息进行保护，提高智能门锁的安全性。
在第二方面的一种可实现方式中，处理器还执行下述步骤：显示模糊处理后的第一场景图像，第一场景图像上包括模糊处理后的目标ROI区域。智能门锁可以在显示屏中显示模糊处理后的第一场景图像，第一场景图像上包括模糊处理后的目标ROI区域。这样，即使所拍摄的第一场景图像中携带有隐私信息，用户在使用智能门锁时也看不到隐私对象的隐私信息。避免侵犯隐私对象的隐私信息，提升了电子设备的安防效果以及用户的使用体验。
第三方面，提供了一种电子设备，电子设备包括存储器、一个或多个处理器；存储器与处理器耦合；其中，存储器中存储有计算机程序代码，计算机程序代码包括计算机指令，当计算机指令被处理器执行时，使得电子设备执行如第一方面所述的图像处理方法。
第四方面，提供了一种计算机可读存储介质，计算机可读存储介质中存储有指令，当指令在计算机上运行时，使得计算机可以执行如第一方面所述的图像处理方法。
附图说明
图1为本申请实施例提供的一种图像处理方法的场景示意图；
图2为本申请实施例提供的一种智能门锁的硬件结构示意图；
图3为本申请实施例提供的一种智能门锁的软件结构示意图；
图4为本申请实施例提供的一种标定过程的示意图；
图5为本申请实施例提供的一种第一图像传感器和第二图像传感器标定过程的二维平面示意图；
图6为本申请实施例提供的一种第一图像传感器和第二图像传感器标定过程的三维立体示意图；
图7为本申请实施例提供的一种图像处理方法的流程示意图；
图8为本申请实施例提供的一种第一图像传感器和第二图像传感器标定过程的流程示意图；
图9为本申请实施例提供的一种初始ROI区域以及对应区域类型的示意图；
图10为本申请实施例提供的一种模糊目标ROI区域中局部区域的示意图；
图11为本申请实施例提供的一种模糊全部目标ROI区域的示意图；
图12为本申请实施例提供的一种电子设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述。其中，在本申请的描述中，除非另有说明，“/”表示前后关联的对象是一种“或”的关系，例如，A/B可以表示A或B；本申请中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况，其中A,B可以是单数或者复数。并且，在本申请的描述中，除非另有说明，“多个”是指两个或多于两个。“以下至少一项(个)”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b，或c中的至少一项(个)，可以表示：a，b，c，a-b，a-c，b-c，或a-b-c，其中a，b，c可以是单个，也可以是多个。另外，为了便于清楚描述本申请实施例的技术方案，在本申请的实施例中，采用了“第一”、“第二”等字样对功能和作用基本相同的相同项或相似项进行区分。本领域技术人员可以理解“第一”、“第二”等字样并不对数量和执行次序进行限定，并且“第一”、“第二”等字样也并不限定一定不同。同时，在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念，便于理解。
此外，本申请实施例描述的网络架构以及业务场景是为了更加清楚的说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定，本领域普通技术人员可知，随着网络架构的演变和新业务场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
随着电子设备的普及，很多场所都会使用智能安防设备。如，智能门锁和智能监控摄像头等。其中，智能门锁能够提供用户多种不同开锁方式、智能猫眼功能、语音提示以及低电量提醒等。用户还可以通过终端设备与智能门锁进行交互。例如，用户可以通过终端设备中的应用程序(Application，APP)实时查看智能门锁的猫眼画面或者回看智能门锁的猫眼视频。
然而，参见图1，用户在实时查看猫眼画面或者回看猫眼视频的过程中，智能门锁采集的均为用户家门外的图像和视频。从图像和视频中能够看到门口附近的门外用户。其中，门外用户包括家中目标访客以及除目标访客之外的非目标访客。这样，就会容易侵犯到非目标访客的隐私。同样如果用户自身不是目标访客也会被其他用户的智能门锁采集到。这样，也会泄露用户自身的隐私，导致隐私安全性较差。
并且，为了保护用户的隐私信息，如果关闭智能门锁的拍照功能、摄像功能以及上传图像和视频功能并仅保留语音通过功能，那么则无法获取到关于目标访客对应的关键信息。降低了智能门锁的安防效果，降低用户的使用体验。
基于上述内容，为了能够获取到门外目标访客的关键信息的同时避免侵犯非目标访客的隐私信息，本申请提供了一种图像处理方法，可以应用于电子设备中。
需要说明的是，该电子设备可以是具有摄像头的智能设备，比如可以包括智能猫眼、智能门锁、智能监控摄像头等智能安防设备，还可以包括手机、平板电脑、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、个人数字助理(personal digital assistant，PDA)、可穿戴设备、人工智能(artificial intelligence，AI)设备或其他设备等。
下面以智能门锁为示例进行阐述。智能门锁具有智能猫眼的功能，比如智能门锁上可以安装有智能猫眼。智能门锁可以采集门外的场景图像并进行模糊等处理，还可以在智能门锁配置的显示屏上显示处理后的图像。
在用户实际使用智能门锁过程中，可以根据门外用户与智能门锁之间的距离，决定是否隐藏 门外用户的隐私信息。例如，通过采用智能门锁中的图像传感器(Color sensor)和深度图像传感器(TOF sensor)联合工作，可以获取到更加丰富的关于门外用户的图像信息。其中，第一图像传感器用于采集场景图像，该第一图像传感器例如可以是RGB传感器；第二图像传感器用于采集深度图像，该第二图像传感器例如可以是TOF传感器。为便于后续描述，将图像传感器称为第一图像传感器，以及将深度图像传感器称为第二图像传感器。
之后，基于图像信息得到门外用户与智能门锁之间的距离。当门外用户距离智能门锁满足特定条件时，例如当门外用户距离智能门锁较远时，智能门锁可以将图像信息中关于门外用户的隐私信息进行隐藏。可以理解的是，需要隐藏隐私信息的门外用户为非目标访客。例如，偶然路过家门的非目标访客。这样，通过将其隐私信息隐藏，避免侵犯非目标访客对应的隐私。而当门外用户距离智能门锁满足第二特定条件时，例如当门外用户距离智能门锁较近时，则确定门外用户为目标访客，不需要隐藏对应的隐私信息。因此，智能门锁既可以获取到目标访客的关键信息，如目标访客的人脸信息、目标访客的动作信息以及目标访客的携带物品信息等等。同时还可以对关键信息进行处理，从而隐藏非目标访客的隐私信息。避免侵犯非目标访客隐私的同时保证了智能门锁的安防效果，提高了智能门锁的安全性能，提高用户的使用体验。为便于后续描述，将非目标访客称为隐私对象，以及将目标访客称为非隐私对象。
下面介绍本申请实施例涉及的智能门锁。
图2示出了智能门锁500的硬件结构示意图。
智能门锁500可以包括处理器510，内部存储器520，通用串行总线(universal serial bus，USB)接口530，充电管理模块540，电源管理模块541，电池542，天线，无线通信模块550，音频模块560，扬声器560A，传感器模块570，按键580，指示器581，摄像头582，显示屏583等。其中，传感器模块570可以包括陀螺仪传感器570A，加速度传感器570B，指纹传感器570C，触摸传感器570D、环境光传感器570E、第一图像传感器570F以及第二图像传感器570G等。
可以理解的是，本发明实施例示意的结构并不构成对智能门锁500的具体限定。在本申请另一些实施例中，智能门锁500可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器510可以包括一个或多个处理单元，例如：处理器510可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
其中，控制器可以是电子设备100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器510中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器510中的存储器为高速缓冲存储器。该存储器可以保存处理器510刚用过或循环使用的指令或数据。如果处理器510需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器510的等待时间，因而提高了系统的效率。
在一些实施例中，处理器510可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器510可以包含多组I2C总线。处理器510可以通过I2C接口耦合触摸传感器570D，使处理器510与触摸传感器570D通过I2C总线接口通信，实现智能门锁500的触摸功能。
I2S接口可以用于音频通信。在一些实施例中，处理器510可以包含多组I2S总线。处理器 510可以通过I2S总线与音频模块560耦合，实现处理器510与音频模块560之间的通信。在一些实施例中，音频模块560可以通过I2S接口向无线通信模块550传递音频信号，实现通过智能门锁500输入语音指令的功能。
PCM接口也可以用于音频通信，音频模块560也可以通过PCM接口向无线通信模块550传递音频信号，实现通过智能门锁500输入语音指令的功能。
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器510与无线通信模块550。例如：处理器510通过UART接口与无线通信模块550中的蓝牙模块通信，实现发送或扫描蓝牙广播等蓝牙功能。
MIPI接口可以被用于连接处理器510与显示屏583，摄像头582等外围器件。在一些实施例中，处理器510和摄像头582通过CSI接口通信，实现智能门锁500的拍摄功能。处理器510和显示屏583通过DSI接口通信，实现智能门锁500的显示功能。
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器510与摄像头582，显示屏583，无线通信模块550，音频模块560，传感器模块570等。
USB接口530是符合USB标准规范的接口，具体可以是Mini USB接口，Micro USB接口，USB Type C接口等。USB接口530可以用于连接充电器为智能门锁500充电，也可以用于智能门锁500与外围设备之间传输数据。
充电管理模块540用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块540可以通过USB接口530接收有线充电器的充电输入。在一些无线充电的实施例中，充电管理模块540可以通过智能门锁500的无线充电线圈接收无线充电输入。充电管理模块540为电池542充电的同时，还可以通过电源管理模块541为智能门锁500供电。
电源管理模块541用于连接电池542，充电管理模块540与处理器510。电源管理模块541接收电池542和/或充电管理模块540的输入，为处理器510，内部存储器520，外部存储器，显示屏583，摄像头582，和无线通信模块550等供电。电源管理模块541还可以用于监测电池容量，电池循环次数，电池健康状态(漏电，阻抗)等参数。
智能门锁500的无线通信功能可以通过天线和无线通信模块550，调制解调处理器以及基带处理器等实现。
天线用于发射和接收电磁波信号。智能门锁500中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。
无线通信模块550可以提供应用在智能门锁500上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络))以及蓝牙(bluetooth，BT)等无线通信的解决方案。无线通信模块550可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块550经由天线接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器510。无线通信模块550还可以从处理器510接收待发送的信号，对其进行调频，放大，经天线转为电磁波辐射出去。
智能门锁500通过GPU，显示屏583，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏583和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器510可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏583用于显示图像，视频等。显示屏583包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，智能门锁500可以包括1个或N个显示屏583，N为大于1的正整数。
智能门锁500可以通过ISP，摄像头582，视频编解码器，GPU，显示屏583以及应用处理器 等实现拍摄功能。
ISP用于处理摄像头582反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头582中。
摄像头582用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，智能门锁500可以包括1个或N个摄像头582，N为大于1的正整数。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现智能门锁500的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
内部存储器520可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器510通过运行存储在内部存储器520的指令，从而执行智能门锁500的各种功能应用以及数据处理。内部存储器520可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储智能门锁500使用过程中所创建的数据(比如密码数据、音频数据等)等。此外，内部存储器520可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
智能门锁500可以通过音频模块560和扬声器560A以及应用处理器等实现音频功能。扬声器560A，也称“喇叭”，用于将音频电信号转换为声音信号。智能门锁500可以通过扬声器560A收听提示音。
陀螺仪传感器570A可以用于确定智能门锁500的运动姿态。在一些实施例中，可以通过陀螺仪传感器570A确定智能门锁500是否处于移动状态。
距离传感器570B，用于测量距离。智能门锁500可以通过红外或激光测量距离。在一些实施例中，拍摄场景，智能门锁500可以利用距离传感器570B测距以实现快速对焦。
环境光传感器570E用于感知环境光亮度。智能门锁500可以根据感知的环境光亮度自适应调节显示屏583亮度。环境光传感器570E也可用于拍照时自动调节白平衡。
指纹传感器570C用于采集指纹。智能门锁500可以利用采集的指纹特性实现指纹解锁。
触摸传感器570D，也称“触控面板”。触摸传感器570D可以设置于显示屏583，由触摸传感器570D与显示屏583组成触摸屏，也称“触控屏”。触摸传感器570D用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏583提供与触摸操作相关的视觉输出。
第一图像传感器570F用于根据接收到的反射光中的可见光信号转换为可见光信息，该可见光信息是以电信号的形式存在，进而基于可见光信息生成可见光图像。可见光图像也可以称为场景图像。在本申请实施例中，第一图像传感器570F可以用于采集门外场景中的场景图像。
第二图像传感器570G通常是通过飞行时间法来测量该被测物体(或被测目标)的深度数据，具体地，飞行时间法(Time Of Flight，TOF)通过测量测量仪器主动发出的脉冲信号从发射到接收的时间间隔T(常被称为脉冲测距法)或激光往返被测物体一次所产生的相位差(相位差测距法)，以换算成被拍摄景物的距离，用于产生深度数据来实现对被测物体(或被测物体检测区域)的三维结构或三维轮廓的测量，进而获得该被测物体的灰度图像和深度数据。同理，第二图像传感器将接收到的反射光中的红外光信号转换为深度信息，该深度信息是以电信号的形式存在，进而基于深度信息生成灰度图像和深度数据。在本申请实施例中，第二图像传感器570G可以用于采集门外用户与智能门锁之间的深度数据。
在另一些实施例中，上述第一图像传感器570F和第二图像传感器570G还可以设置于一个或 几个摄像头582中。本申请实施例不对部件布置进行具体限定。
按键580包括开机键，音量键等。按键580可以是机械按键。也可以是触摸式按键。智能门锁500可以接收按键输入，产生与智能门锁500的用户设置以及功能控制有关的键信号输入。
指示器581可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。
图3示出了智能门锁500的软件结构示意图。
智能门锁500的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本申请实施例以分层架构的系统为例，示例性说明智能门锁500的软件结构。
分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将系统分为三层，从上至下分别为应用程序层，硬件抽象层以及内核层。
应用程序层可以包括一系列应用程序包。
如图3所示，应用程序包可以包括第一应用、第二应用以及第三应用等应用程序。其中，第一应用用于进行徘徊逗留检测。第一应用可以检测是否有目标在指定区域内滞留超过一定的时间。第二应用用于与访客进行通话。第三应用用于对指定区域进行智能抓拍。
硬件抽象层(Hardware Abstraction Layer，HAL)位于应用程序层与内核层之间，用于衔接应用程序层与内核层，从而在硬件变更的情况下使得上层业务软件对于硬件变更基本无感知。其中，硬件抽象层可以实现应用程序框架层的功能，例如，为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。因此，硬件抽象层可以包括一些预先定义的函数。
在一些实施例中，硬件抽象层还包括多个模块，每个模块与一类硬件相对应，如视频输入(Video Input，VI)模块、视频处理子系统(Video Process sub-system，VPSS)模块、视频编码(Video Encoder，VENC)模块、感兴趣(region of interest，ROI)区域识别模块、ROI区域处理模块、标定模块以及视频输出(Video Output，VO)模块等等。视频处理子系统模块可以分别与视频输入模块、视频输出模块以及视频编码模块等电连接。
其中，标定模块用于标定第一图像传感器和第二图像传感器。视频输入模块用于接收多路视频信号传输给视频处理子系统模块。视频处理子系统模块用于接收解码后的数据并进行视频处理，将处理后的视频传输至ROI区域识别模块。在一些实施例中，VPSS模块还可以用于实现视频处理功能。例如，支持视频裁剪，设置边框大小和颜色配置，分块处理，视频遮挡，视频叠加以及视频数据压缩等视频处理功能。
ROI区域识别模块用于识别视频中的ROI区域。ROI区域处理模块用于对视频中的ROI区域进行处理，并将处理后的视频传输至视频编码模块。在一些实施例中，ROI区域处理模块可以对识别得到的ROI区域进行筛选，剔除无效ROI区域。还可以对识别得到的ROI区域进行区域划分，得到ROI前景区域和ROI背景区域等。
视频编码模块用于接收到ROI区域处理模块发送处理后的视频，对该视频进行编码处理。在本申请实施例中，视频编码模块可以用于对视频中的ROI区域进行模糊处理。视频输出模块用于将模糊处理后的视频进行输出。
内核层是硬件和软件之间的层。内核层至少包含第一图像传感器驱动和第二图像传感器驱动等等。
需要说明的是，在一些实施例中，电子设备可以包括摄像头、处理器和显示屏，摄像头用于采集场景图像，处理器用于处理摄像头采集的场景图像如进行模块处理等，显示屏用于显示处理器处理完成的场景图像。比如，该电子设备可以是智能猫眼、智能门锁或具有智能猫眼功能的智能门锁。其中，摄像头、处理器和显示屏可以设置在一起，也可以分开设置。比如，当电子设备为智能门锁时，摄像头可以位于门的外侧，显示屏位于门的内侧。
在另一些实施例中，电子设备也可以只包括摄像头和处理器，而不包括显示屏。处理器可以对摄像头采集的场景图像进行模糊等处理。电子设备可以把处理后的场景图像发送给其他设备进行显示。比如电子设备可以是不设有显示屏的智能门锁，智能门锁可以将处理后的场景图像发送 给手机进行显示。
在一些实施例中，在同一个摄像头中可以设置多个图像传感器。例如，单镜头双传感器的相机，将第一图像传感器和第二图像传感器均设置在一个相机中。再例如，双镜头双传感器的相机以及单镜头三传感器的相机，这些传感器用于对相同的被拍摄物体成像。当镜头数目少于传感器数目时，在镜头和传感器之间还可以设置分光器，以便把一个镜头进入的光线分解到多个传感器上，确保每个传感器都能接收到光线。同时，在这些摄像头中，处理器的数量可以是一个或多个。多个图像传感器相配合可以采集到被拍摄物体的图像并获取得到被拍摄物体的距离。
例如，在本申请实施例中，智能门锁500中设置有第一图像传感器和第二图像传感器，第二图像传感器和第一图像传感器可以分别设置在各自的深度图像摄像头和图像摄像头中，也可以设置在一个摄像头中。深度图像摄像头也可以称为深度图像相机，图像摄像头也可以称为图像相机。在设置第一图像传感器和第二图像传感器的过程中，可以将第二图像传感器和第一图像传感器的光敏面可以平行设置。由于第二图像传感器采集的为红外光波图像，第一图像传感器采集的为可见光图像。所以，将两者平行设置可以保证可见光和红外光波图像的匹配，以便于获得更加丰富的图像信息。
再例如，在本申请实施例中，智能门锁500中还可以设置有其他类型的图像传感器和距离传感器。其他类型的图像传感器还可以包括色标传感器。其他类型的距离传感器可以包括激光距离传感器、超声波传感器以及红外测距传感器等。可以理解的是，本申请实施例中还可以将其他类型的图像传感器和其他类型的距离传感器设置在智能门锁中，以便于获得更加丰富的图像信息。在一些实施例中，在使用第一图像传感器和第二图像传感器出厂之前，通常会进行相机标定，以使得第一图像传感器采集的图像信息和第二图像传感器采集的深度信息更加精确。
需要说明的是，标定是指确定空间物体表面某点的三维几何位置与第一图像传感器和第二图像传感器采集的图像中对应点之间相互关系的过程。进而，在标定过程中需要建立成像几何模型，这些几何模型参数可以被概括为标定参数。其中，标定参数包括内参、外参以及畸变参数等。在大多数情况下，标定参数可以通过实验和计算才能够求解得到，通过实验和计算求解标定参数的过程被即为标定过程。标定过程是机器视觉/计算机视觉等领域的重要环节，因此，做好标定过程是做好后续工作的前提，快速准确地完成标定过程则是提高后续工作效率的基础。
本申请实施例智能门锁通过确定世界坐标系下三维空间点与像素平面像素点间的转换关系以及在成像过程中的畸变系数，完成第一图像传感器和第二图像传感器的标定过程，以便于后续进行图像处理。并且智能门锁可以将上述第一图像传感器和第二图像传感器采集到的图像数据及深度数据进行配准融合，得到信息丰富的图像。
如图4所示，标定过程中会涉及到的基本原理为：现实空间中的物体是三维的，而其对应的图像中的物体是二维的。因此，从三维场景中的物体到其对应的二维图像之间可以认为存在一个三维到二维的几何模型，该几何模型使得三维场景中的物体到其对应的图像之间形成从三维到二维或者二维到三维的转换。那么，容易理解的是，在第一图像传感器和第二图像传感器对三维场景中的物体进行图像采集时，第一图像传感器和第二图像传感器就可以被认为是这个几何模型，而标定参数就是这个几何模型的参数。因此，只要求得标定参数，就可以由物体在图像中的像素坐标反推出物体在空间中的世界坐标，由此实现视觉检测、生物特征识别、距离测量、三维重建等功能。
本申请实施例提供一种图像处理方法，可以应用于电子设备，方法包括：首先，标定电子设备中的第一图像传感器和第二图像传感器，得到标定结果。之后，同步获取第一图像传感器采集的第一场景图像，以及第二传感器采集的第一深度图像。在确定第一场景图像中包括目标感兴趣ROI区域的情况下，基于第一深度图像获取目标ROI区域对应的目标深度数据。其中，目标ROI区域包括隐私对象对应的隐私信息；之后，根据目标ROI区域对应的目标深度数据，在第一场景图像上的目标ROI区域内进行模糊处理。这样，通过根据目标深度数据在目标ROI区域内进行模糊处理。可以隐藏第一场景图像中隐私对象对应的隐私信息，避免侵犯隐私对象的隐私信息，同时也可以对隐私对象的隐私信息进行保护。
下面将以智能门锁为例，对本申请实施例提供的图像处理方法进行具体说明。图7为本申请 实施例示出的一种图像处理方法的流程示意图，如图7所示，该方法可以包括如下步骤S701-S708。
步骤S701、智能门锁标定第一图像传感器和第二图像传感器。
在一些实施例中，通常第一图像传感器和第二图像传感器安装于同一平面，安装位置不能重叠。因此，第一图像传感器和第二图像传感器会根据自身可视角度形成两个圆锥体视场，两个视场在平面投影参见图6。需要说明的是，第一图像传感器的接收镜头视场角可以大于第二图像传感器的接收镜头的视场角。也就是说，第一图像传感器视场β可以覆盖第二图像传感器视场γ。目的是可以在可见光的视场范围内截取出红外光波的视场，使第一图像传感器采集的可见光图像和第二图像传感器采集红外光波图像中的深度数据能够一一对应，便于后续图像处理。
在实际场景中，参见图5，图5为本申请实施例示出的用户在门外的俯视图。可以理解的是，位置1即为用户站在门外，且距离智能门锁有一定距离的位置。示例性的，用户在位置1时，第二图像传感器采集深度图像并解析，可以得到智能门锁与用户之间的深度值L2为3米。同时，第一图像传感器采集场景图像并解析，可以得到场景图像中智能门锁与用户之间的距离为L1。可以看出L1等于L2也为3米。也就是说，只要第一图像传感器采集场景图像为用户所在平面的图像。无论用户向左平移或者向右平移，解析场景图像中智能门锁与用户之间的距离均为空间内用户与智能门锁之间垂直方向的距离，即3米。
然而，当用户向左平移到位置2时，此时，解析第二图像传感器采集的深度图像，得到智能门锁与用户之间的距离为L3。可以看出，L3大于L2，且应等于L2/sin(ɑ)。但实际上，用户即使向左平移到位置2时，智能门锁与用户之间的距离也应该为3米。因此，为了使第二图像传感器采集的深度数据为用户与智能门锁之间垂直方向的距离。本申请实施例可以通过联合标定第二图像传感器和第一图像传感器，以便于保证可见光和红外光波图像的匹配，具有精度高、稳定性好的特点。
由此，本申请实施例提供的标定过程可以包括：首先，控制第一图像传感器和第二图像传感器同步采集第二场景图像和第二深度图像。之后，基于第二场景图像计算得到第一图像传感器对应的标定参数。最后，根据第一图像传感器对应的标定参数确定标定结果，标定结果包括第二图像传感器对应的距离修正参数；其中，距离修正参数用于建立第一图像传感器采集的第一场景图像与第二图像传感器采集的第一深度图像之间的映射关系。
下面结合图8对本申请实施例提供的标定第一图像传感器和第二图像传感器的过程进行具体详细描述。参见图8，标定第一图像传感器和第二图像传感器的过程包括如下步骤：
步骤S801、智能门锁控制第一图像传感器和第二图像传感器同步采集第二场景图像和第二深度图像。
其中，第二场景图像可以包括红绿蓝(Red Green Blue，RGB)图像。第二场景图像和第二深度图像为同一平面上沿同一拍摄方向同步采集的同一幅图像。第二深度图像包括第二场景图像上各个像素点对应的拍摄对象到上述平面的深度数据。
在一种可实现方式中，通常在第一图像传感器和第二图像传感器共同的视场区域内配置标定板。配置标定板的目的是为了使得第一图像传感器和第二图像传感器能够同时对标定板进行拍摄，进而获得标定板对应的第二场景图像和第二深度图像。以便于后续对第一图像传感器和第二图像传感器进行标定。
在配置标定板的过程中，可以对标定板进行无约束的放置。例如，任意改变标定板的位置。例如，将标定板沿同一平面左右平移，并在每次改变标定板的位置后，可以重复采集第二场景图像和第二深度图像。应当理解的是，后续第一图像传感器和第二图像传感器的标定参数是需基于第二场景图像和第二深度图像计算的。因此，在本申请实施例中可以同时对不同位置的标定板进行多组采集。这样，可以基于多组第二场景图像和第二深度图像标定第一图像传感器和第二图像传感器，提高标定精度。
需要说明的是，标定板上可以设置预设图案，如印制棋盘格图案、圆点图案、二维码图案或者其他特定图案。对于标定板来说，其中的任意一个图案单元。例如，棋盘格图案的任意一个方格、圆点图案的任意一个圆点或者位于中心的任意形状区域，均可以称作一个标志点，以用于后续计算标定参数。
在一些可实现方式中，第一图像传感器和第二图像传感器同步采集第二场景图像和第二深度图像中，需要保持同一幅图像且尺寸大小相等。也就是说，第二场景图像和第二深度图像的分辨率(即像素)要保持相同。通常情况下，如果第二图像传感器采集的第二深度图像分辨率大于第一图像传感器采集的第二场景图像的分辨率，那么，可以在进行第一图像传感器和第二图像传感器标定之前，可以预先对第一图像传感器采集的第二场景图像进行降采样等处理方式，以获得与第二深度图像分辨率相同的第二场景图像。
步骤S802、智能门锁基于第二场景图像计算得到第一图像传感器对应的标定参数。
步骤S803、智能门锁基于第一图像传感器对应的标定参数确定标定结果，标定结果包括第二图像传感器对应的距离修正参数。
在本申请实施例中，选取每组第二场景图像中位于中心的圆点为标志点。进而，通过采集的多组第二场景图像中的标志点以及各个坐标系之间的转换矩阵，可以计算得到第一图像传感器对应的标定参数。其中，标定参数可以包括第一图像传感器对应的内参矩阵、第一图像传感器对应的外参矩阵以及第一图像传感器对应的畸变系数等。需要说明的是，本申请实施例对标志点和计算标定参数的方法不作具体的限定，本领域技术人员可以根据实际需求自行选择标志点以及计算方法，这些设计都没有超出本申请实施例的保护范围。
进一步地，本申请实施例可以根据第一图像传感器对应的标定参数确定第二图像传感器对应的距离修正参数。示例性的，当采集到的第二场景图像存在畸变时，如该第二场景图像中的标志点相对于标定板图像中的标志点向左平移了20个像素的距离。同时，第二图像传感器由于自身结构原因，在采集对应的深度图像后，该深度图像中标志点对应的深度值也会与标定板图像中标志点对应的深度值不同。因此，可以根据第二场景图像对应的标定参数对第二图像传感器进行标定，以完成第一图像传感器与第二图像传感器之间的联合标定。
为了便于理解，参见图6，可以将采集到的第二场景图像中的标志点定义为图中的用户。当第一图像传感器采集的第二场景图像中用户沿第一方向移动20个像素的距离后，解析第二场景图像。可以得出无论用户处于未移动状态或移动后状态，在空间范围内与第一图像传感器之间的距离是不变的，均为垂直方向距离即L4。
由于第二图像传感器的测距原理是确定第二图像传感器与物体或周围环境之间的距离，即通过第二图像传感器测量发射光到达物体并反射回第二图像传感器所用的时间，并将其转换为距离。因此，解析第二图像传感器采集的深度图像后，可以得出用户与第二图像传感器之间的距离为L6。而在实际空间范围内第二图像传感器与用户之间的距离应为L5。而L5也应垂直方向的距离，且等于即L4。
在本申请实施例中，可以基于第二场景图像中标志点的坐标和第一图像传感器对应的标定参数，计算得到距离修正参数，以便于将用户处于移动后状态与第二图像传感器之间的距离L6修正为用户处于未移动状态与第二图像传感器之间的距离L5。在一种可实现方式中，距离修正参数可以包括标志点相对于第二图像传感器偏移角度ɑ的正弦值。
步骤S804、智能门锁根据距离修正参数校正第二深度图像中的深度数据，以完成第一图像传感器和第二图像传感器的标定过程。
在本申请实施例中，可以根据距离修正参数校正第二图像传感器输出的第二深度图像对应的深度数据，进而完成第一图像传感器和第二图像传感器的联合标定。继续参见图5，例如，标志点相对于第二图像传感器偏移的角度ɑ为30°，第二图像传感器与用户之间垂直方向的深度值为0.5米，第二图像传感器输出的深度数据即深度值为1米。那么，将偏移角度30°的正弦值与深度值1米相乘，即可得到校正后的深度值0.5米。这样，校正后第二图像传感器输出的深度数据与在实际空间范围内第二图像传感器与用户之间垂直方向的深度值相同，完成标定过程。
可见，通过上述标定过程，本申请实施例可以将第二深度图像中的深度值映射至第二场景图像中，进而，建立可见光图像到红外光波图像之间的映射关系。并为后续第二图像传感器采集的深度数据(如第一深度图像)和第一图像传感器采集的图像数据(如第一场景图像)建立相互之间的对应关系。在第一图像传感器采集第二场景图像后，即可同步获取同一视场内第二场景图像对应的深度数据。同时，通过采用第一图像传感器和第二图像传感器的联合标定，能够实现快速 精确配准场景图像数据与深度数据，提高标定精确度以及简化标定流程。
可以理解的是，通常第一图像传感器和第二图像传感器在标定过程中或在实际使用过程中采集的图像是不同的。示例性的，第一图像传感器和第二图像传感器在标定过程中采集的为包括标定板的图像，而第一图像传感器和第二图像传感器在实际使用过程中采集的为门外的图像。因此，为便于描述，在标定过程中第一图像传感器和第二图像传感器采集的为第二场景图像和第二深度图像。在后续实际使用过程中，第一图像传感器和第二图像传感器采集的为第一场景图像和第一深度图像。
步骤S702、若智能门锁满足第一预设条件，则控制第一图像传感器和第二图像传感器同步采集第一场景图像和第一深度图像。
在实际应用场景中，智能门锁可以一直持续采集用户家门外的场景图像。但当门外不存在用户时，智能门锁无需对采集图像进行关键信息处理并隐藏隐私信息。
为了确定门外是否存在用户，本申请实施例可以通过设置第一预设条件，若智能门锁满足第一预设条件，即可确定门外存在用户。之后控制第一图像传感器和第二图像传感器同步采集第一场景图像和第一深度图像，并基于采集第一场景图像和第一深度图像进行关键信息处理，以隐藏隐私信息。
在一种可实现方式中，第一预设条件包括：智能门锁检测到隐私对象的声音分贝大于预设分贝，和/或，智能门锁检测到隐私对象针对智能门锁的触碰操作，和/或，智能门锁检测到隐私对象的停留时间大于预设时间。
例如，门外用户声音超过预设分贝等。再例如，门外用户针对智能门锁的触碰操作。这样，当智能门锁检测到门外用户针对智能门锁的触碰操作，再或者检测到门外用户声音超过预设分贝时，可以确定门外存在用户，即满足第一预设条件。
或者，当门外出现的用户为短暂路过的情况下，如该门外用户在下楼梯的过程中短暂路过用户家门。此时，智能门锁也无需隐藏图像中的隐私信息。
再例如，门外用户在智能门锁前停留大于或等于3秒。这样，当智能门锁检测到门外用户在智能门锁前停留大于或等于3秒时，可以确定门外用户并非短暂路过用户家门，即满足第一预设条件。
在一些实施例中，智能门锁中可以设置有距离传感器和单片机。距离传感器可以用于测量距离，单片机用于计时。智能门锁可以通过距离传感器中的红外或激光测量距离。例如，当距离传感器检测到门外用户后，距离传感器持续发送第一信号至单片机，第一信号用于触发单片机进行计时。单片机接收到第一信号开始计时。若单片机计时超过预设阈值后，仍能持续接收到距离传感器发送的第一信号，即可确定门外用户在门外停留。
由此，智能门锁在满足第一预设条件后，可以采集图像并对图像进行关键信息处理，以便于隐藏隐私信息。其中，关键信息如门外用户的人脸信息、动作信息以及携带物品信息等等。
在一种实现方式中，为了减少智能门锁的功耗，在一直持续采集场景图像的过程中，智能门锁可以采用功耗较低的摄像头来采集图像，如使用除第一图像传感器和第二图像传感器之外清晰度较差的摄像头等。而在智能门锁满足第一预设条件后，再控制第一图像传感器和第二图像传感器采集第一场景图像和第一深度图像，以实现后续对图像中隐私信息的隐藏。当然，智能门锁还可以按照预设间隔周期性采集用户家门外的场景图像，以降低功耗。
在另一种实现方式中，智能门锁可以一直利用第一图像传感器和第二图像传感器采集图像。当智能门锁满足第一预设条件后，需要将采集的第一场景图像和第二深度图像进行关键信息处理，比如隐藏隐私信息。而在智能门锁未满足第一预设条件时，无需对采集的第一场景图像和第二深度图像进行关键信息处理。
可见，智能门锁通过设置第一预设条件，并在满足第一预设条件后采集第一场景图像和第二深度图像，以执行后续隐藏门外用户的隐私信息的操作/任务。可以对门外用户的关键信息进行处理，以达到保护用户隐私信息的目的。而在智能门锁未满足第一预设条件时，则无需采集第一场景图像和第二深度图像，并执行后续处理。这样，既可以提高门外存在用户的场景识别度，同时还可以降低智能门锁的使用功耗。
步骤S703、智能门锁识别第一场景图像中包括目标特征的初始感兴趣区域(region of interest，ROI)。
其中，包括目标特征的初始ROI区域表示门外用户对应隐私信息的区域。例如，目标特征包括人脸特征和人形特征等。
在智能门锁检测满足第一预设条件后，得到第一场景图像和第一深度图像。以及基于第一场景图像识别包括目标特征的初始ROI区域，初始ROI区域表示门外用户对应隐私信息的区域。由此，通过识别和处理包括目标特征的初始ROI区域，可以避免泄露门外用户的隐私信息的同时保护用户自身的隐私信息。同时，无需对第一场景图像中除初始ROI区域之外的其他部分进行处理，提高处理效率。
在本申请实施例中，可以将第一场景图像输入至目标模型中进行检测，得到包含人脸特征或者人形特征等目标特征的检测框以及对应的区域类型。其中，区域类型用于表征初始ROI区域中包括的是人脸区域或人形区域。之后，将检测框中的区域确定为初始ROI区域。为了考虑第一场景图像中的人体和/或物体拥有不同的几何特征，且由于透视关系，当人体和/或物体距离智能门锁越远时在图像中的尺寸越小。因此，如果赋予所有初始ROI区域为统一大小的矩形区域容易造成兴趣区域的信息缺失或冗余。由此，本申请中的检测框可以为适当大小的矩形，且初始ROI区域可以为尺寸不同的矩形区域。
可以理解的是，初始ROI区域指的是从图像中选择的一个图像区域，该区域是图像分析所关注的重点。本申请实施例通过圈定包括目标特征的区域作为对图像进一步处理的前提，可以减少图像处理时间，提高图像处理精度。需要说明的是，初始ROI区域可以是矩形区域、圆形区域或任意形状区域，在本申请实施例中可以为矩形区域，该初始ROI区域中包括目标特征指的是该初始ROI区域包括人脸区域和人形区域。
下面结合图9，对本申请实施例中的初始ROI区域以及对应的区域类型进行详细阐述。
在本申请实施例中，智能门锁基于采集的第一场景图像，可以对第一场景图像中包含人脸特征或者人形特征的初始ROI区域以及对应的区域类型进行识别。其中，区域类型用于表征初始ROI区域中包括的是人脸区域或人形区域。人脸区域用于表征门外用户对应的人脸信息。人形区域用于表征门外用户的外形轮廓信息。可以理解的是，本申请识别的包含目标特征的初始ROI区域均可以表征门外用户对应的隐私信息。以便于后续对初始ROI区域执行隐藏或不隐藏门外用户的隐私信息的操作/任务。
在另一些实施例中，门外用户对应的隐私信息包括但不局限于上述初始ROI区域中的人脸区域和人形区域。同样的，在识别初始ROI区域的过程中包括但不限于上述人脸特征和人形特征等目标特征。目标特征还可以包括用户携带的特定物品、特定物品的标识以及携带的宠物对应的特征等。例如，特定物品可以为工牌。再例如，特定物品还可以为背包，特定物品的标识可以为背包对应的品牌等。再例如，特定物品还可以为使用的智能设备。由此，上述与隐私信息对应的目标特征均可以作为识别的对象。
参见图9，可以看到，第一场景图像中可以包括一个或几个初始ROI区域。每个初始ROI区域均对应有区域类型。例如，初始ROI区域A人脸、初始ROI区域B人脸以及初始ROI区域C人形等。其中，每个初始ROI区域对应的区域类型可以相同也可以不同。再例如，初始ROI区域A对应的区域类型包括人脸区域、初始ROI区域B对应的区域类型包括人脸区域以及初始ROI区域C对应的区域类型包括人形区域。那么，初始ROI区域A对应的区域类型可以与初始ROI区域B对应的区域类型相同，并且与初始ROI区域C对应的区域类型不同。智能门锁可以通过每个初始ROI区域对应的区域类型确定初始ROI区域中包括人脸区域或人形区域。
在一些实施例中，不同的初始ROI区域之间还可以存在包含关系。示例性的，一个主初始ROI区域可以包含有一个子初始ROI区域。例如，初始ROI区域C的区域类型为人形区域。并且人形区域中有很大可能包括一个人脸区域。也就是说，初始ROI区域C作为主初始ROI区域，且还可以包含一个子初始ROI区域。该子初始ROI区域的区域类型为人脸。那么，子初始ROI区域也可以对应有一个检测框。可以理解的是，在后续对初始ROI区域处理过程中，可以以主初始ROI区域对应的区域类型执行隐藏或不隐藏的操作。
在一些实施例中，智能门锁在识别初始ROI区域以及对应的区域类型的过程中，可以基于目标特征对应的特征参数对初始ROI区域进行识别。例如，人脸对应有目标人脸特征以及人形对应有目标人形特征。目标人脸特征和目标人形特征均对应有特有的特征参数。
示例性的，智能门锁可以提取每个初始ROI区域的特征数据。根据提取的特征数据与目标人脸特征和目标人形特征对应的特征参数进行相似度匹配。例如，若特征数据与目标人脸特征的特征参数匹配后的相似度最高。那么，该初始ROI区域对应的区域类型中即包括人脸区域。再例如，根据提取的特征数据在人脸数据库中进行检索，以匹配出的与特征数据特征相似的数据样本。例如，匹配出相似度最高的为人脸样本。那么，该初始ROI区域对应的区域类型中即包括人脸区域。进而，智能门锁对每个初始ROI区域进行识别后，生成对应的区域类型。并且可以通过对应区域类型确定初始ROI区域包括哪种类型区域。
在另一些实施例中，区域类型还可以用于表示人脸区域或人形区域在初始ROI区域中的所占位置和所占面积。并且智能门锁还可以将其在初始ROI区域中的所占位置和所占面积同步标记显示在检测框内并且记录在本地存储器中，以便于后续基于初始ROI区域及其对应的区域类型更精确地确定哪些具体区域需要隐藏。
在一些实施例中，对于初始ROI区域的提取，可以采用卷积神经网络(Convolutional Neural Networks，CNN)、区域选取网络(Region Proposal Network，RPN)、基于卷积功能的区域选取网络(Regions with CNN features，RCNN)、快速RCNN网络(Faster-RCNN)、MobileV2网络以及残差网络等，或多种网络的结合来进行ROI区域以及对应区域类型的提取。需要说明的是，本申请实施例提供的目标模型可以包括YOLO算法、SSD算法以及DenseBox算法中一种或几种。这些算法均具有速度快、精度高的优点。本申请实施例不对目标模型的具体形态进行限定。
在一种实现方式中，本申请实施例还会对目标模型进行训练。在训练目标模型时，可以将大量的第一场景图像作为训练样本进行训练。以使得目标模型能学习到识别第一场景图像中包括目标特征的初始ROI区域以及对应区域类型的能力。
可见，由于本申请实施例需要确定是否要隐藏门外访客的关键信息，以达到保护隐私的目地。因此，对于原始的第一场景图像，该图像中除了包括关键信息对应的区域，还包括其他不必要的区域。
因而，为了提高处理效率，以及后续步骤中避免对图像中不必要的区域进行处理，故进行初始ROI区域以及对应区域类型的提取。进而可以得到初始ROI区域中包括哪种类型区域。例如，初始ROI区域中包括人脸区域或者初始ROI区域中包括人形区域。相比直接从第一场景图像获得人脸区域以及人形区域的方式，可以使得实现该整体过程的处理复杂度低，减少时间成本，提高智能门锁的运行效率。
本申请中可以基于初始ROI区域，确定目标ROI区域。具体基于初始ROI区域，确定目标ROI区域的过程，包括：基于第一场景图像，获取初始ROI区域对应的像素数据。之后，基于第一深度图像和像素数据，获取初始ROI区域对应的初始深度数据。并基于距离修正参数修正初始ROI区域对应的初始深度数据，得到初始ROI区域对应的目标深度数据。最后，根据初始ROI区域对应的目标深度数据，对初始ROI区域进行筛选，将筛选后的初始ROI区域确定为目标ROI区域。具体实现方式可以参见下述步骤。
步骤S704、智能门锁基于第一场景图像获取初始ROI区域对应的像素数据。
步骤S705、智能门锁基于第一深度图像和像素数据获取初始ROI区域对应的初始深度数据。
在本申请实施例中，智能门锁能够建立第一深度图像和第一场景图像之间的对应关系，以确定第一场景图像中的每个像素的三维空间信息。可以理解的是，第一深度图像和第一场景图像之间的对应关系指的是将第一深度图像中的深度数据映射至第一场景图像中而形成的映射关系。在本申请实施例中，可以通过坐标转换的方式实现将第一深度图像中的深度值映射至第一场景图像。
例如，将第一场景图像划分为多个像素区域，将第一深度图像中的每个深度数据定义成一个深度值。这样，基于第一深度图像和第一场景图像之间的对应关系，每个像素区域均对应有一个深度值。进而，可以确定第一场景图像中每个像素区域对应的深度值，生成每个像素区域的三维空间信息。
再例如，将第一场景图像中每个图像区域中的每个像素定义成一个像素点，将第一深度图像中的每个深度数据定义成一个深度值。这样，基于第一深度图像和第一场景图像之间的对应关系，每个像素点均对应有一个深度值。进而，可以确定第一场景图像中每个像素点对应的深度值，生成每个像素区域的三维空间信息。
由此，智能门锁可以首先基于第一场景图像获取上述识别到的初始ROI区域的像素数据。可以理解的是，每个初始ROI区域包括多个像素，初始ROI区域中的每个像素被定义为一个像素点。那么每个初始ROI区域对应的像素数据包括多个像素点以及对应的像素值。之后，智能门锁基于第一深度图像和第一场景图像之间的对应关系，获取初始ROI区域对应的初始深度数据。由于第一深度图像和第一场景图像的采集位置和采集时刻是相同的。因此，像素数据中的每个像素点均在第一深度图像中对应有一个深度值。那么，多个像素点对应的多个深度值即构成初始ROI区域对应的初始深度数据。
可以理解的是，在本申请实施例中，由于每个初始ROI区域中均为一个整体区域，且区域内包括多个像素点。因此，本申请实施例中可以将每个初始ROI区域中的每个像素点对应有一个深度值。需要说明的是，本申请实施例仅以一个像素点对应有一个深度值为例，本领域技术人员可根据实际情况自行进行设计。
步骤S706、智能门锁对初始ROI区域对应的初始深度数据进行校正，生成初始ROI区域对应的目标深度数据。
在本申请实施例中，初始ROI区域对应的初始深度数据包括多个深度值。在空间场景下，因为第二图像传感器距离初始ROI区域的左侧端面以及右侧端面的直线距离不同，第二图像传感器拍摄初始ROI区域的左侧端面以及右侧端面的像素点对应的深度值也是不同的。然而初始ROI区域中无论任何位置的像素点均位于同一平面。因此，为了提高数据的精准度，智能门锁需要对初始深度数据进行校正。
在智能门锁对初始深度数据进行校正的过程中，为了提高数据处理速度，可以只选取初始ROI区域中的中心区域进行校正，以生成目标深度数据。其中，目标深度数据包括校正后的目标深度值。
可以理解的是，该中心区域对应的目标深度数据即可代表该中心区域对应初始ROI区域的目标深度数据。并且该中心区域为包围初始ROI区域中中心点的区域。同样地，该中心区域对应的初始深度数据同样也包括有多个深度值。之后，智能门锁可以通过计算该中心区域对应的初始深度数据的算术平均数，得到目标深度值。最后，基于距离修正参数校正目标深度值，完成校正过程。其中，距离修正参数是由上述联合标定第二图像传感器和第一图像传感器过程中得到的。由于一个初始ROI区域对应有一个校正后的目标深度值。因此，可以将一个或几个初始ROI区域中对应校正后的目标深度值构成目标深度数据。
示例性的，该中心区域为3×3的正方形区域。该正方形区域包括9个像素点以及对应有9个深度值。9个深度值分别为：1.5米、1.6米、1.5米、1.2米、0.8米、0.7米、0.5米、0.6米以及0.6。距离修正参数包括标志点相对于第二图像传感器偏移角度ɑ的正弦值，ɑ为30°。之后，将上述9个深度值计算算数平均数等于1米，即目标深度值为1米。最后，将偏移角度30°的正弦值与目标深度值1米相乘，即可得到校正后的目标深度值0.5米。由此，即可生成在实际空间范围内第二图像传感器与该初始ROI区域之间垂直方向的目标深度值为0.5米。
在一种可实现方式中，智能门锁还可以将初始ROI区域等分成多个子区域，并分别计算每个子区域的目标深度值。此时，每个子区域的目标深度值可被认为是每个子区域相对应的与第二图像传感器之间的深度值。然后，再通过计算多个子区域对应的目标深度值的算术平均数，即可得到该初始ROI区域对应的目标深度值。最后，基于距离修正参数校正目标深度值，完成校正过程。
可以理解的是，上述计算初始ROI区域对应的目标深度值可采用多种方法或公式。例如，每个子区域)的目标深度值的计算方法通过计算该区域内所有像素点对应的初始深度数据的中位数得到。再例如，还可以通过计算全部子区域内所有像素点对应的初始深度数据的算术平均数得到。再例如，还可以通过确定每个子区域中的平面中心的像素点和以该像素点对应的初始深度数据作为该区域的目标深度值。
步骤S707、智能门锁基于目标深度数据以及对应的置信度对初始ROI区域进行筛选，将筛选后的初始ROI区域确定为目标ROI区域。
在一些实施例中，为了进一步提高数据处理速度以及数据的准确性，智能门锁可以获取初始ROI区域对应的目标深度数据的置信度；并根据置信度和第二图像传感器的有效距离范围对初始ROI区域进行筛选；有效距离范围根据第二图像传感器的硬件条件设定。目的是过滤无效初始ROI区域。
可以理解的是，由于第二图像传感器自身根据器件的硬件条件，通常在采集深度数据时会对应有相应的有效距离范围。示例性的，智能门锁可以根据第二图像传感器的设定位置，设定近端阈值以及远端阈值。基于近端阈值和远端阈值确定有效距离范围。例如，第二图像传感器最近的采集距离为0.02米，即近端阈值为0.02米。第二图像传感器最远的采集距离为8米，即远端阈值为8米。进而，有效距离范围即为0.02米-8米。那么，也就是说如果第二图像传感器采集的深度数据位于有效距离范围之外，即可确定该深度数据为无效深度数据。
例如，如果在距离第二图像传感器10米处存在人脸区域或人形区域，且该区域已被识别成初始ROI区域。那么，由于距离第二图像传感器10米已超出第二图像传感器的有效距离范围，那么该初始ROI区域将不存在对应的深度数据，即为无效初始ROI区域。由此，可以通过将不具备空间位置的初始ROI区域进行过滤，提高数据处理速度。
在一些实施例中，置信度可以用于表示深度数据的可靠性，该置信度与接收到的反射光脉冲的光强呈正比，即接收到的反射光脉冲的光强越大，置信度值越大，深度数据的可靠性也越大。接收到的反射光脉冲的光强越小，置信度值越小，深度数据的可靠性也越小。在一种可实现方式中，置信度大小均在0-1之间(包括0和1)。
由于，第二图像传感器所采集的深度数据会存在噪声或者基于材质、遮挡以及距离等因素导致深度数据的缺失，形成空洞，使数据质量低。对后续的工作很有可能会造成影响。因此，还可以通过获取到初始ROI区域对应目标深度数据的置信度对初始ROI区域进行筛选，剔除无效初始ROI区域。
在一种可实现方式中，可以将初始ROI区域对应目标深度数据的置信度与第三预设阈值进行比对。如果初始ROI区域对应目标深度数据的置信度大于或者等于第三预设阈值，则可以确定该初始ROI区域为有效初始ROI区域。如果初始ROI区域对应目标深度数据的置信度小于第三预设阈值，则可以确定该初始ROI区域为无效初始ROI区域。最终，将筛选完成的初始ROI区域确定为目标ROI区域。
例如，特定阈值为20。如果初始ROI区域对应目标深度数据的置信度大于或者等于20，则可以确定该初始ROI区域为有效初始ROI区域。如果初始ROI区域对应目标深度数据的置信度小于20，则可以确定该初始ROI区域为无效初始ROI区域，并剔除该初始ROI区域。
可见，本申请实施例通过进一步对识别到的初始ROI区域进行筛选，剔除置信度较低的初始ROI区域以及深度数据位于有效距离范围之外的初始ROI区域，得到目标ROI区域。以便于更能精确地获取到门外用户的区域类型以及门外用户与智能门锁之间的距离。同时在后续对门外用户的区域类型进行处理的过程中，减少不必要的处理工作量，只需对目标ROI区域进行数据处理。提高了数据处理的效率，降低了智能门锁的功耗。
并且，上述通过基于第一深度图像和距离修正参数，可以获取目标ROI区域对应的目标深度数据；其中，距离修正参数根据第一图像传感器和第二图像传感器的标定结果获得，距离修正参数用于修正第一深度图像上的深度数据。进而，根据目标深度数据在目标ROI区域内进行模糊处理，能够精确根据目标深度数据对隐私对象的隐私区域进行隐藏。
步骤S708、智能门锁根据目标ROI区域对应的目标深度数据确定目标ROI区域满足第二预设条件时，对目标ROI区域进行模糊处理。
在本申请实施例中，根据门外用户与智能门锁之间的距离，可以设定多个并列的第二预设条件，从而决定是否隐藏门外用户的隐私信息。可以理解的是，需要隐藏隐私信息的门外用户应不是真正来家中到访的访客即非目标访客。同时，该非目标访客通常会距离智能门锁有一定的距离。
在一些实施例中，第二预设条件包括：目标ROI区域对应的目标深度数据所对应的目标深度 值位于第一预设范围内；或，目标ROI区域对应的目标深度数据所对应的目标深度值位于第二预设范围内，且目标ROI区域对应的目标深度数据的置信度大于第一预设阈值。
示例性的，第二预设条件包括：目标ROI区域对应的目标深度数据所对应的目标深度值位于第一预设范围内，且目标ROI区域中的隐私信息包括人脸。
例如，当目标ROI区域对应的目标深度数据满足第二预设条件时，对目标ROI区域执行模糊处理任务。参见图10，例如，第一预设范围为大于1.5米且小于3米。那么，当目标ROI区域B对应的目标深度数据大于1.5米且小于3米时，并且目标ROI区域B包括人脸区域，则对目标ROI区域B进行模糊处理。也就是说，为了获取到门外目标访客的图像信息，并隐藏非目标访客的隐私信息。可以根据目标ROI区域对应的目标深度数据确定门外用户是目标访客还是非目标访客。当目标ROI区域对应的目标深度数据所对应的目标深度值在第一预设范围内，可以认为目标ROI区域中表征的用户为非目标访客。因此，需要把非目标访客的关键信息进行隐藏，如隐藏目标ROI区域中包括的人脸区域，避免侵犯非目标访客的隐私信息。
在另一些实施例中，智能门锁还可以根据目标ROI区域的区域信息，对目标ROI区域进行模糊处理。其中，区域信息可以包括目标ROI区域对应的区域类型。
在一种可实现方式中，上述第二预设条件还可以包括目标ROI区域对应的目标深度数据的置信度大于第二预设阈值。例如，第二预设阈值为50。那么，当目标ROI区域B对应的目标深度数据大于1.5米且小于3米且对应的置信度大于50，以及目标ROI区域B包括人脸区域，则对目标ROI区域B进行模糊处理。
在一些实施例中，第二预设条件包括目标ROI区域对应的目标深度数据所对应的目标深度值位于第二预设范围内且置信度大于第一预设阈值。
示例性的，当目标ROI区域对应的目标深度数据在第二预设范围内，且置信度大于第一预设阈值时，对目标ROI区域执行模糊处理任务。继续参见11，例如，第二预设范围为大于且等于3米。那么，当目标ROI区域C对应的目标深度数据大于且等于3米时，并且目标深度数据对应的置信度大于80，则对目标ROI区域C进行模糊处理。可以理解的是，当目标ROI区域对应的目标深度数据在第二预设范围内，可以表征门外用户与智能门锁的距离更远，更精确确定需要对该目标ROI区域进行模糊处理。并且还可以结合置信度进行筛选。当目标深度数据的置信度大于第一预设阈值时，可以说明该目标深度数据的可靠性更高以及该目标深度数据为有效深度数据。因此，需要将目标ROI区域进行隐藏，避免侵犯非目标访客的隐私信息。
在本申请实施例中，智能门锁在目标ROI区域对应的目标深度数据满足第三预设条件时，不对目标ROI区域执行模糊处理任务。
示例性的，第三预设条件可以包括目标ROI区域对应的目标深度数据所对应的目标深度值位于第三预设范围内。例如，第三预设范围为小于或者等于1.5米。那么，当目标ROI区域对应的目标深度数据所对应的目标深度值小于或者等于1.5米时，则不对该ROI区域进行模糊处理。也就是说，当目标ROI区域对应的目标深度数据在第三预设范围内时，可以认为目标ROI区域中表征的用户为目标访客。因此，需要获取目标访客的关键信息，无需隐藏目标ROI区域。需要说明的是，本申请实施例并不对第二预设条件和第三预设条件进行具体限定。
由此，当智能门锁对目标ROI区域进行模糊处理之后，显示模糊处理后的第一场景图像；第一场景图像上包括模糊处理后的目标ROI区域。并且将模糊处理后的第一场景图像进行本地存储。示例性的，当用户查看智能门锁的猫眼画面时，即可看到如图10所示的第一场景图像。用户可以在第一场景图像中获取目标访客的关键信息，但无法获取非目标访客的关键信息。同样地，智能门锁也可以将多帧第一场景图像中的目标ROI区域进行模糊处理，生成多个模糊处理后的第一场景图像。以及将多个模糊处理后的第一场景图像进行叠加，生成目标视频，并存储在本地。这样，当用户回看目标视频时，可以在目标视频中获取目标访客的关键信息，但无法获取非目标访客的关键信息。避免侵犯非目标用户的隐私，提升智能门锁的安防效果。
在一些实施例中，上述模糊处理可以是对目标ROI区域进行像素值更改，也可以是对目标ROI区域进行马赛克算法。其中，像素值更改可以是将目标ROI区域内的像素点用同一个像素值进行替换，也可以是将目标ROI区域内的像素点对应的像素值进行高斯变换，以更改对应的像素值。 马赛克算法可以是将目标ROI区域分为多个像素块，每个像素块包含多个像素点。在每个像素块中随机取一个像素点的像素值，用该像素值替换对应像素块中其他像素点的像素值。
在一些实施例中，为了便于美观以及提升用户的使用体验，上述模糊处理还可以利用预设图案将目标ROI区域进行遮挡。其中，预设图案可以是卡通头像、物品图案以及表情包等。例如，对目标ROI区域中的人脸区域采用卡通头像进行遮挡。再例如，对目标ROI区域中的人形区域采用表情包进行遮挡。本申请实施例不对模糊处理的实现方式进行具体限定。
在一种可实现方式中，上述模糊处理可以是对整个目标ROI区域进行模糊。示例性的，参见图11，可以将目标ROI区域B和目标ROI区域C对应的检测框中的全部区域进行模糊。
在另一些实施例中，根据目标ROI区域对应的目标深度数据对目标ROI区域中的隐私区域进行模糊处理；其中，隐私区域用于表征隐私信息对应的区域，隐私区域包括隐私对象对应的人脸区域或人形区域。示例性的，还可以是对目标ROI区域中的局部进行模糊。继续参见图10，可以将目标ROI区域对应的检测框中的局部区域进行模糊。例如，仅将目标ROI区域B中的人脸区域进行模糊处理。再例如，仅将目标ROI区域C中的人形区域进行模糊处理。
可见，本申请实施例的智能门锁可以联合标定第一图像传感器和第二图像传感器。在联合标定第一图像传感器和第二图像传感器之后，实时采集第一场景图像和第一深度图像，并且识别包括人脸区域或人形区域的初始ROI区域。之后，再对初始ROI区域进行筛选，得到目标ROI区域。最后，若目标ROI区域对应的深度数据满足预设条件后，则对目标ROI区域进行模糊处理。由此，本申请实施例可以获取目标访客的区域类型，还可以隐藏非目标访客对应的隐私信息。避免侵犯非目标访客隐私的同时保证了智能门锁的安防效果，提高了智能门锁的安全性能，提高用户的使用体验。
需要说明的是，本申请实施例提供的图像处理方法并不仅局限应用于用户家中的智能门锁中。还可以应用于其他电子设备中。例如，应用于安装在汽车车门的智能门锁以及各种场所中的智能监控摄像头。再例如，还可以应用在手机中，当用户在户外使用手机进行视频通话时，手机中的摄像头很有可能采集到除了用户之外其他用户。再例如，还可以应用在笔记本电脑中，当用户在户外使用笔记本电脑进行视频会议时，笔记本电脑的摄像头也有可能采集到除了用户之外的其他用户。那么，无论用户在进行视频通话、视频会议过程中或者录制上述过程的屏幕，基于本申请实施例提供的图像处理方法可以提高隐私安全性。
在另一些实施例中，在对目标ROI区域模糊处理之前，智能门锁还可以对第一场景图像进行非安全元素的识别。如在第一场景图像中识别到非安全元素，则直接给予用户安全提醒。例如，智能门锁控制显示屏显示提醒信息以进行安全提醒。再例如，智能门锁还可以通过设置报警器并采用语音提示等方式对用户进行安全提醒。
在另一些实施例中，在对目标ROI区域模糊处理的过程中，如在第一场景图像中识别到非安全元素且该非安全元素存在目标ROI区域中，则不对该ROI区域进行模糊处理。
在另一些实施例中，在对目标ROI区域进行模糊处理的过程中，智能门锁还可以预先将未模糊的图像存储在本地，并自动设置该未模糊图像的查看权限。例如，查看权限包括仅能特定用户查看。特定用户可以为警察等。这样，既可以有效管理涉及门外用户隐私信息的查看权限，确保信息安全。还可以在使用智能门锁的用户侧隐藏非目标访客对应的隐私信息，提高用户的使用体验。
本申请实施例还提供一种智能门锁，智能门锁包括存储器、一个或多个处理器和摄像头；存储器与处理器耦合；其中，摄像头用于采集场景图像和深度图像；存储器中存储有计算机程序代码，计算机程序代码包括计算机指令，当计算机指令被处理器执行时，使得处理器执行上文所提供的对应的方法。
本申请实施例还提供一种电子设备，如图12所示，该电子设备可以包括一个或者多个处理器1010、存储器1020和通信接口1030。
其中，存储器1020、通信接口1030与处理器1010耦合。例如，存储器1020、通信接口1030与处理器1010可以通过总线1040耦合在一起。
其中，通信接口1030用于与其他设备进行数据传输。存储器1020中存储有计算机程序代码。 计算机程序代码包括计算机指令，当计算机指令被处理器1010执行时，使得电子设备执行本申请实施例中的图像处理方法。
其中，处理器1010可以是处理器或控制器，例如可以是中央处理器(Central Processing Unit，CPU)，通用处理器，数字信号处理器(Digital Signal Processor，DSP)，专用集成电路(Application-Specific Integrated Circuit，ASIC)，现场可编程门阵列(Field Programmable Gate Array，FPGA)或者其他可编程逻辑器件、晶体管逻辑器件、硬件部件或者其任意组合。其可以实现或执行结合本公开内容所描述的各种示例性的逻辑方框，模块和电路。所述处理器也可以是实现计算功能的组合，例如包含一个或多个微处理器组合，DSP和微处理器的组合等等。
其中，总线1040可以是外设部件互连标准(Peripheral Component Interconnect，PCI)总线或扩展工业标准结构(Extended Industry Standard Architecture，EISA)总线等。上述总线1040可以分为地址总线、数据总线、控制总线等。为便于表示，图12中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
本申请实施例还提供一种计算机可读存储介质，该计算机存储介质中存储有计算机程序代码，当上述处理器执行该计算机程序代码时，电子设备执行上述方法实施例中的相关方法步骤。
本申请实施例还提供了一种计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述方法实施例中的相关方法步骤。
其中，本申请提供的智能门锁、电子设备、计算机存储介质或者计算机程序产品均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
通过以上实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其他的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其他的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上内容，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (16)

  1. 一种图像处理方法，其特征在于，应用于电子设备，所述方法包括：
    获取第一场景图像和所述第一场景图像对应的第一深度图像；
    若确定所述第一场景图像中包括目标感兴趣ROI区域，则基于所述第一深度图像获取所述目标ROI区域对应的目标深度数据；所述目标ROI区域包括隐私对象对应的隐私信息；
    根据所述目标ROI区域对应的目标深度数据，在所述第一场景图像上的所述目标ROI区域内进行模糊处理。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述目标ROI区域对应的目标深度数据，在所述第一场景图像上的所述目标ROI区域内进行模糊处理，包括：
    根据所述目标ROI区域对应的目标深度数据对所述目标ROI区域中的隐私区域进行模糊处理；其中，所述隐私区域用于表征所述隐私信息对应的区域，所述隐私区域包括隐私对象对应的人脸区域或人形区域。
  3. 根据权利要求1或2所述的方法，其特征在于，所述方法还包括：
    显示模糊处理后的所述第一场景图像，所述第一场景图像上包括模糊处理后的所述目标ROI区域。
  4. 根据权利要求3所述的方法，其特征在于，所述电子设备包括第一图像传感器和第二图像传感器，所述第一图像传感器用于采集所述第一场景图像，所述第二图像传感器用于采集所述第一深度图像；所述获取第一场景图像和所述场景图像对应的第一深度图像，包括：
    在满足第一预设条件时，控制所述第一图像传感器和所述第二图像传感器同步采集的所述第一场景图像和所述第一深度图像；
    其中，所述第一预设条件包括：所述电子设备检测到隐私对象的声音分贝大于预设分贝，和/或，所述电子设备检测到隐私对象针对所述电子设备的触碰操作，和/或，所述电子设备检测到隐私对象的停留时间大于预设时间。
  5. 根据权利要求4所述的方法，其特征在于，所述根据所述目标ROI区域对应的目标深度数据对所述目标ROI区域进行模糊处理，包括：
    根据所述目标ROI区域对应的目标深度数据确定所述目标ROI区域满足第二预设条件时，对所述目标ROI区域进行模糊处理；
    其中，所述第二预设条件包括：
    所述目标ROI区域对应的目标深度数据所对应的目标深度值位于第一预设范围内；
    或，所述目标ROI区域对应的目标深度数据所对应的目标深度值位于第二预设范围内，且所述目标ROI区域对应的目标深度数据的置信度大于第一预设阈值。
  6. 根据权利要求5所述的方法，其特征在于，所述第二预设条件还包括：所述目标ROI区域中的所述隐私信息包括人脸。
  7. 根据权利要求4-6任一项所述的方法，其特征在于，所述基于所述第一深度图像获取所述目标ROI区域对应的目标深度数据，包括：
    基于所述第一深度图像和距离修正参数，获取所述目标ROI区域对应的目标深度数据；所述距离修正参数根据所述第一图像传感器和所述第二图像传感器的标定结果获得，所述距离修正参数用于修正所述第一深度图像上的深度数据。
  8. 根据权利要求7所述的方法，其特征在于，所述确定所述第一场景图像中的目标ROI区域，包括：
    识别所述第一场景图像中包括目标特征的初始ROI区域；其中，所述目标特征用于表征隐私对象对应的所述隐私信息，所述隐私信息包括隐私对象对应的人脸或人形；
    基于所述初始ROI区域，确定所述目标ROI区域。
  9. 根据权利要求8所述的方法，其特征在于，所述基于所述初始ROI区域，确定所述目标ROI区域，包括：
    基于所述第一场景图像，获取所述初始ROI区域对应的像素数据；
    基于所述第一深度图像和所述像素数据，获取所述初始ROI区域对应的初始深度数据；
    基于所述距离修正参数修正所述初始ROI区域对应的初始深度数据，得到所述初始ROI区域对应的目标深度数据；
    根据所述初始ROI区域对应的目标深度数据，对所述初始ROI区域进行筛选，将筛选后的所述初始ROI区域确定为所述目标ROI区域。
  10. 根据权利要求9所述的方法，其特征在于，所述根据所述初始ROI区域对应的目标深度数据对所述初始ROI区域进行筛选，包括：
    获取所述初始ROI区域对应的目标深度数据的置信度；
    根据所述置信度和所述第二图像传感器的有效距离范围对所述初始ROI区域进行筛选；所述有效距离范围根据所述第二图像传感器的硬件条件设定。
  11. 根据权利要求7-10任一项所述的方法，其特征在于，所述方法还包括：
    控制所述第一图像传感器和所述第二图像传感器同步采集第二场景图像和第二深度图像；
    基于所述第二场景图像计算得到所述第一图像传感器对应的标定参数；
    根据所述第一图像传感器对应的标定参数确定标定结果，所述标定结果包括所述第二图像传感器对应的所述距离修正参数；所述距离修正参数用于建立所述第一场景图像与所述第一深度图像之间的映射关系。
  12. 一种智能门锁，其特征在于，所述智能门锁包括存储器、一个或多个处理器和摄像头；所述存储器与所述处理器耦合；其中，所述摄像头用于采集第一场景图像和第一深度图像；所述存储器中存储有计算机程序代码，所述计算机程序代码包括计算机指令，当所述计算机指令被所述处理器执行时，使得所述处理器执行下述步骤：
    获取所述第一场景图像和所述第一场景图像对应的所述第一深度图像；
    若确定所述第一场景图像中包括目标感兴趣ROI区域，则基于所述第一深度图像获取所述目标ROI区域对应的目标深度数据；所述目标ROI区域包括隐私对象对应的隐私信息；
    根据所述目标ROI区域对应的目标深度数据，在所述第一场景图像上的所述目标ROI区域内进行模糊处理。
  13. 根据权利要求12所述的智能门锁，其特征在于，所述处理器执行所述根据所述目标ROI区域对应的目标深度数据，在所述第一场景图像上的所述目标ROI区域内进行模糊处理，包括：
    根据所述目标ROI区域对应的目标深度数据对所述目标ROI区域中的隐私区域进行模糊处理；其中，所述隐私区域用于表征所述隐私信息对应的区域，所述隐私区域包括隐私对象对应的人脸区域或人形区域。
  14. 根据权利要求12或13所述的智能门锁，其特征在于，所述处理器还执行下述步骤：
    显示模糊处理后的所述第一场景图像，所述第一场景图像上包括模糊处理后的所述目标ROI区域。
  15. 一种电子设备，其特征在于，所述电子设备包括存储器、一个或多个处理器；所述存储器与所述处理器耦合；其中，所述存储器中存储有计算机程序代码，所述计算机程序代码包括计算机指令，当所述计算机指令被所述处理器执行时，使得所述电子设备执行如权利要求1-11任一项所述的图像处理方法。
  16. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有指令，当所述指令在计算机上运行时，使得计算机可以执行如权利要求1-11任一项所述的图像处理方法。
PCT/CN2023/121062 2022-10-28 2023-09-25 一种图像处理方法及电子设备 WO2024087982A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211336084.7 2022-10-28
CN202211336084.7A CN117994121A (zh) 2022-10-28 2022-10-28 一种图像处理方法及电子设备

Publications (1)

Publication Number Publication Date
WO2024087982A1 true WO2024087982A1 (zh) 2024-05-02

Family

ID=90829990

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/121062 WO2024087982A1 (zh) 2022-10-28 2023-09-25 一种图像处理方法及电子设备

Country Status (2)

Country Link
CN (1) CN117994121A (zh)
WO (1) WO2024087982A1 (zh)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104982029A (zh) * 2012-12-20 2015-10-14 微软技术许可有限责任公司 具有隐私模式的摄像机
CN109085603A (zh) * 2017-06-14 2018-12-25 浙江舜宇智能光学技术有限公司 光学三维成像系统和彩色三维图像成像方法
CN109194876A (zh) * 2018-10-31 2019-01-11 Oppo广东移动通信有限公司 图像处理方法、装置、电子设备和计算机可读存储介质
CN110443744A (zh) * 2018-05-03 2019-11-12 安讯士有限公司 用于要应用于图像数据的模糊程度的方法、设备和系统
CN113538592A (zh) * 2021-06-18 2021-10-22 深圳奥锐达科技有限公司 一种距离测量装置与相机融合系统的标定方法及装置
WO2022127918A1 (zh) * 2020-12-18 2022-06-23 杭州海康威视数字技术股份有限公司 双目相机的立体标定方法、装置、系统及双目相机
CN114693760A (zh) * 2020-12-25 2022-07-01 虹软科技股份有限公司 图像校正方法、装置及系统、电子设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104982029A (zh) * 2012-12-20 2015-10-14 微软技术许可有限责任公司 具有隐私模式的摄像机
CN109085603A (zh) * 2017-06-14 2018-12-25 浙江舜宇智能光学技术有限公司 光学三维成像系统和彩色三维图像成像方法
CN110443744A (zh) * 2018-05-03 2019-11-12 安讯士有限公司 用于要应用于图像数据的模糊程度的方法、设备和系统
CN109194876A (zh) * 2018-10-31 2019-01-11 Oppo广东移动通信有限公司 图像处理方法、装置、电子设备和计算机可读存储介质
WO2022127918A1 (zh) * 2020-12-18 2022-06-23 杭州海康威视数字技术股份有限公司 双目相机的立体标定方法、装置、系统及双目相机
CN114693760A (zh) * 2020-12-25 2022-07-01 虹软科技股份有限公司 图像校正方法、装置及系统、电子设备
CN113538592A (zh) * 2021-06-18 2021-10-22 深圳奥锐达科技有限公司 一种距离测量装置与相机融合系统的标定方法及装置

Also Published As

Publication number Publication date
CN117994121A (zh) 2024-05-07

Similar Documents

Publication Publication Date Title
EP3869462A1 (en) Method for obtaining depth information and electronic device
WO2021078001A1 (zh) 一种图像增强方法及装置
WO2019080797A1 (zh) 一种活体检测方法、终端和存储介质
CN112037162B (zh) 一种面部痤疮的检测方法及设备
CN104463112B (zh) 一种采用rgb+ir图像传感器进行生物识别的方法及识别系统
US20220327720A1 (en) Sports timing based on a camera system
CN108647504B (zh) 实现信息安全显示的方法及系统
CN112036331A (zh) 活体检测模型的训练方法、装置、设备及存储介质
CN108764139B (zh) 一种人脸检测方法、移动终端及计算机可读存储介质
KR20180086048A (ko) 카메라 및 카메라의 영상 처리 방법
CN108629278B (zh) 基于深度相机实现信息安全显示的系统及方法
CN109040746A (zh) 摄像头校准方法和装置、电子设备、计算机可读存储介质
CN111708998A (zh) 一种人脸解锁方法及电子设备
CN109726613B (zh) 一种用于检测的方法和装置
EP3485423A1 (en) Method and system for smart group portrait
CN112153300A (zh) 多目摄像头曝光方法、装置、设备及介质
US9536133B2 (en) Display apparatus and control method for adjusting the eyes of a photographed user
WO2024087982A1 (zh) 一种图像处理方法及电子设备
WO2022161011A1 (zh) 生成图像的方法和电子设备
WO2020237542A1 (zh) 一种图像处理方法及装置
CN111062969B (zh) 目标跟踪方法及相关产品
CN114399622A (zh) 图像处理方法和相关装置
JP7228509B2 (ja) 識別装置及び電子機器
CN116109828B (zh) 图像处理方法和电子设备
CN113405674B (zh) 体温测量方法及摄像设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23881561

Country of ref document: EP

Kind code of ref document: A1