CN112204572A

CN112204572A - Data protection method, device and system for image recognition

Info

Publication number: CN112204572A
Application number: CN201980036875.8A
Authority: CN
Inventors: 张建永
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-01-04
Filing date: 2019-01-04
Publication date: 2021-01-08
Also published as: WO2020140296A1

Abstract

The embodiment of the application provides a data protection method, a device and a system for image recognition, relates to the technical field of image recognition, and can more effectively protect data and prevent the data from being attacked by replay. The method comprises the following steps: the method comprises the steps that an image recognition device obtains an image data frame to be recognized, wherein the image data frame to be recognized comprises a first time stamp, and the first time stamp is a time point of the image processing device generating the image data frame to be recognized; acquiring a second time stamp which is a time point when the image recognition device acquires the image data frame to be recognized; and the image recognition device compares the second time stamp with the first time stamp to obtain a time difference between the second time stamp and the first time stamp; if the time difference is smaller than or equal to a preset time threshold, the image recognition device determines that the image data frame to be recognized is valid.

Description

Data protection method, device and system for image recognition

Technical Field

The embodiment of the application relates to the technical field of image recognition, in particular to a data protection method, device and system for image recognition.

Background

In recent years, face recognition technology is becoming more mature, and user identity can be authenticated through the face recognition technology, for example, identity authentication in scenes such as mobile phone unlocking, payment or entrance guard.

In the process of face recognition, a face recognition system needs to be processed by several links, such as face image acquisition, image processing, face detection, living body detection, feature extraction, feature comparison and the like, so as to obtain a face recognition result, and the links may have a data security problem, for example, the acquired face image may not be a real face image, may be a photograph, a video, a false mask and the like, and the image processing result may be intercepted or tampered, the living body detection result is tampered, the feature extraction result is stolen or replaced, the feature comparison result is tampered and the like. At present, in order to solve the problems of data tampering, stealing, replacing, and the like in the face recognition process, some or all of the processing links may be migrated to a Trusted Execution Environment (TEE) based on the design specification of a trust zone (trustzone) security architecture for execution, which may also be referred to as a secure environment or a secure zone, and for example, image processing, face detection, living body detection, feature extraction, and feature comparison are performed in the TEE.

However, in the above method, even if the above links are migrated to the TEE for execution, the problem that the face data is attacked by replay may not be avoided.

Disclosure of Invention

The embodiment of the application provides a data protection method, device and system for image recognition, which can more effectively protect data and prevent the data from being attacked by replay.

In a first aspect, an embodiment of the present application provides a data protection method for image recognition, where the method may include: the method comprises the steps that an image recognition device obtains an image data frame to be recognized, wherein the image data frame to be recognized comprises a first time stamp, and the first time stamp is a time point of the image processing device generating the image data frame to be recognized; acquiring a second time stamp which is a time point when the image recognition device acquires the image data frame to be recognized; further, the image recognition device compares a second time stamp with the first time stamp to obtain a time difference between the second time stamp and the first time stamp; and if the time difference is smaller than or equal to the preset time threshold, the image recognition device determines that the image data frame to be recognized is valid.

In this embodiment of the application, when the time difference between the second timestamp and the first timestamp is less than or equal to the preset time threshold, it may be determined that the image data frame to be recognized is the image data frame generated by the image processing apparatus for the first time, and does not belong to replay attack, and then the image recognition apparatus determines that the image data frame to be recognized is valid.

In a possible implementation manner, if the time difference between the second time stamp and the first time stamp is greater than a preset time threshold, the image recognition apparatus determines that the image data frame to be recognized is invalid.

In the embodiment of the application, when the time difference between the second timestamp and the first timestamp is greater than the preset time threshold, it is described that the first timestamp in the image data frame to be recognized is relatively small, that is, the first timestamp is added to the image data frame to be recognized by the image processing apparatus at an earlier time, so that it can be determined that, in the current recognition process, the image data frame to be recognized is not an image data frame generated by the image processing apparatus for the first time, but a historical image data frame to be recognized is repeatedly used, and belongs to replay attack, and then the image recognition apparatus determines that the image data frame to be recognized is invalid.

In a possible implementation manner, the timestamp (including the first timestamp and the second timestamp) may be in different formats, including but not limited to original information of the timestamp, encrypted time information of the timestamp, or private-key-signed time information of the timestamp.

In a possible implementation manner, after the image recognition apparatus determines that the image data frame to be recognized is valid, the data protection method for image recognition provided in the embodiment of the present application may further include: the image recognition device recognizes the image data to be recognized, namely the image recognition device executes at least one of algorithms such as face detection, living body detection, feature extraction and feature comparison to obtain an image recognition result.

In a possible implementation manner, if the image recognition device determines that the image data frame to be recognized is invalid, the image recognition device discards the image data frame to be recognized, and does not recognize the image data frame to be recognized any more, and the image recognition device may continue to determine whether the next image data frame is valid.

In a possible implementation manner, the image data frame to be recognized includes a 2D image data frame and a depth image data frame corresponding to the 2D image data frame.

In a possible implementation manner, the first timestamp is included in the depth image data frame, and in the field of face recognition, the depth information may reflect personal information that a face is relatively sensitive (the depth information includes physical information of the acquired image), and recognition by using the depth information has a good recognition effect.

In a possible implementation manner, the preset time threshold is an identification period of a frame of image data. In the embodiment of the present application, the identification period of one image data frame refers to a time period from the image acquisition of the image acquisition device to the final output of the image identification result by the image identification device.

In a second aspect, an embodiment of the present application provides a data protection method for image recognition, where the method may include: the image processing device generates an image data frame to be identified, and the image processing device acquires a first time stamp, wherein the first time stamp is a time point of the image data frame to be identified generated by the image processing device; adding a first time stamp in the image data frame to be identified; and the image processing device sends the image data frame to be identified with the first timestamp added to the image identification device.

In the embodiment of the application, after the image processing apparatus generates the image data frame to be recognized, the image processing apparatus may read a first timestamp from an existing RTC device in a system-on-chip of the face recognition system, where the first timestamp is a time point at which the image processing apparatus generates the image data frame to be recognized.

Further, the RTC device is a peripheral device configurable under a trusted zone security architecture, the RTC device is connected to the face recognition system through an I2C peripheral bus, and since an I2C driver (i.e., a driver of an I2C peripheral bus) operates in a secure environment, the RTC device also operates in the secure environment, and a non-secure area cannot access time information in a register of the RTC device.

In a possible implementation manner, a source of the timestamp (including the first timestamp and the second timestamp) may also be a system timer, or may also be other devices with a timing function, which may be specifically selected according to an actual use requirement, and this is not limited in this embodiment of the application.

In the embodiment of the application, when the image processing device acquires one image data frame to be identified for the first time, the first timestamp added to the image data frame to be identified has uniqueness, and the first timestamp in the image data frame to be identified cannot be changed any more.

In one possible implementation manner, after the image processing apparatus generates the image data frame to be recognized, the image data frame to be recognized needs to be sent to the image recognition apparatus, and for example, the image data frame to be recognized may be transmitted between the image processing apparatus and the image recognition apparatus in a memory sharing manner, an IPC mailbox manner, or a remote network connection manner. Taking the mode of sharing the memory as an example, the face recognition system further includes a secure memory dynamically allocated for the service application, which is called a dynamic secure memory, and the dynamic secure memory is a shared memory domain in the TEE and can be used for storing an operation result of each device in the face recognition system. The process of sending the image data frame to be identified added with the first timestamp to the image identification device by the image processing device may specifically include: the image processing device buffers the image data frame to be identified which is generated by the image processing device and added with the first time stamp in the dynamic security memory, so that the image identification device can read the image data frame to be identified from the dynamic security memory.

In a possible implementation manner, the method for adding the first timestamp to the image data frame to be recognized by the image processing apparatus may include: the image processing apparatus adds a first time stamp in the depth image data frame.

Optionally, the first timestamp may also be added to both the 2D image data frame and the depth image data frame, which is not specifically limited in this embodiment of the application.

In addition, when the first time stamp is added to the depth image data frame, the image processing apparatus may add the first time stamp to an arbitrary position of the depth image data frame. For example, the first timestamp may be added at the head of the depth image data frame, or may be added at the tail of the depth image data frame, or may be added at the middle of the depth image data frame, so that the first timestamp does not need to be added by adding an additional field, and the bit overhead can be saved.

With reference to the first aspect and the second aspect, in the image recognition process, after the image processing device generates the image data frame to be recognized, a time point (the first time stamp) at which the image data frame to be recognized is generated is added to the image data frame to be recognized and sent to the image recognition device, and then the image recognition device acquires the image data frame to be recognized and acquires a time point (the second time stamp) at which the image recognition device acquires the image data frame to be recognized, and if a time difference between the second time stamp and the first time stamp is less than or equal to a preset time threshold, the image recognition device determines that the image data frame to be recognized is valid; if the time difference between the second timestamp and the first timestamp is greater than the preset time threshold, the image recognition device determines that the image data frame to be recognized is invalid.

Further, the data protection method for image recognition provided by the embodiment of the application adds the first time stamp to the image data frame to be recognized, acquires the second time stamp, and determines whether the image data frame to be recognized is valid without depending on special hardware, thereby reducing the complexity of data protection.

In a third aspect, an embodiment of the present application provides an image recognition apparatus, which includes an obtaining module, a comparing module, and a determining module. The image processing device comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring an image data frame to be identified and a second timestamp, the image data frame to be identified comprises a first timestamp, and the first timestamp is a time point of the image processing device for generating the image data frame to be identified; the acquisition module is further used for acquiring a second time stamp, wherein the second time stamp is a time point when the image identification device acquires the image data frame to be identified; the comparison module is used for comparing the time difference between the second time stamp and the first time stamp; the determining module is used for determining that the image data frame to be identified is valid under the condition that the time difference is smaller than or equal to a preset time threshold.

In a possible implementation manner, the determining module is further configured to determine that the image data frame to be recognized is invalid when a time difference between the second timestamp and the first timestamp is greater than a preset time threshold.

In a possible implementation manner, the image recognition apparatus provided in the embodiment of the present application further includes a recognition module; the identification module is used for identifying the image data frame to be identified after the determination module determines that the image data frame to be identified is valid.

In one possible implementation, the first timestamp is included in the frame of depth image data.

In a possible implementation manner, the preset time threshold is an identification period of a frame of image data.

In a fourth aspect, an embodiment of the present application provides an image processing apparatus, which includes a generating module, an obtaining module, an adding module, and a sending module. The generation module is used for generating an image data frame to be identified; the acquisition module is used for acquiring a first time stamp, wherein the first time stamp is a time point of generating an image data frame to be identified by the image processing device; the adding module is used for adding the first time stamp acquired by the acquiring module in the image data frame to be identified; the sending module is used for sending the image data frame to be identified added with the first time stamp to the image identification device.

In a possible implementation manner, the adding module is specifically configured to add a first timestamp to the depth image data frame.

In a fifth aspect, an embodiment of the present application provides an image recognition apparatus, including a processor and a memory coupled to the processor; the memory is used for storing computer instructions, and when the image recognition device runs, the processor executes the computer instructions stored in the memory, so that the image recognition device executes the data protection method for image recognition described in any one of the first aspect and the possible implementation manners.

In a sixth aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium may include computer instructions, which, when executed on a computer, cause an image recognition apparatus to perform the data protection method for image recognition described in any one of the first aspect and possible implementation manners thereof.

In a seventh aspect, an embodiment of the present application provides a computer program product including computer instructions, which, when run on a computer, causes an image recognition apparatus to execute the data protection method for image recognition described in any one of the first aspect and its possible implementation manners.

In an eighth aspect, the present application provides an image recognition apparatus, which exists in the form of a chip product, and the image recognition apparatus includes a processor and a memory, where the memory is configured to be coupled to the processor, and the memory is configured to store computer instructions, and the processor is configured to execute the computer instructions stored in the memory, so that the image recognition apparatus executes the data protection method for image recognition described in any one of the first aspect and possible implementation manners thereof.

In a ninth aspect, an embodiment of the present application provides an image processing apparatus, including a processor and a memory coupled to the processor; the memory is used for storing computer instructions, and when the image processing device runs, the processor executes the computer instructions stored in the memory, so that the image processing device executes the data protection method for image recognition in any one of the second aspect and the possible implementation manners.

In a tenth aspect, embodiments of the present application provide a computer-readable storage medium, which may include computer instructions that, when executed on a computer, cause an image processing apparatus to execute the data protection method for image recognition described in any one of the second aspect and possible implementation manners.

In an eleventh aspect, the present application provides a computer program product including computer instructions, which when run on a computer, causes an image processing apparatus to execute the data protection method for image recognition described in any one of the second aspect and its possible implementation manners.

In a twelfth aspect, the present application provides an image processing apparatus, which exists in the form of a chip product, and the image processing apparatus includes a processor and a memory, the memory is configured to be coupled to the processor, the memory can be used to store computer instructions, and the processor is configured to execute the computer instructions stored in the memory, so that the image processing apparatus executes the data protection method for image recognition described in any one of the second aspect and possible implementation manners thereof.

In a thirteenth aspect, the present application provides an image recognition system including the image recognition apparatus of the third aspect and the image processing apparatus of the fourth aspect, or including the image recognition apparatus of the fifth aspect and the image processing apparatus of the ninth aspect. Optionally, the image recognition system further comprises an image acquisition device.

It is to be understood that the image recognition apparatuses according to the third, fifth and eighth aspects, the image processing apparatuses according to the fourth, ninth and twelfth aspects, the computer storage media according to the sixth and tenth aspects, and the computer program products according to the seventh and eleventh aspects are all configured to perform the corresponding methods provided above, and therefore, the beneficial effects achieved by the computer program products can refer to the beneficial effects in the corresponding methods provided above, and are not repeated herein.

Drawings

Fig. 1 is a schematic diagram of an architecture of a face recognition system according to an embodiment of the present application;

fig. 2 is a schematic diagram of an algorithm flow of a face recognition process according to an embodiment of the present application;

fig. 3 is a first schematic diagram illustrating a data protection method for image recognition according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram illustrating a data protection method for image recognition according to an embodiment of the present application;

fig. 5 is a first schematic structural diagram of an image recognition apparatus according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of an image recognition apparatus according to an embodiment of the present application;

fig. 7 is a first schematic structural diagram of an image processing apparatus according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.

Detailed Description

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. The terms "first" and "second," and the like, in the description and in the claims of the embodiments of the present application are used for distinguishing between different objects and not for describing a particular order of the objects. For example, the first and second timestamps, etc. are used to distinguish between different timestamps and are not used to describe a particular order of the timestamps. In the embodiments of the present application, words such as "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion. In the description of the embodiments of the present application, the meaning of "a plurality" means two or more unless otherwise specified. For example, a plurality of processing units refers to two or more processing units; the plurality of systems refers to two or more systems. First, some concepts related to the data protection method and apparatus for image recognition provided in the embodiments of the present application are explained.

Replay attacks: the method can also be called replay attack or replay attack, and refers to that an attacker sends or inputs data (namely, history data is recycled) which has been received by a destination device for fraud authentication, for example, by taking face recognition as an example, a face recognition device acquires an image data frame to be recognized (called a first image data frame to be recognized), performs face recognition on the first image data frame to be recognized, and the recognition is successful (namely, the authentication is successful), the face recognition device acquires an image frame to be recognized again (called a second image data frame to be recognized), and if the second image data frame to be recognized is the same as the first image data frame to be recognized, performing face authentication again with the second image data frame to be recognized is called replay attack, so that the security of the face recognition system is low.

The trusted execution environment: a trusted execution environment (e.g., TEE) based on design specifications of a trusted zone security architecture, a device or module configured to operate in the TEE, can deny access from an insecure area (e.g., prevent data from being tampered with, etc.), i.e., can achieve a security isolation effect, so that the device or module can operate in a secure state.

Based on the problems in the background art, the embodiments of the present application provide a data protection method for image recognition, in the process of image recognition, after an image processing apparatus generates an image data frame to be recognized, a time point (referred to as a first time stamp) at which the image data frame to be recognized is generated is added to the image data frame to be recognized and sent to an image recognition apparatus, and then the image recognition apparatus acquires the image data frame to be recognized and obtains a time point (referred to as a second time stamp) at which the image recognition apparatus acquires the image data frame to be recognized, the image recognition apparatus determines whether the image data frame to be recognized is valid (i.e. whether there is a replay attack) by comparing a time difference between the second time stamp and the first time stamp, and specifically, if the time difference between the second time stamp and the first time stamp is greater than a preset time threshold, the image recognition apparatus determines that the image data frame to be recognized is invalid (i.e. there is a replay attack), the image recognition device discards the image data frame to be recognized and does not recognize the image data frame; if the time difference between the second time stamp and the first time stamp is less than or equal to the preset time threshold, the image recognition device determines that the image data frame to be recognized is valid (no replay attack exists), and the image recognition device continues to recognize the image data frame to be recognized. Through the technical scheme provided by the embodiment of the application, data protection can be more effectively carried out, and data is prevented from being attacked by replay.

The data protection method and apparatus for image recognition provided in the embodiment of the present application may be applied to an image recognition system, such as face recognition, landscape recognition, animal recognition, and the like, for example, taking a face recognition system as an example, fig. 1 is a schematic structural diagram of a face recognition system 100 provided in the embodiment of the present application, and as shown in fig. 1, the face recognition system 100 includes: the system comprises an image acquisition device 101, an image processing device 102, an image recognition device 103 and the like, wherein the image acquisition device 101 is used for acquiring a human face image and sending the acquired human face image to the image processing device 102; the image processing device 102 is configured to process the received face image, generate a face image data frame from the face image, and send the face image data frame to the image recognition device 103; the image recognition device 103 is configured to recognize face data and output a face recognition result.

Specifically, the image acquisition device 101 is composed of a sensor for acquiring a face image and other auxiliary control components (e.g., a control register, a flash device, etc.), and the image acquisition device 101 is mainly responsible for acquiring the face image, including acquisition of a 2D face image and/or acquisition of a 3D face image (the 3D face image includes a depth image corresponding to the 2D face image and includes depth information of the face image). The following description will take the example of acquiring and processing a 3D image, but the invention is not limited thereto.

The image processing apparatus 102 may include a 2D image processing unit and a depth information processing unit, or the image processing apparatus 102 may include a Digital Signal Processor (DSP) and some other auxiliary control circuits (e.g., a Central Processing Unit (CPU), a register, a Direct Memory Access (DMA), a memory, etc.), and firmware running on the DSP (the firmware refers to a driver of a device stored inside the device, and may be understood as a software system). The 2D image processing unit may convert the 2D face image acquired by the image acquisition device 101 into a format of a 2D face image data frame, and the depth information processing unit may convert the 3D face image acquired by the image acquisition device 101 into a format of a 3D face image data frame (in the following embodiments, the 3D image data frame is collectively referred to as a depth image data frame). Alternatively, the image processing apparatus 102 may include an Image Signal Processor (ISP). Alternatively, the 2D image processing unit and the depth information processing unit may be hardware or software running on a DSP or ISP.

The image recognition device 103 mainly executes a face recognition algorithm, where the face recognition algorithm includes face detection, live body detection, feature extraction, feature comparison, and the like, where the face detection is to primarily detect whether an image to be recognized is a face, and the live body detection is to detect whether face data is from a real living body (i.e., whether the data is real biological information), so as to exclude a face image forged by a photograph, a video, a mask, silica gel, or the like. The feature extraction is mainly to extract some features, such as mouth features, eye features and the like, from the face image data frame detected by living bodies so as to facilitate face recognition. The feature comparison is to compare the features extracted from the human face image data frame to be recognized with the corresponding features of the human face sample stored in the human face database, if the features are matched with the features of the human face sample, the recognition is successful, otherwise, the recognition is failed.

In this embodiment, the image recognition device 103 may include an algorithm scheduling processing module (which may also be divided into an algorithm processing module and an algorithm scheduling module) and an Artificial Intelligence (AI) computing module, where the algorithm scheduling processing module is a trusted application (trust application) running in the TEE, and is a pure software application module, and may be implemented based on a CPU, and is responsible for processing and scheduling various algorithms in the face recognition process, such as face detection, living body detection, feature extraction, feature comparison, and the like. It should be noted that the AI calculation module is implemented by a software method, such as a neural Network Processor (NPU) -based method, and is mainly used for accelerating some algorithms in face recognition, for example, accelerating part of algorithms in feature extraction and feature comparison (i.e., providing computational support). Therefore, the image recognition device 103 may also include a CPU, an NPU, or other processor to execute the above modules, and the embodiment of the present application is not particularly limited.

It should be noted that, in the embodiment of the present application, the image processing apparatus 102 in the face recognition system 100 complies with the design specification of the security architecture of the trust zone, that is, the image processing apparatus 102 can operate in the TEE and can deny access from the non-secure domain when operating in the TEE, so as to achieve the protection effect of security isolation. And the image recognition device 103 described above also operates in the TEE. In addition, the face recognition system 100 provided in the embodiment of the present application may further include a dynamic secure memory, where the dynamic secure memory is a secure memory allocated for some service applications (for example, face recognition) in the system, and may also be considered as a memory domain in the TEE.

In the face recognition system 100 provided in this embodiment of the application, a peripheral device, for example, an RTC (real _ time clock) device, including a driver, may be configured in the TEE, where the RTC device is a device independent from a system timer, and is used to set a system clock, provide an alarm or a periodic timer, and the RTC device may be considered as a time-ketone device or a timer. In the embodiment of the present application, the RTC device is connected to the image processing apparatus 102 and the image recognition apparatus 103 in the face recognition system through the I2C peripheral bus, respectively, in the embodiment of the present application, when the image processing apparatus 102 generates the frame of image data to be recognized, the current time may be read from the RTC device as the time point for generating the frame of image data to be recognized. When the image recognition device 103 acquires the image data frame to be recognized, the current time may be read from the RTC device as a time point at which the image data frame to be recognized is acquired.

It is understood that in fig. 1, the RTC device is located outside the image processing apparatus 102 and the image recognition apparatus 103, but this drawing is only an example. In fact, the RTC device may not be a device independent from the image processing apparatus 102 and the image recognition apparatus 103, and may be, for example, a plurality of devices, respectively distributed in the image processing apparatus 102 and the image recognition apparatus 103, for implementing time recording or timing, so that the processing units or processors of the image processing apparatus 102 and the image recognition apparatus 103 respectively perform time timing or obtain a timestamp according to the distributed RTC devices therein, which is not limited in this embodiment.

Exemplarily, in the process of face recognition, the nth face image of the N face images to be recognized acquired by the image acquisition device 101 is taken as an example to describe the process of face recognition performed by cooperation of each device in the face recognition system 100. As shown in fig. 2, after the image acquisition device 101 acquires an nth face image (the face image includes a 2D image and a depth image corresponding to the 2D image), the 2D image is sent to the 2D image processing unit of the image processing device 102 for processing, the depth image is sent to the depth information processing unit of the image processing device 102 for processing, an nth frame of image data frame (including an nth frame of 2D image data frame and an nth frame of depth image data frame) is obtained, the nth frame of image data frame is further sent to the image recognition device 103, the nth frame of image data frame is subjected to face detection, living body recognition, feature extraction and feature comparison by the algorithm scheduling processing module, and in the process of feature extraction by the algorithm scheduling processing module, part of algorithms for feature extraction and part of algorithms for feature comparison are scheduled to the AI calculation module for processing, the AI computing module returns the processing result (which can be regarded as an intermediate result) to the algorithm scheduling processing module, and then the algorithm scheduling processing module further completes feature extraction and feature comparison according to the processing result returned by the AI computing module to obtain a face recognition result.

Optionally, in an embodiment of the present application, in practical application, the face recognition system may be located in one device, for example, both the image processing apparatus and the image recognition apparatus are located in a terminal device, or the face recognition system may be located in multiple devices, for example, the image processing apparatus is located in one terminal, and the image recognition apparatus is located in one server. The previous and subsequent embodiments focus on the case where the image processing apparatus and the image recognition apparatus are both located in the same device, but this is merely an example and is not intended to limit the practical application.

In the following embodiments, taking a scene of face recognition as an example, in the field of face recognition, the data protection method for image recognition provided in the embodiments of the present application mainly refers to that an image data frame for protecting a face is replayed and attacked, and the data protection method for image recognition provided in the embodiments of the present application is described in detail below from the perspective of interaction between an image processing apparatus and an image recognition apparatus.

It should be noted that, in the embodiment of the present application, in a process of one face recognition, the data acquisition device needs to continuously acquire a plurality of groups of face images (i.e., a plurality of face images), for example, face images acquired from different angles, and then the image processing device and the image recognition device complete recognition of the plurality of groups of face images to obtain a final face recognition result, that is, each face image in the plurality of face images is processed according to the technical scheme provided in the embodiment of the present application to obtain a recognition result. In the following embodiments, an example description is given by taking one face image in a plurality of sets of face data as an example.

As shown in fig. 3, the data protection method for image recognition provided in the embodiment of the present application may include S101-S107: in step S101, the image processing apparatus 102 generates an image data frame to be recognized. With reference to fig. 1 and fig. 2, in the face recognition system, after the image to be recognized is collected by the image collection device 101, the image to be recognized is transmitted to the image processing device 102, the image to be recognized includes a 2D image and a depth image corresponding to the 2D image, and then the image processing device 102 processes the image to be recognized to generate an image data frame to be recognized, that is, the collected image to be recognized is converted into an image data frame format, where the image data frame to be recognized includes a 2D image data frame and a depth image data frame.

S102, the image processing apparatus 102 acquires a first timestamp, which is a time point when the image data frame to be recognized is generated. In this embodiment of the application, after the image processing apparatus 102 generates the image data frame to be recognized, the image processing apparatus 102 may read a first timestamp from an existing RTC device in a system-on-chip of the face recognition system, where the first timestamp is a time point at which the image processing apparatus 102 generates the image data frame to be recognized. Optionally, the source of the first timestamp may also be a system timer, or may also be other devices with a timing function, which may be specifically selected according to actual use requirements, and the embodiment of the present application is not limited.

S103, the image processing device 102 adds a first time stamp to the image data frame to be identified. In this embodiment of the application, the method for adding the first timestamp to the image data frame to be recognized by the image processing apparatus 102 specifically may include: the first time stamp is added in the depth image data frame of the image data frame to be recognized, and in the field of face recognition, the depth information can reflect personal information (the depth information comprises physical information of the collected image) sensitive to the face, and recognition by utilizing the depth information has a good recognition effect, so that the first time stamp is added in the depth image data frame. As an example, in the embodiment of the present application, the adding of the first timestamp in the image data frame to be recognized refers to adding the first timestamp in the depth image data frame of the image data frame to be recognized. Optionally, in this embodiment of the application, the first timestamp may also be added to both the 2D image data frame and the depth image data frame, and this embodiment of the application is not particularly limited.

In the embodiment of the present application, when the first timestamp is added to the depth image data frame, the image processing apparatus 102 may add the first timestamp to an arbitrary position of the depth image data frame. For example, the first time stamp may be added to the head of the depth image data frame, the first time stamp may be added to the tail of the depth image data frame, or the first time stamp may be added to the middle of the depth image data frame. Optionally, the first timestamp occupies 64 bits. The first time stamp is added to the image data frame to be identified, so that an additional field is not required to be added to add the first time stamp, and bit overhead can be saved.

Optionally, in this embodiment of the application, the first timestamp may be in different formats, including but not limited to original information of the first timestamp, time information obtained by encrypting the first timestamp, or time information obtained by performing private key signature on the first timestamp. Moreover, it should be noted that, in the whole process of face recognition (including various subsequent recognition algorithms), the first timestamp will be carried in the image data frame to be recognized (including the generated image data frame to be recognized and the feature data extracted subsequently). It is particularly noted that, in the embodiment of the present application, when the image processing apparatus 102 acquires one image data frame to be identified for the first time, the first timestamp added to the image data frame to be identified has uniqueness, and the first timestamp in the image data frame to be identified is not changed any more.

S104, the image processing device 102 sends the image data frame to be identified added with the first time stamp to the image identification device 103. In the embodiment of the present application, after the image processing apparatus 102 generates the image data frame to be recognized, the image data frame to be recognized needs to be sent to the image recognition apparatus 103, for example, the image data frame to be recognized may be transmitted between the image processing apparatus 102 and the image recognition apparatus 103 in a memory sharing manner, an inter-processor communication (IPC) mailbox manner, or a remote network connection manner.

Exemplarily, taking a mode of sharing a memory as an example, the face recognition system further includes a secure memory dynamically allocated for a service application, which is called a dynamic secure memory, and the dynamic secure memory is a shared memory domain in the TEE and can be used for storing an operation result of each device in the face recognition system. The process of sending the image data frame to be recognized with the first timestamp added thereto to the image recognition apparatus 103 by the image processing apparatus 102 may specifically include: the image processing device 102 buffers the frame of image data to be recognized, which is generated by the image processing device and added with the first timestamp, in the dynamic secure memory, so that the image recognition device 103 can read the frame of image data to be recognized from the dynamic secure memory. For the description of the manner of IPC mailbox and the manner of remote network connection, reference may be made to the related art, and the embodiments of the present application are not described in detail.

S105, the image recognition device 103 acquires the image data frame to be recognized. And the image data frame to be identified comprises the first time stamp. In conjunction with the above description of S104, correspondingly, the acquiring, by the image recognition device 103, the frame of image data to be recognized may include: the image recognition device 103 reads the image data frame to be recognized from the dynamic security memory, and the image data frame to be recognized carries the first time stamp.

S106, the image recognition device 103 obtains a second timestamp, where the second timestamp is obtained by the image recognition device 103, for example, a time point when the frame of image data to be recognized is received. Simultaneously with or after the image recognition device 103 acquires the frame of image data to be recognized, the image recognition device 103 reads a second time stamp from the RTC device, where the second time stamp is used as a time point when the image recognition device 103 acquires the frame of image data to be recognized. The first time stamp and the second time stamp are used for determining whether the image data frame to be recognized is a valid image data frame, namely whether the image data frame to be recognized can be continuously used for face recognition. For the description of other contents of the second timestamp, reference may be made to the description of the first timestamp in the foregoing embodiment, and details are not described here.

S107, the image recognition device 103 compares the second time stamp with the first time stamp to obtain a time difference between the second time stamp and the first time stamp.

S108, the image recognition device 103 determines whether a time difference between the first time stamp and the second time stamp is greater than a preset time threshold. In this embodiment of the application, the preset time threshold may be an identification period of an image data frame, where the identification period of an image data frame refers to a time period from image acquisition by the image acquisition device to final output of an image identification result by the image identification device 103. The preset time threshold may also be other values meeting the use requirement, and the embodiment of the present application is not particularly limited. After S108 described above, S109 described below or S110 shown in fig. 4 may be performed.

And S109, if the time difference between the second time stamp and the first time stamp is less than or equal to the preset time threshold, the image recognition device 103 determines that the image data frame to be recognized is valid. In this embodiment of the application, the time difference between the second timestamp and the first timestamp is less than or equal to the preset time threshold, which indicates that the image data frame to be recognized is the image data frame generated by the image processing apparatus 102 for the first time, and does not belong to replay attack, and then the image recognition apparatus 103 determines that the image data frame to be recognized is valid. If the image recognition device 103 determines that the image data frame to be recognized is valid, the image recognition device 103 starts to recognize the image data frame to be recognized, that is, algorithms such as face detection, living body detection, feature extraction, feature comparison and the like are executed to obtain an image recognition result.

As shown in fig. 4, in S110, if the time difference between the second time stamp and the first time stamp is greater than the preset time threshold, the image recognition device 103 determines that the image data frame to be recognized is invalid. In this embodiment, the image processing apparatus 102 generates the image data frame to be recognized until the image recognition apparatus 103 acquires the image data frame to be recognized, where the time difference is a time difference between the second time stamp and the first time stamp, and when the time difference is greater than a preset time threshold (such as an image recognition period), it is described that the first time stamp in the image data frame to be recognized is relatively small, that is, the first time stamp is added to the image data frame to be recognized by the image processing apparatus 102 at an earlier time, so that, in the current recognition process, the image data frame to be recognized is not the image data frame first generated by the image processing apparatus 102, but a historical image data frame to be recognized is repeatedly used, which belongs to a replay attack, and the image recognition apparatus 103 determines that the image data frame to be recognized is invalid. If the image recognition device 103 determines that the image data frame to be recognized is invalid, the image recognition device 103 discards the image data frame to be recognized, and does not recognize the image data frame to be recognized any more, and the image recognition device 103 may continue to determine whether the next image data frame is valid.

With reference to fig. 1 and fig. 2, in the process of identifying the image data frame to be identified by the image identification device 103, a part of the identification algorithm is executed by the algorithm scheduling processing module of the image identification device 103, and a part of the algorithm is executed by the AI calculation module of the image identification device 103. In the data protection method for image recognition provided by the embodiment of the application, in the process of image recognition, after the image processing device 102 generates an image data frame to be recognized, a time point (referred to as a first time stamp) for generating the image data frame to be recognized is added to the image data frame to be recognized and is sent to the image recognition device 103, and then the image recognition device 103 acquires the image data frame to be recognized and acquires a time point (referred to as a second time stamp) for acquiring the image data frame to be recognized by the image recognition device 103, if a time difference between the second time stamp and the first time stamp is greater than a preset time threshold, the image recognition device 103 determines that the image data frame to be recognized is invalid; if the time difference between the second timestamp and the first timestamp is less than or equal to the preset time threshold, the image recognition device 103 determines that the image data frame to be recognized is valid.

Further, the data protection method for image recognition provided by the embodiment of the application realizes the process of adding the first timestamp in the image data frame to be recognized, acquiring the second timestamp and realizing whether the image data frame to be recognized is effective or not by a software method, does not need to depend on special hardware, and reduces the complexity of data protection.

The above-mentioned scheme provided by the embodiment of the present application is introduced mainly from the perspective of interaction between network elements. It is to be understood that each network element, such as the image recognition device, the image processing device, etc., contains corresponding hardware structures and/or software modules for performing each function in order to realize the functions. Those of skill in the art will readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiment of the present application, the image recognition device, the image processing device, and the like may be divided into functional modules according to the above method examples, for example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, in the embodiment of the present application, the division of the module is schematic, and is only one logic function division, and there may be another division manner in actual implementation.

In the case of dividing each functional module according to each function, fig. 5 shows a schematic diagram of a possible structure of the image recognition apparatus involved in the above embodiment, and as shown in fig. 5, the image recognition apparatus 1000 may include: an obtaining module 1001, a comparing module 1002 and a determining module 1003. The obtaining module 1001 may be configured to support the image recognition apparatus 1000 to perform S105 and S106 in the above method embodiment; the comparing module 1002 may be configured to support the image recognition apparatus 1000 to perform S107 and S108 in the above method embodiment; the determination module 1003 may be configured to support the image recognition apparatus 1000 to execute S109 or S110 in the above method embodiment; optionally, as shown in fig. 5, the image recognition apparatus 1000 may further include a recognition module 1004, where the recognition module 1004 may be configured to support the image recognition apparatus 1000 to recognize the image data to be recognized under the condition that the image data frame to be recognized is valid; in the case where the image data frame to be recognized is invalid, the image data frame to be recognized is discarded. All relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and are not described herein again.

Fig. 6 shows a schematic diagram of a possible structure of the image recognition apparatus according to the above-described embodiment, in the case of an integrated unit. As shown in fig. 6, the image recognition apparatus 2000 may include: a processing module 2001 and a communication module 2002. The processing module 2001 may be used to control and manage the actions of the image recognition apparatus 2000, for example, the processing module 2001 may be used to support the image recognition apparatus 2000 to perform S105, S106, S107, S108, S109 or S110 in the above-described method embodiments, and/or other processes for the techniques described herein. The communication module 2002 may be used to support communication between the image recognition device 2000 and other network entities, for example, the communication module 2002 may be used to support communication between the image recognition device 2000 and the image processing device. Optionally, as shown in fig. 6, the image recognition apparatus 2000 may further include a storage module 2003 for storing program codes and data of the image recognition apparatus 2000.

The processing module 2001 may be a processor or a controller, and may include, for example, a Central Processing Unit (CPU), a general purpose processor, a Digital Signal Processor (DSP), an NPU, an application-specific integrated circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure of the embodiments of the application. The processor described above may also be a combination of computing functions, e.g., comprising one or more microprocessors, DSPs and microprocessors, and the like. The communication module 2002 may be a transceiver, a transceiver circuit, a communication interface, or the like. The storage module 2003 may be a memory.

When the processing module 2001 is a processor, the communication module 2002 is a transceiver, and the storage module 2003 is a memory, the processor, the transceiver, and the memory may be connected by a bus. The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc.

In the case of dividing each functional module by corresponding functions, fig. 7 shows a schematic diagram of a possible structure of the image processing apparatus according to the above embodiment, and as shown in fig. 7, the image processing apparatus 3000 may include: a generation module 3001, an acquisition module 3002, an addition module 3003 and a transmission module 3004. Wherein the generating module may be configured to support the image processing apparatus 3000 to execute S101 in the above method embodiment; the acquisition module 3002 may be used to support the image processing apparatus 3000 to execute S102 in the above-described method embodiment; the adding module 3003 may be used to support the image processing apparatus 3000 to execute S103 in the above-described method embodiment; the transmission module 3004 may be used to support the image processing apparatus 3000 to execute S104 in the above-described method embodiment. All relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and are not described herein again.

In the case of an integrated unit, fig. 8 shows a schematic diagram of a possible configuration of the image processing apparatus according to the above-described embodiment. As shown in fig. 8, the image processing apparatus 4000 may include: a processing module 4001 and a communication module 4002. The processing module 4001 may be used to control and manage the actions of the image processing apparatus 4000, and for example, the processing module 4001 may be used to support the image processing apparatus 4000 to execute S101, S102, and S103 in the above-described method embodiments. The communication module 4002 may be used to support communication of the image processing apparatus 4000 with other network entities, for example, the communication module 4002 may be used to support the image processing apparatus 4000 to execute S104 in the above-described method embodiment. Alternatively, as shown in fig. 8, the image processing apparatus 4000 may further include a memory module 4003 for storing program codes and data of the image processing apparatus 4000.

The processing module 4001 may be a processor or a controller, and may include, for example, a CPU, a general purpose processor, a DSP, an NPU, an ASIC, an FPGA, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure of the embodiments of the application. The processor described above may also be a combination of computing functions, e.g., comprising one or more microprocessors, DSPs and microprocessors, and the like. The communication module 4002 may be a transceiver, a transceiver circuit, a communication interface, or the like. The memory module 4003 may be a memory. When the processing module 4001 is a processor, the communication module 4002 is a transceiver, and the memory module 4003 is a memory, the processor, the transceiver, and the memory may be connected by a bus. The bus may be a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented using a software program, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produce, in whole or in part, the processes or functions described in the embodiments of the application. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., floppy disk, magnetic tape), an optical medium (e.g., Digital Video Disk (DVD)), or a semiconductor medium (e.g., Solid State Drive (SSD)), among others.

Through the above description of the embodiments, it is clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions. For the specific working processes of the system, the apparatus and the unit described above, reference may be made to the corresponding processes in the foregoing method embodiments, and details are not described here again.

In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or make a contribution to the prior art, or all or part of the technical solutions may be implemented in the form of a software product stored in a storage medium and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: flash memory, removable hard drive, read only memory, random access memory, magnetic or optical disk, and the like.

The above description is only a specific implementation of the embodiments of the present application, but the scope of the embodiments of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the embodiments of the present application should be covered by the scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

A data protection method for image recognition, comprising:

the method comprises the steps that an image recognition device obtains an image data frame to be recognized, wherein the image data frame to be recognized comprises a first time stamp, and the first time stamp is a time point of the image processing device generating the image data frame to be recognized;

the image recognition device acquires a second time stamp, wherein the second time stamp is a time point when the image recognition device acquires the image data frame to be recognized;

the image recognition device compares the second time stamp with the first time stamp to obtain a time difference between the second time stamp and the first time stamp;

and if the time difference is smaller than or equal to a preset time threshold, the image recognition device determines that the image data frame to be recognized is valid.
The method of claim 1, further comprising:

and if the time difference is larger than the preset time threshold, the image recognition device determines that the image data frame to be recognized is invalid.
The method according to claim 1 or 2, wherein after the image recognition device determines that the frame of image data to be recognized is valid, the method further comprises:

and the image recognition device recognizes the image data frame to be recognized.
The method according to any one of claims 1 to 3, wherein the image data frames to be identified comprise 2D image data frames and depth image data frames corresponding to the 2D image data frames.
The method of any of claims 1 to 4, wherein the first timestamp is included in the frame of depth image data.
A data protection method for image recognition, comprising:

the image processing device generates an image data frame to be identified;

the image processing device acquires a first timestamp, wherein the first timestamp is a time point when the image processing device generates the image data frame to be identified;

the image processing device adds the first time stamp in the image data frame to be identified;

the image processing device sends the image data frame to be identified added with the first timestamp to an image identification device.
The method of claim 6, wherein the image data frames to be identified comprise 2D image data frames and depth image data frames corresponding to the 2D image data frames.
The method according to claim 6 or 7, wherein the image processing apparatus adds the first timestamp in the image data frame to be identified, including:

the image processing device adds the first timestamp in the depth image data frame.
The image recognition device is characterized by comprising an acquisition module, a comparison module and a determination module;

the acquisition module is used for acquiring an image data frame to be identified, wherein the image data frame to be identified comprises a first timestamp, and the first timestamp is a time point of the image data frame to be identified generated by the image processing device;

the acquiring module is further configured to acquire a second timestamp, where the second timestamp is a time point when the image recognition device acquires the image data frame to be recognized;

the comparing module is configured to compare the second timestamp with the first timestamp to obtain a time difference between the second timestamp and the first timestamp;

the determining module is configured to determine that the image data frame to be identified is valid when the time difference is smaller than or equal to a preset time threshold.
The image recognition apparatus according to claim 9,

the determining module is further configured to determine that the image data frame to be identified is invalid when the time difference is greater than the preset time threshold.
The image recognition apparatus according to claim 9 or 10, further comprising a recognition module;

the identification module is used for identifying the image data frame to be identified after the determination module determines that the image data frame to be identified is valid.
The image recognition apparatus according to any one of claims 9 to 11, wherein the image data frame to be recognized includes a 2D image data frame and a depth image data frame corresponding to the 2D image data frame.
The image recognition apparatus according to any one of claims 9 to 12, wherein the first time stamp is included in the depth image data frame.
An image processing device is characterized by comprising a generation module, an acquisition module, an adding module and a sending module;

the generation module is used for generating an image data frame to be identified;

the acquisition module is used for acquiring a first timestamp, wherein the first timestamp is a time point of the generation module for generating the image data frame to be identified;

the adding module is used for adding the first timestamp acquired by the acquiring module in the image data frame to be identified;

and the sending module is used for sending the image data frame to be identified added with the first timestamp to an image identification device.
The apparatus according to claim 14, wherein the image data frame to be identified comprises a 2D image data frame and a depth image data frame corresponding to the 2D image data frame.
The image processing apparatus according to claim 14 or 15,

the adding module is specifically configured to add the first timestamp to the depth image data frame.
An image recognition apparatus comprising a processor and a memory coupled to the processor;

the memory is used for storing computer instructions, and when the image recognition device runs, the processor executes the computer instructions stored by the memory to enable the image recognition device to execute the data protection method for image recognition according to any one of claims 1 to 5.
An image processing apparatus comprising a processor and a memory coupled to the processor;

the memory is used for storing computer instructions, and when the image processing device runs, the processor executes the computer instructions stored by the memory to enable the image processing device to execute the data protection method for image recognition according to any one of claims 6 to 8.
An image recognition system characterized by comprising the image recognition apparatus of any one of claims 9 to 13 and the image processing apparatus of any one of claims 14 to 16; or the image recognition system comprises the image recognition apparatus of claim 17 and the image processing apparatus of claim 18.