CN113128537A

CN113128537A - Sample processing method and related device and storage medium

Info

Publication number: CN113128537A
Application number: CN201911419642.4A
Authority: CN
Inventors: 罗涛; 周敬华; 苏有泽
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-12-31
Filing date: 2019-12-31
Publication date: 2021-07-16

Abstract

The embodiment of the application provides a sample processing method and a related device. The sample processing method comprises the following steps: acquiring a first feature space sample; performing feature space transformation on the first feature space sample to obtain a second feature space sample, wherein the second feature space is different from the first feature space; performing perturbation injection of a stimulus component on the second eigenspace sample to obtain a perturbed-injected second eigenspace sample, wherein the perturbation injection does not change a masking state of the second eigenspace sample; performing a feature space inverse transform on the perturbed and injected second feature space samples to obtain inverse transformed first feature space samples; transmitting or storing the inverse transformed first feature space samples. The technical scheme of the embodiment of the application is beneficial to effectively resisting the attack related to the countersample.

Description

Sample processing method and related device and storage medium

Technical Field

The present application relates to the field of artificial intelligence technology, and in particular, to methods for processing artificial intelligence samples, related devices, and storage media.

Background

In recent years, Artificial Intelligence (AI) technology and applications have been rapidly developed, covering more and more industries. Meanwhile, the safety risk of the AI technology is gradually exposed, and the related safety problems are deeply influenced. For example, a security attacker makes a wrong decision by using an AI model vulnerability and elaborately constructing an input sample to deceive the AI model, or implants a backdoor in the AI model to implement a high-level attack, or interactively queries and constructs a similar model through a service interface of the AI model, or queries the service interface of the AI model to obtain privacy information in user data in a scene where a user provides training data, and the like.

Therefore, how to improve the attack resistance of the AI system and further improve the safety of the AI technology is a technical subject which needs to be researched urgently in the field.

Disclosure of Invention

The embodiment of the application provides a processing method, a related device, a storage medium and the like of some artificial intelligence samples.

A first aspect of an embodiment of the present application provides a sample processing method, which may include: obtaining first feature space samples (the manner in which the first feature space samples are obtained may be generated locally or received from other devices); performing feature space transformation on the first feature space sample to obtain a second feature space sample, wherein the second feature space is different from the first feature space; performing perturbation injection of a stimulus component on the second feature space sample to obtain a perturbed and injected second feature space sample, wherein the perturbation injection does not change the masking state of the second feature space sample; performing a feature space inverse transform on the perturbed and injected second feature space samples to obtain inverse transformed first feature space samples; transmitting or storing the inverse transformed first feature space samples.

Among them, the Feature space (Feature space) may also be called a Feature domain. It will be appreciated that the first feature space samples represent samples in the first feature space. The second eigenspace samples represent samples in a second eigenspace.

The first feature space may be, for example, one of the following feature spaces: frequency domain space, time domain space, luminance space, mode space, or texture space. The second feature space may be, for example, one of the following feature spaces: frequency domain space, time domain space, luminance space, mode space, or texture space.

The samples mentioned in the embodiments of the present application may be, for example, audio samples, video samples, picture samples, or other types of samples.

Assuming that the first feature space is a time domain space and the first feature space is a frequency domain space, performing feature space transformation on the first feature space sample to obtain a second feature space sample, specifically, performing fourier transform (time-frequency transform) on the sample in the time domain space to obtain a sample in the frequency domain space. Further, assuming that the first feature space is a frequency domain space and a time domain space, the feature space transformation is performed on the first feature space sample to obtain a second feature space sample, specifically, the inverse fourier transform (frequency-time transform) is performed on the sample in the frequency domain space to obtain the sample in the time domain space. Other samples are transformed between feature spaces, and so on.

The masking state of the second feature space sample is not changed, specifically, the masking state of each channel of the second feature space sample is not changed. For example, whether the masking channel is a masking channel both before and after the perturbation injection, i.e., the masking state of the channel remains masked both before and after the perturbation injection. The stimulus component in the masking channel is not perceptible to the user both before and after the perturbation injection.

The sample processing method of the above example can be applied to an AI client, an AI server, a third-party sample database, or a transit network device between the AI client and the AI server. When the sample processing method is applied to the AI client, the AI client may send the inverse-transformed first feature space sample to the AI server after obtaining the inverse-transformed first feature space sample. When the sample processing method is applied to the AI server, the AI server may store the inverse transformed first feature space samples after obtaining the inverse transformed first feature space samples (e.g., may store the inverse transformed first feature space samples in a training set or other location) so as to input the inverse transformed first feature space samples into the AI model when needed subsequently. When the sample processing method is applied to the third party sample database, the third party sample database may store the inverse-transformed first feature space samples, so as to provide the stored inverse-transformed first feature space samples to the AI server and the like when needed.

It can be seen that the sample processing method is a sample processing mechanism based on a masking effect, and as the masking state of the sample is not changed by the disturbance injection, the fragile structure of the countersample can be destroyed in the imperceptible secret state of the user, which is beneficial to realizing a good attack prevention effect without affecting the perception experience of the user. Practice shows that the sample processing mechanism based on the masking effect can effectively prevent evasion attack, stealing attack, bait attack and the like based on fighting samples. For AI input samples such as audio/picture/video samples, a sample processing mechanism based on masking effect is beneficial to not introducing perceptual damage of the samples, and further beneficial to improving multi-scene adaptability of the scheme.

In some possible embodiments, performing perturbation injection of a stimulus component on the second eigenspace sample to obtain a perturbation-injected second eigenspace sample may include:

determining N masking channels for the second feature space samples; and performing disturbance injection of stimulation components on M masking channels in the N masking channels, wherein the N is greater than or equal to the M, and the masking states of the M masking channels are not changed by the disturbance injection.

Wherein the M masking channels are selected, for example, randomly or non-randomly from the N masking channels. Under the condition of randomly selecting a masking channel for disturbance injection, the difficulty of analyzing the disturbance injection rule by an attacker is favorably improved, and the attack defense capability is favorably improved.

For example, determining N masking channels for the second feature space samples comprises: calculating masking thresholds for X channels of the second feature space samples (X being, for example, the total number of channels of the second feature space samples, the masking thresholds for different channels may be the same or different); selecting N masking channels (N is the total number of the masking channels of the second feature space sample, for example) from the X channels based on the masking thresholds and the stimulus components of the X channels, wherein the stimulus component of a masking channel i is smaller than or equal to the masking threshold of the masking channel i, and the masking channel i is any one of the N masking channels.

It can be seen that by calculating the masking threshold of each channel of the sample, flexible selection of the adjustment amplitude and the adjustment range is easy to realize under the constraint of the masking threshold, which is beneficial to increasing the difficulty of an attacker in conjecturing the disturbance injection rule.

In some possible embodiments, the stimulus component perturbation injected into the masking channel i of the M masking channels may be obtained based on a random algorithm, or the stimulus component perturbation injected into the masking channel i of the M masking channels may be obtained based on a preset sequence (the preset sequence is, for example, a digital watermark).

And the stimulus component disturbance injected into the masking channel i in the M masking channels is positive stimulus component disturbance or negative stimulus component disturbance. It will be appreciated that injecting a perturbation of a positive stimulus component causes the stimulus component to increase and injecting a perturbation of a negative stimulus component causes the stimulus component to decrease.

As can be seen. Under the condition that the disturbance is obtained based on the random algorithm, the difficulty of analyzing the disturbance injection rule by an attacker is further improved, and the attack defense capability is further improved. In addition, under the condition that the stimulation component disturbance is obtained by using the digital watermark, whether a sample is credible or not and whether the sample is tampered or not can be detected by using the digital watermark, and active prevention is further facilitated.

A second aspect of the embodiments of the present application provides a sample processing method, which may include: acquiring a first feature space sample; performing disturbance injection on the first feature space sample to obtain a disturbance-injected first feature space sample, wherein the disturbance injection does not change the masking state of the first feature space sample; transmitting or storing the perturbed injected first feature space samples.

The first feature space may be, for example, one of the following feature spaces: frequency domain space, time domain space, luminance space, mode space, or texture space.

The masking state of the first feature space sample is not changed, specifically, the masking state of each channel of the first feature space sample is not changed. For example, whether the masking channel is a masking channel both before and after the perturbation injection, i.e., the masking state of the channel remains masked both before and after the perturbation injection. The stimulus component in the masking channel is not perceptible to the user both before and after the perturbation injection.

The sample processing method of the embodiment of the application can be applied to an AI client, an AI server, a third-party sample database, or transit network equipment between the AI client and the AI server. When the sample processing method is applied to the AI client, the AI client may send the first feature space sample subjected to disturbance injection to the AI server after obtaining the first feature space sample subjected to disturbance injection. When the sample processing method is applied to the AI server, after obtaining the first feature space sample subjected to perturbation injection, the AI server may store the first feature space sample subjected to perturbation injection (for example, may store the first feature space sample into a training set or other locations), so that the first feature space sample subjected to perturbation injection may be input into the AI model when needed subsequently. When the sample processing method is applied to the third-party sample database, the third-party sample database may store the perturbed and injected first feature space sample, so as to provide the stored perturbed and injected first feature space sample to the AI server and the like when needed.

It can be seen that the sample processing method is a sample processing mechanism based on a masking effect, and as the masking state of the sample is not changed by the disturbance injection, the fragile structure of the countersample can be destroyed in the imperceptible secret state of the user, which is beneficial to realizing a good attack prevention effect without affecting the perception experience of the user. Practice shows that the sample processing mechanism based on the masking effect can effectively prevent evasion attack, stealing attack, bait attack and the like based on fighting samples. For AI input samples such as audio/picture/video samples, a sample processing mechanism based on masking effect is beneficial to not introducing perceptual damage of the samples, and further beneficial to improving the multi-scene universality of the scheme.

For example, perturbing the first eigenspace sample for a stimulus component to obtain a perturbed first eigenspace sample may include: determining N masking channels of the first feature space samples; and performing disturbance injection of stimulation components on M masking channels in the N masking channels, wherein N is greater than or equal to M, and the masking states of the M masking channels are not changed by the disturbance injection.

For example, determining N masking channels of the first feature space samples comprises: calculating masking thresholds for X channels of the first feature space samples (X being, for example, the total number of channels of the first feature space samples, the masking thresholds for different channels may be the same or different); selecting N masking channels (N is the total number of the masking channels of the first feature space sample, for example) from the X channels based on the masking thresholds and the stimulus components of the X channels, wherein the stimulus component of a masking channel i is smaller than or equal to the masking threshold of the masking channel i, and the masking channel i is any one of the N masking channels.

A third aspect of the embodiments of the present application provides a sample processing apparatus, including:

an obtaining unit is used for obtaining a first feature space sample.

And the feature space transformation unit is used for performing feature space transformation on the first feature space samples to obtain second feature space samples, wherein the second feature space is different from the first feature space.

And the disturbance injection unit is used for carrying out disturbance injection on the stimulation component on the second feature space sample to obtain a second feature space sample subjected to disturbance injection, and the disturbance injection does not change the masking state of the second feature space sample.

And the feature space inverse transformation unit is used for performing feature space inverse transformation on the second feature space samples subjected to disturbance injection to obtain inverse transformed first feature space samples.

An output unit for transmitting or storing the inverse transformed first feature space samples.

In some possible embodiments, the perturbation injection unit is specifically configured to determine N masking channels of the second feature space samples; and performing disturbance injection of stimulation components on M masking channels in the N masking channels, wherein N is greater than or equal to M, and the masking states of the M masking channels are not changed by the disturbance injection.

A fourth aspect of the present embodiments provides a sample processing apparatus, including:

an obtaining unit is used for obtaining a first feature space sample.

And the disturbance injection unit is used for carrying out disturbance injection on the stimulus component on the first feature space sample to obtain a disturbance-injected first feature space sample, and the disturbance injection does not change the masking state of the first feature space sample.

And the output unit is used for sending or storing the perturbed and injected first feature space samples.

In some possible embodiments, the perturbation injection unit is specifically configured to determine N masking channels of the first feature space sample; and performing disturbance injection of stimulation components on M masking channels in the N masking channels, wherein N is greater than or equal to M, and the masking states of the M masking channels are not changed by the disturbance injection.

A fifth aspect of the embodiments of the present application provides a sample processing apparatus, including:

a processor and a memory coupled to each other; the processor is configured to call the computer program stored in the memory to complete part or all of the steps of any one of the methods provided by the embodiments of the present application. For example, the processor may be configured to call a program stored in the memory to perform part or all of the steps of any one of the methods provided by the first or second aspects.

A sixth aspect of embodiments of the present application provides a computer-readable storage medium, including:

the computer readable storage medium stores a computer program, which when executed by a processor is capable of implementing some or all of the steps of any one of the methods provided by the embodiments of the present application. For example, the computer program, when executed by a processor, is capable of carrying out part or all of the steps of any one of the methods provided by the first or second aspects.

A sixth aspect of embodiments of the present application provides a computer program product, wherein when the computer program product is run on a computer device, the computer device is caused to perform part or all of the steps of any one of the methods provided by the embodiments of the present application. For example, when run on a computer device, cause the computer device to perform some or all of the steps of any one of the methods provided by the first or second aspects.

Drawings

Some drawings to which embodiments of the present application relate will be described below.

Fig. 1-a and fig. 1-B are schematic diagrams of network architectures of an AI system according to an embodiment of the present disclosure.

Fig. 2-a illustrates one possible implementation of an input reconstruction technique.

Fig. 2-B illustrates a schematic diagram of masking effects in an audio scene.

Fig. 3 is a schematic flowchart of a sample processing method according to an embodiment of the present disclosure.

Fig. 4 is a schematic flowchart of a sample processing method according to an embodiment of the present application.

Fig. 5-a is a flowchart illustrating a method for processing an audio sample according to an embodiment of the present application.

Fig. 5-B is a schematic diagram illustrating masking states and masking thresholds of audio samples in a frequency domain space according to an embodiment of the present application.

Fig. 5-C shows a schematic diagram of the effect of perturbation injection on audio samples based on masking effects.

Fig. 6-a to 6-C are schematic diagrams of the architecture of several improved AI systems provided by exemplary embodiments of the present application.

Fig. 7 is an architecture diagram of attack countermeasure detection in an intelligent home scenario according to an embodiment of the present application.

Fig. 8 is a schematic diagram of a process of injecting a digital watermark into an audio sample based on a masking effect according to an embodiment of the present application.

Fig. 9 is a schematic diagram of signal effects before and after digital watermark is injected into an audio sample based on masking effect according to an embodiment of the present application.

Fig. 10 is a schematic diagram of a system architecture for solving counterattack against a sample attack in an end-to-end internet communication scenario provided by an embodiment of the present application.

Fig. 11 is a schematic system architecture diagram of a model theft defense scenario according to an embodiment of the present application.

Fig. 12 is a schematic structural diagram of a sample processing device according to an embodiment of the present disclosure.

Fig. 13 is a schematic structural diagram of another sample processing device according to an embodiment of the present disclosure.

Fig. 14 is a schematic structural diagram of another sample processing device according to an embodiment of the present disclosure.

Detailed Description

Embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

Referring to fig. 1-a, a network architecture of an AI system is illustrated below, and the AI system illustrated in fig. 1-a may include an AI client (AI client) and an AI Server (AI Server). The AI Server is the service provider and the AI Custom is the service requester, which may be a normal user or an attacker. The AI client can submit the sample to the AI server, and the AI client can also request the AI server for AI inference services and other relevant AI services. The AI server can obtain samples from the AI client, train the AI model by using the samples, and provide AI inference service for the AI client.

Referring to fig. 1-B, fig. 1-B illustrates an internal functional module structure of an AI Server, which may include an AI model (which may provide an inference service), a Training Set (Training Set), and a Training module (which may provide an online Training service). The AI Server may perform data interaction with the AI store or other devices through an Application Programming Interface (API).

For example, the AI store may submit samples such as audio/video samples to the AI Server, and wait for the AI Server to return an inference decision result of its AI model for the submitted audio/video samples. An attacker may attack the AI client, the AI server, a transit network device between the AI client and the AI server, or a third-party database for storing samples.

The inventor of the present application has found that common AI security-related attack means include the following categories: evasion attack, bait attack, backdoor attack, theft attack, and the like.

Dodging attack-meaning that an attacker modifies the input to fool the AI model into making a false decision. The attack mainly occurs in the model reasoning phase; such as constructing confrontation samples in the digital or physical world, which are often difficult for humans to perceive, using small perturbations in the samples to escape detection of the AI model and mislead it to make erroneous inferential decisions.

And (3) bait attack, aiming at the scene of retraining the AI model, providing a training sample injected with specific noise pollution to the AI model by an attacker, so that the trained AI model can not work normally. The attack occurs in the model training phase; for example, an attacker adding a small amount of malicious data to a health database can cause significant changes to the dosage recommendations of the semi-patients.

And (4) backdoor attack, namely, similar to the traditional mode, an attacker implants a backdoor in the AI model, so that the judgment of the AI model on the special input is controlled by the attacker. The attack often occurs in the model generation or transmission phase; since AI models such as neural network models are mainly composed of parameters and network structures, and lack of interpretability, backdoors in AI models are more difficult to perceive.

Stealing attack, namely, an attacker can estimate parameters of the model or information in training data by combining input and output through multiple times of inquiry by using a service interface provided by the AI model, so that the model or private data is stolen. The stolen model not only causes the intellectual property damage, but also causes more serious consequences such as evasion attack and the like.

AI security technology is emerging, and currently, related research is mainly focused on academic circles, most of which is technology exploration aiming at specific business scenes, and a mature or unified scheme is still lacking. The inventor of the present application has summarized that AI security-related defense means may include the following categories: network distillation technology, countermeasure training technology, countermeasure sample detection technology, input reconstruction technology and differential privacy technology.

In the network distillation technology, a plurality of models are connected in series in a training stage, and the knowledge transfer and allocation can enhance the robustness of the models and partially offset the influence of micro disturbance on the models.

And (3) a countermeasure training technology, namely adding the generated countermeasure samples in a training set, enhancing the robustness of the model through retraining, and preventing specific types of countermeasure sample attacks in advance.

The countermeasure sample detection technology is characterized in that a detection module is added in the inference stage to judge whether the model input is a countermeasure sample, and the judgment rule can be sample distribution, sample historical input characteristics, characteristic difference between an input sample and a normal sample and the like.

The input reconstruction technology is to make some transformation to the input sample in the inference stage to counteract the attack of resisting sample, and the transformation needs not to affect the normal function of the model, and the common transformation modes include noise adding and removing, etc.

The differential privacy technology is characterized in that data or a training step is subjected to noise adding by using a differential privacy method in a training stage so as to protect user privacy information potentially contained in published data, such as a crowdsourcing privacy protection crowdsourcing technology and a homomorphic encryption technology which use random response.

The AI technology utilizes the structure and depth of a complex network to obtain strong learning capability, and simultaneously introduces the inexplicability of a model and the one-sidedness of learned knowledge. Based on the defects, an attacker discovers the vulnerability of the model by elaborately constructing a sample, and deceives the model to make a wrong decision; furthermore, the unexplainable nature of the model may also make it difficult to detect attacks against sample attacks, post-implantation door attacks, and the like.

Some possible attack defense schemes are discussed below.

Referring to fig. 2-a, fig. 2-a illustrates a possible implementation of an input reconstruction technique, in which a gaussian blurring algorithm, a median blurring algorithm, a color depth bit quantization algorithm, etc. are performed during filtering of input samples, so as to make certain changes to the input samples to counteract the attack of resisting the samples. Such techniques for reconstruction of input samples by filtering may affect the subjective perception of the user by causing too large a change to the samples.

The inventor of the present application has found that constructing a countermeasure sample is the biggest attack means facing AI security at present. The countersample is an AI model input designed by an attacker with the purpose of causing an erroneous output of the AI model. To date, the challenge sample has been the most widely studied in the image field. In the field of images, countermeasure samples can be constructed by subtle modifications of images, thereby causing misclassification, and are practical in the real world. The confrontational sample in the image domain is difficult for humans to distinguish, but the confrontational sample in the speech recognition domain is often perceptible. But also challenge samples generated by new generation methods in the speech field become difficult to perceive in recent years. Some researchers have used improvement methods to generate imperceptible challenge samples that can compete with image-like challenge samples. Therefore, audio, image and video are exposed to a more and more serious attack risk against samples, are often hard to be perceived by people and can be generated silently in a real environment (such as adding a disturbance of a specific sound by disturbing a certain sound source or modifying certain pixels during image and video acquisition), and therefore, a new defense method is urgently needed to be researched.

According to research, the countermeasure sample is generally a small perturbation added to the normal sample, so that the difference is not easy to be perceived by a human, and the AI model can be fooled. The generation of the confrontation sample is to change the input sample to make the existing model output wrong decision on the premise of smaller subjective difference, and generally adopt a gradient method to solve the gradient of the loss function in a sample space. The challenge sample itself is vulnerable from a certain point of view.

Based on the discovery, the inventor of the application proposes that the fragile structure of the confrontation sample can be damaged by actively injecting a disturbance component into the sample under the premise of not influencing the subjective perception and the model accuracy of the audio and video sample as much as possible. The masking effect in the visual and auditory perception provides a theoretical basis for the injection method, and is beneficial to ensuring that the audio and video perception difference before and after the injection is not perceived. The masking effect refers to information that a subjective subject cannot completely receive all stimulation due to the occurrence of a plurality of similar stimulation.

For example, in the example shown in FIG. 2-B, a single pure tone at a frequency of 0.47kHz may be heard with a minimum sound intensity of 8 dB; if a pure tone with the frequency of 0.24kHz and the sound intensity of 60dB is added at the same time, the previous pure tone with the frequency of 0.47kHz cannot be heard; if the two pure tones are to be heard simultaneously, the sound intensity of 0.47kHz is usually required to be raised to more than 29dB, and the threshold raising phenomenon comes from the masking of weak sounds by strong sounds of different frequencies. Sound signals are not only concealed in the frequency domain, but also masked in the time domain. Similar brightness masking and pattern masking also exist for image stimuli. With the masking effect, as long as a small perturbation is added on the channel of the frequency domain/time domain/mode of the weak stimulus, and it is satisfied that the added weak stimulus component is still below the masking threshold (i.e. the masking state of the channel is not changed after the added weak stimulus component), the added perturbation will not be perceived by human. In addition, since the amount of disturbance against the sample is carefully designed, for example, a disturbance of a specific amplitude is injected at a specific position and frequency. In contrast, if a sufficient amount of disturbance in accordance with the masking effect is injected over a wide range of the sensing channel, time domain/spatial domain, an effect of destroying the fragile structure of the antagonistic sample is expected.

Some defense schemes against the sample based on masking effects are discussed in detail below.

Referring to fig. 3, fig. 3 is a schematic flow chart of a sample processing method according to an embodiment of the present disclosure. A sample processing method may include:

301. a first feature space sample is acquired.

The first feature space sample may be locally generated or received from another device (e.g., an AI client, etc.).

302. And performing feature space transformation on the first feature space sample to obtain a second feature space sample.

Wherein the second feature space is different from the first feature space. The first feature space may be, for example, one of the following feature spaces: frequency domain space, time domain space, luminance space, mode space, or texture space. The second feature space may be, for example, one of the following feature spaces: frequency domain space, time domain space, luminance space, mode space, or texture space.

303. Perturb injection of a stimulus component (stimulus) is performed on the second eigenspace samples to obtain perturb-injected second eigenspace samples.

Wherein the perturbation injection does not change a masking state of the second feature space samples.

One sample in a feature space may be divided into a plurality of channels (channels), also called sensing channels, and the perception of a user to different channels may be different.

The masking state (masking state) of the second eigenspace sample specifically refers to the masking state of each channel of the second eigenspace sample. The masking state of a channel may be "masked" or "unmasked". A channel whose masking state is masking may be referred to as a masking channel (masking channel), where a masking channel is also referred to as a masked channel or a suppressed channel or an imperceptible channel. A channel whose masking state is unmasked may be referred to as an unmasked channel or an uninhibited channel or a perceptual channel.

304. And performing feature space inverse transformation on the perturbation injected second feature space samples to obtain inverse transformed first feature space samples.

305. Transmitting or storing the inverse transformed first feature space samples.

The sample processing method of the embodiment of the application can be applied to an AI client, an AI server, a third-party sample database, or transit network equipment between the AI client and the AI server. When the sample processing method is applied to the AI client, the AI client may send the inverse-transformed first feature space sample to the AI server after obtaining the inverse-transformed first feature space sample. When the sample processing method is applied to the AI server, the AI server may store the inverse transformed first feature space samples after obtaining the inverse transformed first feature space samples (e.g., may store the inverse transformed first feature space samples in a training set or other location) so as to input the inverse transformed first feature space samples into the AI model when needed subsequently. When the sample processing method is applied to the third party sample database, the third party sample database may store the inverse-transformed first feature space samples, so as to provide the stored inverse-transformed first feature space samples to the AI server and the like when needed.

For example, perturbing the second eigenspace sample for a stimulus component to obtain a perturbed injected second eigenspace sample may include: determining N masking channels for the second feature space samples; and performing disturbance injection of stimulation components on M masking channels in the N masking channels, wherein N is greater than or equal to M, and the masking states of the M masking channels are not changed by the disturbance injection.

Where N may be equal to 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 20, 32, 40, 64, 128, 256, 300, or other values, for example.

Where M may be equal to 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 19, 32, 40, 64, 128, 253, 300, or other values, for example.

Wherein the M masking channels are selected, for example, randomly or non-randomly from the N masking channels.

In addition, by calculating the masking threshold of each channel of the sample, the flexible selection of the adjustment amplitude and the adjustment range is easy to realize under the constraint of the masking threshold, and the difficulty of the attacker in conjecturing the disturbance injection rule is increased; furthermore, under the condition that the stimulation component disturbance is obtained by using the digital watermark, whether the sample is credible or not and whether the sample is tampered or not can be detected by using the digital watermark, and active prevention is further facilitated.

Referring to fig. 4, fig. 4 is a schematic flow chart of another sample processing method provided in the embodiment of the present application. The main difference between the fig. 4 example and the fig. 3 example is that the fig. 4 example does not require a transformation of the feature space of the sample. Another sample processing method may include:

401. a first feature space sample is acquired.

402. And performing stimulation component perturbation injection on the first feature space sample to obtain a perturbation-injected first feature space sample. The perturbation injection does not change the masking state of the first feature space samples.

403. Transmitting or storing the perturbed injected first feature space samples.

The masking state (masking state) of the first feature space sample specifically refers to a masking state of each channel of the first feature space sample. The masking state of a channel may be "masked" or "unmasked". A channel whose masking state is masking may be referred to as a masking channel (masking channel), which is also referred to as a masked channel or an imperceptible channel, and a channel whose masking state is unmasked may be referred to as an unmasked channel or a perceptible channel.

Referring to fig. 5-a, in fig. 5-a, the samples are taken as audio samples, and after transforming the audio samples from the time domain space to the frequency domain space, the perturbation injection of the stimulus component is performed.

In a specific implementation, the original audio samples are first transformed from the time domain space to the frequency domain space using a fourier transform. The masking state and masking threshold for each channel (here, frequency band) are then computed in the frequency domain space. For example, referring to fig. 5-B, the fluctuation curve in fig. 5-B represents the frequency spectrum (frequency vs sound pressure amplitude) of an audio sample, and the staircase shape broken line represents the masking threshold corresponding to each frequency band. The frequency positions of the spectral curve above the threshold polyline correspond to unmasked frequency bands (perceptual channels) which can be perceived audibly; conversely, frequency locations below the threshold polyline correspond to masked frequency bands (masking channels) that are not audibly perceivable because they are suppressed by other frequency bands.

One or more frequency bands (partial masking channels) to be suppressed are randomly selected, and the sound pressure amplitudes for these frequency bands are somehow injected with the disturbance such that the sound pressure amplitude after injection of the disturbance is still below the masking threshold. The way in which the perturbations are injected may for example follow a certain probability distribution, such as a gaussian distribution, a uniform distribution, etc.; the injected perturbation may be a perturbation having a particular physical significance, such as a digital watermark or the like.

And transforming the disturbance injected frequency domain space samples into time domain space samples through Fourier inverse transformation. The time-domain spatial sample obtained by the transformation is a sample that can resist attack. Referring to fig. 5-C, fig. 5-C illustrates an example of the effect of perturbation injection on audio samples based on masking effects. The audio samples are subjected to disturbance injection to change the masking state of the audio samples, so that the user cannot perceive the disturbance injection and the audio perception experience of the user is not influenced.

Of these, fig. 5-a shows frequency domain masking of audio samples, although masking of other feature spaces of audio samples may also be performed. For the image samples, frequency domain masking/texture masking, etc. of the image samples may be performed. For video samples, luminance masking/texture masking, etc. of the video samples may be performed. For a specific masking manner, a frequency domain masking mechanism of the audio samples may be referred to, and details are not repeated here.

In some specific products, the exemplary sample processing method can be implemented by using a software function device, which can be referred to as a Mask based Positive perturbation Injection (MPI) module, for example. The MPI module can be used for carrying out disturbance injection which is not sensed by a user on an input audio/video/image sample and then outputting the audio/video/image sample subjected to the disturbance injection.

Referring to fig. 6-a through 6-C, fig. 6-a through 6-C illustrate the architecture of several improved AI systems. The MPI module can be flexibly integrated in a plurality of selectable positions within the AI system.

For example, as shown in fig. 6-a, the MPI module may be integrated inside the AI store, i.e., the AI store implements the sample processing method exemplified above. Before submitting input samples to the AI Server, the MPI module in the AI store may perform forward perturbation injection on the samples.

For example, as shown in fig. 6-B, the MPI module may be integrated inside the AI Server, i.e. the AI Server implements the sample processing method of the above example. After the AI Server receives the input sample through the API, the MPI module performs forward perturbation injection on the sample, and then submits the perturbed and injected sample to a subsequent module for further processing (e.g., to an AI model, a training set, an online training module, etc.).

As another example, as shown in fig. 6-C, the MPI module may be integrated in a third-party sample database, that is, the sample processing method of the above example is implemented by the third-party sample database. After the AI store submits the input sample, the input sample may be redirected to a third-party sample database for forward disturbance injection, and then the disturbance-injected sample is submitted to the AI Server.

An example of an anti-attack detection scenario follows.

In an intelligent home scene, an AI server in the network provides AI service for the AI Client, and the AI server can return an inference result to guide the AI Client to implement operation. An attacker may tamper with the sample submitted by the AI Client to add something that the user cannot perceive and fool the AI server into making a false inference result.

Referring to fig. 7, for this scenario, the sample processing scheme (i.e., integrating the MPI module inside the AI store) is applied in the AI Client, and the AI Client injects forward disturbance containing a digital watermark into the audio or video sample before submitting the audio or video sample. Taking an audio sample as an example, a case of injecting a digital watermark in the audio sample based on a masking effect may be as shown in fig. 8. The audio time-domain sound pressure contrast before and after the digital watermark is injected into the audio sample can be as shown in fig. 9. Fig. 9 shows a signal diagram in which (a) represents an original audio sample, (b) represents an audio sample after being injected with a digital watermark, and (c) represents a digital watermark, and it can be seen from fig. 9 that the audio samples before and after being injected with the digital watermark do not change much and cannot be perceived and distinguished by a user.

After the AI server obtains the input samples, the digital watermarks in the input samples can be compared to verify the integrity of the samples. If the sample is not tampered and the identity is credible, submitting the sample to an AI model for decision making or submitting the sample to a training set for on-line training; otherwise, an alarm may be generated, for example, and the service denied.

A scenario for solving the attack of the sample is further illustrated below.

In an end-to-end internet communication scenario, an AI Client and an AI Server are two communication parties, and a certain relay communication Server (such as an instant messaging Server) provides an AI service and a communication relay for the AI Client. An attacker may tamper with the communication object, construct a countermeasure sample, etc. during the communication process in order to fool the AI Server into making erroneous inferences (such as disabling the desired speech/video enhancement functions). Typical scenarios such as Real Time Communication (RTC) systems may face AI security issues facing audio video samples.

For the application scenario, referring to fig. 10, the sample processing scheme (i.e., integrating the MPI module inside the AI Server) in the foregoing example may be applied to the AI Server, and after the AI Server receives the input audio/video sample, the MPI module injects a forward disturbance to the sample to destroy the malicious disturbance in the potentially antagonistic sample, and transmits the disturbance-injected sample to the AI Model or the training set for inference or online training. Since the disturbance injection operation has no influence on the audio and video perception of the user, the MPI module can not cause damage to audio and video communication and AI reasoning.

The following exemplifies a scenario of model theft defense.

Taking fig. 11 as an example, an AI Server providing an AI inference query function to the outside, an attacker trains a local AI model by using an inference result returned by the AI Server through multiple queries, and after multiple training, the local AI model can replace the inference function of the AI model in the online AI Server, thereby achieving the purpose of stealing the AI model.

For the application scenario, referring to fig. 11, the sample processing scheme (i.e., integrating the MPI module inside the AI Server) in the above example may be applied to the AI Server, and the MPI module may inject forward disturbance into the input sample before the AI model in the AI Server obtains the input sample, where the disturbance injection satisfies the non-perception and injection mode randomness, so that an attacker cannot obtain actual input of the online AI model, and the online AI model is difficult to estimate by local training of the attacker, thereby being beneficial to reducing the risk of model theft.

It can be seen that, in several application scenarios for example, the disturbance injection based on the masking effect is performed on the sample, so that the counterattack can be effectively resisted without affecting the perception of the sample by the user.

Some device embodiments are also provided below.

Referring to fig. 12, embodiments of the present application provide a sample processing device 1200, which may include:

an obtaining unit 1210 is configured to obtain a first feature space sample.

An eigenspace transformation unit 1220, configured to perform an eigenspace transformation on the first eigenspace sample to obtain a second eigenspace sample, where the second eigenspace is different from the first eigenspace.

A perturbation injection unit 1230, configured to perform perturbation injection of a stimulus component on the second feature space sample to obtain a perturbed-injected second feature space sample, where the perturbation injection does not change a masking state of the second feature space sample.

An eigenspace inverse transform unit 1240 for performing an eigenspace inverse transform on the perturbed injected second eigenspace samples to obtain inverse transformed first eigenspace samples.

An output unit 1250 for sending or storing the inverse transformed first feature space samples.

In some possible embodiments, the perturbation injection unit 1230 may be specifically configured to determine N masking channels of the second feature space samples; and performing disturbance injection of stimulation components on M masking channels in the N masking channels, wherein N is greater than or equal to M, and the masking states of the M masking channels are not changed by the disturbance injection.

It is understood that the functions of the functional blocks of the sample processing apparatus 1200 can be implemented based on the description of the embodiment shown in fig. 3, and are not described herein again.

Referring to fig. 13, an embodiment of the present application provides a sample processing device 1300, including:

an obtaining unit 1310 configured to obtain a first feature space sample.

A perturbation injection unit 1320, configured to perform perturbation injection on the first feature space sample to obtain a perturbation injected first feature space sample, where the perturbation injection does not change a masking state of the first feature space sample.

An output unit 1330 that sends or stores the perturbed injected first feature space samples.

In some possible embodiments, the perturbation injection unit 1320 is specifically configured to determine N masking channels of the first feature space sample; and performing disturbance injection of stimulation components on M masking channels in the N masking channels, wherein N is greater than or equal to M, and the masking states of the M masking channels are not changed by the disturbance injection.

It is understood that the functions of the functional blocks of the sample processing apparatus 1300 can be implemented based on the description of the embodiment shown in fig. 3, and are not described herein again.

Referring to fig. 14, an embodiment of the present application further provides a sample processing apparatus 1400, where the sample processing apparatus 1400 may implement part or all of the functions of the AI client, the AI server, or the third-party sample database provided in the foregoing embodiments, and the sample processing apparatus 1400 specifically includes: a processor 1410 and a memory 1420 coupled to each other.

The processor 1410 may be configured to call a computer program stored in the memory 1420 to complete part or all of the steps of any one of the methods that may be executed by the AI client, the AI server, or the third-party sample database or the MPI module in the embodiment of the present application.

Among them, the processor 1410 is also called a Central Processing Unit (CPU). The components of the image prediction apparatus in a particular application are coupled together, for example, by a bus system. The bus system may include a power bus, a control bus, a status signal bus, and the like, in addition to a data bus. For clarity of illustration, however, the various buses are designated in the figure as bus system 1430. The methods disclosed in the embodiments of the present application may be implemented in the processor 1410 or implemented by the processor 1410. Processor 1410 may be an integrated circuit chip having signal processing capabilities. In some implementations, some or all of the steps of the above methods may be performed by instructions in the form of hardware, integrated logic circuits, or software in the processor 1410. The processor 1410 may be a general purpose processor, a digital signal processor, an application specific integrated circuit, an off-the-shelf programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components. Processor 1410 may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present application. The general purpose processor 1410 may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in a hardware decoding processor, or in a combination of hardware and software modules in a decoding processor. The software modules may be located in ram, flash memory, rom, prom, eprom, or registers, among other storage media as is well known in the art. The storage medium is located in the memory 1420, and the processor 1410 may read the information in the memory 1420, and combine the hardware to complete some or all of the steps of the method.

Embodiments of the present application also provide a computer-readable storage medium storing a computer program, where the computer program is executed by hardware (for example, a processor, etc.) to perform some or all of the steps of any one of the methods performed by any device in the embodiments of the present application.

Embodiments of the present application also provide a computer program product comprising instructions for causing a computer device to perform some or all of the steps of any one of the above aspects when the computer program product runs on the computer device.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optics, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., compact disk), or a semiconductor medium (e.g., solid state disk), among others. In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the foregoing embodiments, the descriptions of the embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is merely a logical division, and the actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted or not executed. In addition, the indirect coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical or in other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage media may include, for example: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A method of sample processing, comprising:

acquiring a first feature space sample;

performing feature space transformation on the first feature space sample to obtain a second feature space sample, wherein the second feature space is different from the first feature space; performing perturbation injection of a stimulus component on the second feature space sample to obtain a perturbed and injected second feature space sample, wherein the perturbation injection does not change the masking state of the second feature space sample;

performing a feature space inverse transform on the perturbed and injected second feature space samples to obtain inverse transformed first feature space samples;

transmitting or storing the inverse transformed first feature space samples.

2. The method according to claim 1, wherein the perturbing the second eigenspace sample with a stimulus component to obtain a perturbed injected second eigenspace sample comprises:

3. The method of claim 2, wherein the determining the N masking channels for the second eigenspace samples comprises:

calculating masking thresholds for X channels of the second feature space samples; selecting N masking channels from the X channels based on the masking threshold and the stimulation component of the X channels, wherein the stimulation component of a masking channel i is smaller than or equal to the masking threshold of the masking channel i, and the masking channel i is any one masking channel in the N masking channels.

4. The method according to claim 2 or 3, wherein the M masking channels are randomly selected from the N masking channels.

5. The method according to any one of claims 2 to 4, wherein the stimulus component perturbation injected into the masking channel i of the M masking channels is obtained based on a random algorithm, or the stimulus component perturbation injected into the masking channel i of the M masking channels is obtained based on a preset sequence.

6. The method according to any one of claims 2 to 5, wherein the stimulus component perturbation injected into a masking channel i of the M masking channels is a positive stimulus component perturbation or a negative stimulus component perturbation.

7. The method according to any one of claims 1 to 6, wherein the second feature space is one of the following feature spaces: frequency domain space, time domain space, luminance space, mode space, or texture space.

8. A method of sample processing, comprising: acquiring a first feature space sample; performing disturbance injection on the first feature space sample to obtain a disturbance-injected first feature space sample, wherein the disturbance injection does not change the masking state of the first feature space sample;

transmitting or storing the perturbed injected first feature space samples.

9. The method of claim 8, wherein the perturbing the first eigenspace sample with a stimulus component to obtain a perturbed first eigenspace sample comprises:

determining N masking channels of the first feature space samples; and performing disturbance injection of stimulation components to M masking channels in the N masking channels, wherein the masking states of the M masking channels are not changed by the disturbance injection, the N and the M are positive integers, and the N is greater than or equal to the M.

10. A sample processing device, comprising: a processor and a memory coupled to each other; the processor is configured to invoke a computer program stored in the memory to perform the method of any one of claims 1 to 7 or 8 to 9.

11. A computer-readable storage medium, characterized in that,

the computer-readable storage medium stores a computer program which, when executed by hardware, implements the method of any of claims 1 to 7 or 8 to 9.