CN117332844A

CN117332844A - Challenge sample generation method, related device and storage medium

Info

Publication number: CN117332844A
Application number: CN202311459119.0A
Authority: CN
Inventors: 请求不公布姓名
Original assignee: Beijing Real AI Technology Co Ltd
Current assignee: Beijing Real AI Technology Co Ltd
Priority date: 2023-11-03
Filing date: 2023-11-03
Publication date: 2024-01-02

Abstract

The application relates to the technical field of artificial intelligence, and discloses an countermeasure sample generation method, a related device and a storage medium, wherein the method comprises the following steps: acquiring an initial challenge sample and a target attack category; constructing an objective function of a model to be trained according to the initial challenge sample and the objective attack category; determining an objective loss function of the model to be trained according to the objective function; and iteratively optimizing the target loss function until an optimal solution of the target loss function is searched, and outputting a countermeasure sample generated by the current model to be trained when the optimal solution is determined to be taken by the target loss function. Therefore, the success rate of the challenge attack under the physical transformation is higher, an attacker can attack the deep learning model with higher success rate in a robust way, and further, based on the challenge sample generation model meeting the actual application requirements, a large number of challenge samples meeting the requirements can be generated to evaluate the target detection model, so that the evaluation efficiency of the target detection model is higher and the model evaluation is more accurate.

Description

Challenge sample generation method, related device and storage medium

Technical Field

The present application relates to the field of artificial intelligence model training, and more particularly, to an countermeasure sample generation method, a related apparatus, and a storage medium.

Background

The existing model attack methods can be divided into two main categories according to the attack results: the first type is directed attack, and an attack sample can enable all recognition results to be target types when the model is predicted. The second type is non-directional attack, i.e. the attack sample can make the model be in error type when predicting all the recognition results.

The directional attack is mainly used for evaluating the target model based on deep learning, and the directional attack needs a great number of countermeasure samples to perform robustness evaluation on the target model.

At present, under the attack of a countermeasure sample, the target model cannot obtain the countermeasure sample which meets the requirements sufficiently, so that the evaluation efficiency of the target model is low and the model evaluation is not accurate enough.

Disclosure of Invention

The embodiment of the application provides a method, a related device and a storage medium for generating an countermeasure sample, which have higher success rate of countermeasure attack under physical transformation, and an attacker can attack a deep learning model with higher success rate in a robust way.

In a first aspect, embodiments of the present application provide an challenge sample generating method, including:

acquiring an initial challenge sample and a target attack category;

constructing an objective function of a model to be trained through a Gaussian process and prior distribution in Bayesian optimization according to the initial challenge sample and the objective attack category, wherein the model to be trained is used for generating a challenge sample aiming at the objective model;

determining an objective loss function of the model to be trained according to the objective function, wherein the objective loss function comprises an initial transformation function and a group of most harmful transformation functions corresponding to the initial transformation function, and the initial transformation function is obtained by sampling the objective function by using an expected improved acquisition function;

and iteratively optimizing the target loss function until an optimal solution of the target loss function is searched, and outputting a countermeasure sample generated by the current model to be trained when the optimal solution is determined to be taken by the target loss function.

In a second aspect, embodiments of the present application provide a model evaluation method, which generates a plurality of challenge samples using the challenge sample generation method described in the first aspect, and evaluates the robustness of a target detection model.

In a third aspect, the present application provides an challenge sample generating device, the device comprising:

the acquisition module is used for acquiring an initial challenge sample and a target attack category;

the construction module is used for constructing an objective function of a model to be trained through a Gaussian process and prior distribution in Bayesian optimization according to the initial challenge sample and the objective attack category, and the model to be trained is used for generating a challenge sample aiming at the objective model;

the determining module is used for determining an objective loss function of the model to be trained according to the objective function, wherein the objective loss function comprises an initial transformation function and a group of most harmful transformation functions corresponding to the initial transformation function, and the initial transformation function is obtained by sampling the objective function by using an expected improved acquisition function;

the generation module is used for iteratively optimizing the target loss function until an optimal solution of the target loss function is searched, and outputting a countermeasure sample generated by the current model to be trained when the optimal solution is determined to be taken by the target loss function.

In some embodiments of the present application, the building block is specifically configured to:

acquiring a mean function and a covariance function for representing prior distribution;

Representing a priori distribution by the mean function and the covariance function;

and constructing an objective function of the model to be trained through a Gaussian process in Bayesian optimization and the prior distribution.

In some embodiments of the present application, the determining module is specifically configured to:

sampling the objective function by using an expected improved acquisition function to obtain an initial transformation function;

searching a group of most harmful transformation functions near the initial transformation function to obtain a group of most harmful transformation functions corresponding to the initial transformation function;

and determining an objective loss function of the model to be trained according to the initial transformation function and the most harmful transformation function.

and iteratively solving the gradient of the initial transformation, moving along the direction opposite to the gradient, and searching a group of most harmful transformation functions nearby the initial transformation function to obtain a group of most harmful transformation functions corresponding to the initial transformation function.

determining a first loss function that minimizes classification of the countered sample into a target attack class according to the initial transform function and the most harmful transform function;

Acquiring the distance between the challenge sample and the original sample in the norm sense;

and determining a target loss function of the model to be trained according to the first loss function and the distance.

and substituting the initial transformation function and the most harmful transformation function into the second loss function to obtain a first loss function by taking the minimum value of the objective function as the second loss function.

acquiring initial distances between the challenge sample and the original sample in the norm sense;

acquiring a weight parameter of the initial distance;

and calculating the distance between the countermeasure sample and the original sample in the norm sense according to the weight parameter and the initial distance.

In a fourth aspect, embodiments of the present application provide a computing device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the challenge sample generation method of the first aspect and the model evaluation method of the second aspect when executing the computer program.

In a fifth aspect, embodiments of the present application provide a computer-readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the challenge sample generating method as described in the first aspect, the model evaluating method as described in the second aspect.

In a sixth aspect, embodiments of the present application provide a computer program product comprising instructions, the computer program product comprising program instructions which, when run on a computer or a processor, cause the computer or the processor to perform the challenge sample generating method according to the first aspect, the model evaluating method according to the second aspect.

In a seventh aspect, embodiments of the present application provide a chip system, including:

a communication interface for inputting and/or outputting information;

a processor configured to execute a computer-executable program to cause a device on which the chip system is mounted to execute the challenge sample generation method as described in the first aspect, the model evaluation method as described in the second aspect.

In one possible design, the above chip system further includes a memory for holding program instructions and data necessary for the terminal. The chip system may be formed of a chip or may include a chip and other discrete devices.

Compared with the prior art, the initial challenge sample and the target attack category are acquired in the embodiment of the application; constructing an objective function of a model to be trained through a Gaussian process and prior distribution in Bayesian optimization according to an initial challenge sample and the objective attack category; determining an objective loss function of the model to be trained according to the objective function; and iteratively optimizing a target loss function until an optimal solution of the target loss function is searched, and outputting a countermeasure sample generated by the current model to be trained when the optimal solution is determined to be taken by the target loss function. Therefore, in the embodiment of the application, since the objective loss function includes the initial transformation function and a group of most harmful transformation functions corresponding to the initial transformation function, the initial transformation function is obtained by sampling the objective function by using the expected improved acquisition function, the optimal solution is found by using the expected improved acquisition function, and the group of most harmful transformations is found based on bayesian optimization, so that the robustness of the challenge sample under physical transformation (such as density increase or decrease, rotation, etc.) is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a scenario of an challenge sample generation system provided by an embodiment of the present application;

FIG. 2 is a flow diagram of one embodiment of a challenge sample generation method provided in embodiments of the present application;

FIG. 3 is a schematic flow chart of determining an objective loss function of the model to be trained according to the objective function provided in the embodiment of the present application;

FIG. 4 is a schematic diagram of a challenge sample generating device according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a mobile phone according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a server according to an embodiment of the present application.

In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

In the following description, specific embodiments of the present application will be described with reference to steps and symbols performed by one or more computers, unless otherwise indicated. Thus, these steps and operations will be referred to in several instances as being performed by a computer, which as referred to herein performs operations that include processing units by the computer that represent electronic signals that represent data in a structured form. This operation transforms the data or maintains it in place in the computer's memory system, which may reconfigure or otherwise alter the computer's operation in a manner well known to those skilled in the art. The data structure maintained by the data is the physical location of the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the foregoing text and are not meant to be limiting, and one skilled in the art will recognize that various steps and operations described below may also be implemented in hardware.

The term "module" or "unit" as used herein may be considered a software object executing on the computing system. The various components, modules, engines, and services described herein may be viewed as implementing objects on the computing system. The apparatus and methods described herein are preferably implemented in software, but may of course also be implemented in hardware, all within the scope of the present application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein includes all or any element and all combination of one or more of the associated listed items.

The embodiment of the application provides an countermeasure sample generation method, a related device and a storage medium.

Referring to fig. 1, fig. 1 is a schematic view of a scenario of an challenge sample generating system provided in an embodiment of the present application, where the challenge sample generating system may include a computing device 100, and the computing device 100 is connected through a network, and a challenge sample generating apparatus is integrated in the computing device 100. In the present embodiment, the computing device 100 may be a terminal device or a server.

In the embodiment of the present application, in the case where the computing device 100 is a server, the server may be a stand-alone server, or may be a server network or a server cluster formed by servers, for example, a server described in the embodiment of the present application includes, but is not limited to, a computer, a network host, a single network server, a plurality of network server sets, or a cloud server formed by a plurality of servers. Wherein the Cloud server is composed of a large number of computers or web servers based on Cloud Computing (Cloud Computing). In embodiments of the present application, communication between the server and the client may be achieved by any communication means, including, but not limited to, mobile communication based on third generation partnership project (3rd Generation Partnership Project,3GPP), long term evolution (Long Term Evolution, LTE), worldwide interoperability for microwave access (Worldwide Interoperability for Microwave Access, wiMAX), or computer network communication based on the TCP/IP protocol family (TCP/IP Protocol Suite, TCP/IP), user datagram protocol (User Datagram Protocol, UDP), etc.

It will be appreciated that when the computing device 100 used in embodiments of the present application is a terminal device, the terminal device may be a device that includes both receive hardware and transmit hardware, i.e., a device having receive and transmit hardware capable of performing bi-directional communications over a bi-directional communication link. Such a terminal device may include: a cellular or other communication device having a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific computing device 100 may be a desktop terminal or a mobile terminal, and the computing device 100 may be one of a mobile phone, a tablet computer, a notebook computer, and the like.

The terminal device according to the embodiments of the present application may also be a device that provides voice and/or data connectivity to a user, a handheld device with wireless connection functionality, or other processing device connected to a wireless modem. Such as mobile telephones (or "cellular" telephones) and computers with mobile terminals, which can be portable, pocket, hand-held, computer-built-in or car-mounted mobile devices, for example, which exchange voice and/or data with radio access networks. For example, personal communication services (English full name: personal Communication Service, english short name: PCS) telephones, cordless telephones, session Initiation Protocol (SIP) phones, wireless local loop (Wireless Local Loop, english short name: WLL) stations, personal digital assistants (English full name: personal Digital Assistant, english short name: PDA) and the like.

Those skilled in the art will appreciate that the application environment illustrated in fig. 1 is merely an application scenario of the present application and is not limited to the application scenario of the present application, and that other application environments may also include more or fewer computing devices than those illustrated in fig. 1, or a network connection of computing devices, such as only 1 computing device illustrated in fig. 1, and that the challenge sample generation system may also include one or more other computing devices, or/and one or more other computing devices that are network connected to computing device 100, and is not limited in this regard.

In addition, as shown in fig. 1, the challenge sample generation system may further comprise a memory 300 for storing data, such as a sample picture, challenge sample data.

It should be noted that, the schematic view of the scenario of the challenge sample generating system shown in fig. 1 is only an example, and the challenge sample generating system and scenario described in the embodiments of the present application are for more clearly describing the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application, and those skilled in the art can know that, with the evolution of the challenge sample generating system and the appearance of a new service scenario, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

The solution provided in the embodiments of the present application relates to artificial intelligence (Artificial Intelligence, AI), computer Vision (CV), machine Learning (ML), and the like, and is specifically described by the following embodiments:

the AI is a theory, a method, a technology and an application system which simulate, extend and extend human intelligence by using a digital computer or a machine controlled by the digital computer, sense environment, acquire knowledge and acquire an optimal result by using the knowledge. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.

AI technology is a comprehensive discipline, and relates to a wide range of technologies, both hardware and software. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

At present, AI technology is widely used in various fields such as man-machine interaction and safety protection. Because the input form of the machine learning algorithm is a numerical vector, an attacker can make misjudgment of the machine learning model by designing a specific numerical vector, which is the process of resisting the attack.

Attack methods can be divided into two main categories according to the attack results: the first type is directed attack, and an attack sample can enable all recognition results to be target types when the model is predicted. The second type is non-directional attack, i.e. the attack sample can make the model be in error type when predicting all the recognition results.

Directional attacks are more challenging than the one, which requires the detector to identify the target as a specific class of attack, which is more targeted. Object detection is a multi-task learning problem, and multiple tasks include foreground-background discrimination, multiple object discrimination, and object location recognition, object detection has demonstrated its ability to solve practical problems. Current challenge sample research for target detection scenarios, although numerous, is still mostly focused on the non-directional field.

In one related art, the challenge sample is generated directly. In particular, optimizing the loss function directly, resulting in the challenge samples, is used to mislead the predictions of the model, and the challenge samples resulting from this approach lack robustness in misleading the model, which limits the usefulness of these challenge samples because they can only work in the digital world, but cannot work in the real world, and cannot effectively transform the physical world.

According to the embodiment of the application, the success rate of the challenge attack under the physical transformation is higher, an attacker can attack the deep learning model with higher success rate in a robust way, further, based on the challenge sample generation model meeting the actual application requirements, a large number of challenge samples meeting the requirements can be generated to evaluate the target detection model, so that the evaluation efficiency of the target detection model is higher and the model evaluation is more accurate.

The following describes in detail specific embodiments.

In the present embodiment, description will be made from the viewpoint of an challenge sample generation method, which may be integrated in the computing device 100 in particular.

The present application provides an challenge sample generation method including: acquiring an initial challenge sample and a target attack category; constructing an objective function of a model to be trained through a Gaussian process and prior distribution in Bayesian optimization according to the initial challenge sample and the objective attack category, wherein the model to be trained is used for generating a challenge sample aiming at the objective model; determining an objective loss function of the model to be trained according to the objective function, wherein the objective loss function comprises an initial transformation function and a group of most harmful transformation functions corresponding to the initial transformation function, and the initial transformation function is obtained by sampling the objective function by using an expected improved acquisition function; and iteratively optimizing the target loss function until an optimal solution of the target loss function is searched, and outputting a countermeasure sample generated by the current model to be trained when the optimal solution is determined to be taken by the target loss function.

Referring to fig. 2, a flowchart of an embodiment of a method for generating a challenge sample according to an embodiment of the present application includes the following steps 201 to 204:

201. an initial challenge sample and a target attack class are obtained.

The attack category may be a category in which the target object of the model is identified as a directional attack, for example, a red light category is an attack of a green light category, and the green light category is a selected attack category.

Specifically, the attack category can be selected according to the sample picture category and the actual requirement during attack training. For example: when the method is applied to a video monitoring system, in order to interfere with monitoring, when the sample picture is a person type, the attack type can be a cat type.

Wherein the sample picture may be represented by a vector of pixel integration. In particular, the sample pictures may be from video frames, such as: the sample picture may be a portrait picture in video surveillance, and the specific source of the sample picture is not limited herein.

The challenge sample is proposed by Christian Szegedy et al, which refers to an input sample in the dataset formed by deliberately adding fine interference, resulting in the model giving an erroneous output with high confidence, and in a regularized context, reducing the error rate of the original independent co-distributed test set by challenge training—training the network on the challenge-perturbed training set samples.

After the challenge sample is added to the sample picture, the neural network identifies the sample picture as an attack category, and the attack category and the category to which the sample picture belongs are different categories, i.e., adding the challenge sample to the sample picture can cause the neural network to generate erroneous judgment.

In particular, the challenge sample may be added to each pixel unit of the sample picture, or may be added to several pixel units of the sample picture, where the specific manner of adding the challenge sample is not limited. In addition, the initial challenge sample is not a final challenge sample, and the initial challenge sample may be generated based on a solid-color picture, and is not limited herein.

202. And constructing an objective function of the model to be trained through a Gaussian process and prior distribution in Bayesian optimization according to the initial challenge sample and the objective attack category.

The model to be trained is used for generating a challenge sample aiming at the target model, namely the challenge sample generated by the model to be trained can be used for carrying out attack training on the target model.

The bayesian optimization algorithm (BO) is a sequence model optimization method for global optimization. This algorithm is optimized by constructing a Gaussian Process (GP) of the objective function over a given input space, in combination with Bayes' formula. In brief, in each iteration, the optimal result observed is used to adjust the search direction of the next step and select better candidate points, so as to gradually approach the global optimal solution.

The prior distribution (prior distribution) is translated into a "pre-test distribution" and a "pre-event distribution". Is one of the probability distributions. As opposed to a "posterior distribution". Independent of test results, or independent of random sampling, reflects the distribution obtained from knowledge of other relevant parameters θ prior to statistical testing. Bayesian school believes that one will have some knowledge of θ before looking to obtain a sample. This is known as prior to experimental observation. Thus, bayesian pie considers θ to be a random variable. The distribution function of θ is denoted as H (θ), the density function of θ is denoted as H (θ), and the distribution function and the prior density function are respectively referred to as a priori distribution. Prior to performing bayesian statistical inference, a probability distribution can be established based on known prior knowledge or assumptions, which distribution is referred to as a prior distribution, which represents the degree of knowledge of the probability distribution of the parameter prior to performing the experiment or collecting the data.

203. And determining an objective loss function of the model to be trained according to the objective function.

The target loss function comprises an initial transformation function and a group of most harmful transformation functions corresponding to the initial transformation function, and the initial transformation function is obtained by sampling the target function by using an expected improved acquisition function.

204. And iteratively optimizing the target loss function until an optimal solution of the target loss function is searched, and outputting a countermeasure sample generated by the current model to be trained when the optimal solution is determined to be taken by the target loss function.

Compared with the fact that physical transformation is randomly selected when an countermeasure sample is generated in the prior art, the fact that importance degrees of different physical transformations are different is ignored, so that robustness to physical world transformation is lacking, and an initial countermeasure sample and a target attack class are acquired in the embodiment of the application; constructing an objective function of a model to be trained through a Gaussian process and prior distribution in Bayesian optimization according to an initial challenge sample and the objective attack category; determining an objective loss function of the model to be trained according to the objective function; and iteratively optimizing the target loss function until an optimal solution of the target loss function is searched, and outputting a countermeasure sample generated by the current model to be trained when the optimal solution is determined to be taken by the target loss function. Therefore, in the embodiment of the application, since the objective loss function includes the initial transformation function and a group of most harmful transformation functions corresponding to the initial transformation function, the initial transformation function is obtained by sampling the objective function by using the expected improved acquisition function, the optimal solution is found by using the expected improved acquisition function, and the group of most harmful transformations is found based on bayesian optimization, so that the robustness of the challenge sample under physical transformation (such as density increase or decrease, rotation, etc.) is improved.

In some embodiments of the present application, constructing the objective function of the model to be trained according to the initial challenge sample and the objective attack category through a gaussian process and a priori distribution in bayesian optimization in step 202 may include: acquiring a mean function and a covariance function for representing prior distribution; representing a priori distribution by the mean function and the covariance function; and constructing an objective function of the model to be trained through a Gaussian process in Bayesian optimization and the prior distribution.

Bayesian optimization is an efficient method of solving the global optimization problem, comprising two key components: one is a statistical surrogate, such as a gaussian process or bayesian neural network, that models an unknown object; the other is an acquisition function that is maximized by weighing the utilization and exploration to recommend the next sample point. In the embodiment of the application, a Gaussian process is selected as a statistical substitute, which provides a Bayesian posterior probability distribution describing potential values of the objective function at any candidate point.

Bayesian optimization can find more important physical transformations than direct gradient descent. By gaussian processes and expected improved acquisition functions, one can find physical transformations that make the predictions higher and the model more uncertain, balancing exploration and development. So that more important physical transformation can be found, and the robustness is improved

Specifically, by a Gaussian process in Bayesian optimizationAnd a priori distribution->Modeling objective function C _Mis (t(x _adv ) Y), using the constant mean function μ0 and the Matern kernel as covariance function Σ0, resulting inWherein D is _n Is the observation dataset, f is the modeling of the unknown objective function, x _adv Is a challenge sample, y is an attack target, C _Mis Is a loss function, such as a cross entropy function CE.

The predicted value of test point x, which will be used in the subsequent expected modified acquisition function, depends on the observed data:

wherein the mean value mu _n (x)＝∑ ₀ (x，D _n )∑ ₀ (D _n ，D _n ) ^-1 (f(D _n )-μ ₀ (D _n ))+μ ₀ (x)；

Variance sigma _n (x，x)＝∑ ₀ (x，x)-∑ _n (x，D _n )∑ _n (D _n ，D _n ) ^-1 ∑ _n (D _n ，x)。

In some embodiments of the present application, as shown in fig. 3, the determining, in step 203, the objective loss function of the model to be trained according to the objective function may further include the following steps 301 to 303:

301. the objective function is sampled using the expected modified acquisition function to obtain an initial transformation function.

In one particular example, the initial transformation may be found using an expected modified acquisition function. Wherein the method comprises the steps ofRepresenting the expectation of posterior distribution given the value of f at x1,..>Is the best value observed. By anticipating improved acquisition functions: argmax acquisition function->For objective function C _Mis (t(x _adv ) Y) samples to find the initial transformation function, typically the first sample point (maximizing the acquisition function).

The acquisition function is used for generating observation points to be evaluated next time, and has the main effects of balancing exploration (exploration) and utilization (exploration), so that the acquisition function can not only accelerate convergence by utilizing the previous observation value, but also explore places with strong uncertainty in a decision space, and the local optimization is avoided.

302. Searching a group of most harmful transformation functions near the initial transformation function to obtain a group of most harmful transformation functions corresponding to the initial transformation function.

The searching a group of most harmful transformation functions near the initial transformation function to obtain a group of most harmful transformation functions corresponding to the initial transformation function comprises the following steps: and iteratively solving the gradient of the initial transformation function, moving along the direction opposite to the gradient, and searching a group of most harmful transformation functions nearby the initial transformation function to obtain a group of most harmful transformation functions corresponding to the initial transformation function.

In one specific example, a set of most detrimental transformation functions near the current initial transformation function is found by iteratively solving for the gradient and moving in the opposite direction of the gradientWhere argmax represents a parameter for maximizing, T is a transformation distribution in the real world, T () is a transformation function selected according to the distribution T, and T is a set of transformation functions selected according to the distribution T.

In the above steps (1) and (2), the objective function C is modeled by a gaussian process and a priori distribution in bayesian optimization _Mis (t(x _adv ) Y), finding the initial transformation function by anticipating the improved acquisition function, finding the set of most harmful transformations around the current initial transformation by gradient descentWhere T is the real world transformation distribution, T () is a transformation function selected from the distribution T, and T is a set of transformation functions selected from the distribution T.

303. And determining an objective loss function of the model to be trained according to the initial transformation function and the most harmful transformation function.

Wherein, the determining the target loss function of the model to be trained according to the initial transformation function and the most harmful transformation function may include: determining a first loss function that minimizes classification of the countered sample into a target attack class according to the initial transform function and the most harmful transform function; acquiring the distance between the challenge sample and the original sample in the norm sense; and determining a target loss function of the model to be trained according to the first loss function and the distance.

Specifically, determining a first loss function that minimizes classification of the countered samples into target attack categories based on the initial transform function and the most harmful transform function may include: and substituting the initial transformation function and the most harmful transformation function into the second loss function to obtain a first loss function by taking the minimum value of the objective function as the second loss function.

Wherein, the obtaining the distance between the challenge sample and the original sample in the norm sense can comprise: acquiring initial distances between the challenge sample and the original sample in the norm sense; acquiring a weight parameter of the initial distance; and calculating the distance between the countermeasure sample and the original sample in the norm sense according to the weight parameter and the initial distance.

In a specific example, the objective loss function of the model to be trained is determined according to the initial transformation function and the most harmful transformation function, and may be determined as follows:

(1) Setting the loss function term as

(2) Iterative optimization of challenge objective function to

The challenge objective function in this application isWherein C is _Reg Is a constraint term, x is the original sample, and β is a weight parameter. First item->Representative minimized challenge sample x _adv The loss function classified into the target class y (minimizing the loss, modeling, generating the challenge sample, and minimizing the loss function of the challenge sample x classified into y) is used to derive the first term specific expression through bayesian optimization in the embodiment of the present application to enhance the robustness of the generated challenge sample under the physical world transformation. In particular implementations, we also need to consider constraint terms to ensure that the challenge samples generated meet the respective constraints. Second term beta.C _Reg (x _adv X) represents the distance in the sense of the norm of the challenge sample and the original sample, wherein the norm is an enhanced distance concept used in the challenge sample to measure the magnitude of the disturbance.

The embodiment of the application provides a scheme for improving the robustness against attacks based on Bayesian optimization, and a method for improving the robustness against attacks based on Bayesian optimization aims at finding a group of most harmful transformation functions through Bayesian optimization and improving the robustness of an antagonism sample under physical transformation (such as density increase and decrease, rotation and the like). Compared with the general attack resistance, the method for improving the robustness of the attack resistance based on Bayesian optimization has higher success rate under physical transformation, and by implementing the method, an attacker can attack the deep learning model with higher success rate.

The embodiment of the application also provides a model evaluation method, which comprises the step of evaluating the robustness of the target model by utilizing a plurality of countermeasure samples generated by the countermeasure sample generation method.

According to the method and the device for evaluating the target detection model, through the trained model, a large number of countermeasure samples meeting requirements can be generated to evaluate the target detection model, so that the evaluation efficiency of the target detection model is higher, the model evaluation is more accurate, and the robustness of the model is improved.

In order to facilitate better implementation of the challenge sample generation method provided by the embodiment of the application, the embodiment of the application also provides a device based on the challenge sample generation method. Where the meaning of nouns is the same as in the challenge sample generation method described above, specific implementation details may be referred to in the description of the challenge sample generation method embodiments.

The challenge sample generating device in the embodiment of the present application has a function of realizing a challenge sample generating method corresponding to that provided in the above-described embodiment. The functions may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above, which may be software and/or hardware.

Referring to fig. 4, fig. 4 is a schematic structural diagram of an challenge sample generating device provided in an embodiment of the present application, where the challenge sample generating device may be applied to a computing device in a scenario where challenge sample generation is required, and specifically, the challenge sample generating device 400 may include an obtaining module 401, a constructing module 402, a determining module 403, and a generating module 404, which are specifically as follows:

an obtaining module 401, configured to obtain an initial challenge sample and a target attack class;

A building module 402, configured to build an objective function of a model to be trained according to the initial challenge sample and the objective attack category through a gaussian process and prior distribution in bayesian optimization, where the model to be trained is used to generate a challenge sample for an objective model;

a determining module 403, configured to determine, according to the objective function, an objective loss function of the model to be trained, where the objective loss function includes an initial transform function and a set of most harmful transform functions corresponding to the initial transform function, and the initial transform function is obtained by sampling the objective function using an expected improved acquisition function;

and the generating module 404 is configured to iteratively optimize the objective loss function until an optimal solution of the objective loss function is searched, and when the objective loss function is determined to take the optimal solution, output a challenge sample generated by the current model to be trained.

In some embodiments of the present application, the building block 402 is specifically configured to:

In some embodiments of the present application, the determining module 403 is specifically configured to:

acquiring a weight parameter of the initial distance;

Compared with the prior art, the acquisition module 401 acquires an initial challenge sample and a target attack class in the embodiment of the present application; the construction module 402 constructs an objective function of the model to be trained through a Gaussian process and prior distribution in Bayesian optimization according to the initial challenge sample and the objective attack category; the determining module 403 determines a target loss function of the model to be trained according to the target function; the generating module 404 iteratively optimizes the objective loss function until an optimal solution of the objective loss function is searched, and outputs a countermeasure sample generated by the current model to be trained when the objective loss function is determined to take the optimal solution. Therefore, in the embodiment of the application, since the objective loss function includes the initial transformation function and a group of most harmful transformation functions corresponding to the initial transformation function, the initial transformation function is obtained by sampling the objective function by using the expected improved acquisition function, the optimal solution is found by using the expected improved acquisition function, and the group of most harmful transformations is found based on bayesian optimization, so that the robustness of the challenge sample under physical transformation (such as density increase or decrease, rotation, etc.) is improved.

In order to facilitate better implementation of the challenge sample generating device provided in the embodiments of the present application, the embodiments of the present application further provide a device for performing model evaluation by using the challenge sample generated in the embodiments of the challenge sample generating device. Where the meaning of nouns is the same as in the challenge sample generation method described above, specific implementation details may be referred to in the description of the method embodiments. The apparatus generates a plurality of challenge samples using the challenge sample generating means as described above, and evaluates the robustness of the target detection model.

Specifically, the present application also provides a model evaluation device that evaluates the robustness of a target model using a plurality of challenge samples generated by the challenge sample generation device according to any one of the above embodiments.

By the model evaluation device, the target model is evaluated by adopting the plurality of challenge samples generated in the challenge sample generation method embodiment, and the challenge samples are the challenge samples with robustness for the physical world transformation, so that the success rate and the efficiency of attack under the physical world transformation can be enhanced, and the deep learning model can be successfully and efficiently attacked in the physical world, thereby improving the evaluation efficiency and the evaluation effectiveness for the target model.

The challenge-sample generating device in the embodiment of the present application is described above from the point of view of the modularized functional entity, and the challenge-sample generating device in the embodiment of the present application is described below from the point of view of hardware processing, respectively.

In the embodiment of the present application, when the computing device is a terminal device, as shown in fig. 5, only a portion related to the embodiment of the present application is shown for convenience of explanation, and specific technical details are not disclosed, please refer to a method portion of the embodiment of the present application. The terminal device may be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), a Point of Sales (POS), a vehicle-mounted computer, and the like, taking the terminal device as an example of the mobile phone:

fig. 5 is a block diagram showing a part of the structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Referring to fig. 5, the mobile phone includes: radio Frequency (RF) circuitry 1010, memory 1020, input unit 1030, display unit 1040, sensor 1050, audio circuitry 1060, wireless fidelity (wireless fidelity, wiFi) module 1070, processor 1080, and power source 1090. Those skilled in the art will appreciate that the handset configuration shown in fig. 5 is not limiting of the handset and may include more or fewer components than shown, or may combine certain components, or may be arranged in a different arrangement of components.

The following describes the components of the mobile phone in detail with reference to fig. 5:

the RF circuit 1010 may be used for receiving and transmitting signals during a message or a call, and particularly, after receiving downlink information of a base station, the signal is processed by the processor 1080; in addition, the data of the design uplink is sent to the base station. Generally, RF circuitry 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a Low noise amplifier (Low NoiseAmplifier, LNA), a duplexer, and the like. In addition, the RF circuitry 1010 may also communicate with networks and other devices via wireless communications. The wireless communications may use any communication standard or protocol including, but not limited to, global system for mobile communications (GlobalSystem of Mobile communication, GSM), general Packet radio service (General Packet RadioService, GPRS), code division multiple access (Code Division Multiple Access, CDMA), wideband code division multiple access (Wideband Code Division Multiple Access, WCDMA), long term evolution (Long Term Evolution, LTE), email, short message service (Short Messaging Service, SMS), and the like.

The memory 1020 may be used to store software programs and modules that the processor 1080 performs various functional applications and data processing of the handset by executing the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. In addition, memory 1020 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state memory device.

The input unit 1030 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the handset. In particular, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also referred to as a touch screen, may collect touch operations thereon or thereabout by a user (e.g., operations of the user on the touch panel 1031 or thereabout using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a predetermined program. Alternatively, the touch panel 1031 may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch azimuth of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 1080 and can receive commands from the processor 1080 and execute them. Further, the touch panel 1031 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. The input unit 1030 may include other input devices 1032 in addition to the touch panel 1031. In particular, other input devices 1032 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a track ball, a mouse, a joystick, etc.

The display unit 1040 may be used to display information input by a user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041, and alternatively, the display panel 1041 may be configured in the form of a Liquid crystal display (Liquid CrystalDisplay, LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch panel 1031 may overlay the display panel 1041, and when the touch panel 1031 detects a touch operation thereon or thereabout, the touch panel is transferred to the processor 1080 to determine a type of touch event, and then the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in fig. 5, the touch panel 1031 and the display panel 1041 are two independent components for implementing the input and output functions of the mobile phone, in some embodiments, the touch panel 1031 and the display panel 1041 may be integrated to implement the input and output functions of the mobile phone.

The handset may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of ambient light, and the proximity sensor may turn off the display panel 1041 and/or the backlight when the mobile phone moves to the ear. As one of the motion sensors, the accelerometer sensor can detect the acceleration in all directions (generally three axes), and can detect the gravity and direction when stationary, and can be used for applications of recognizing the gesture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer gesture calibration), vibration recognition related functions (such as pedometer and knocking), and the like; other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that may also be configured with the handset are not described in detail herein.

Audio circuitry 1060, a speaker 1061, and a microphone 1062 may provide an audio interface between a user and a cell phone. Audio circuit 1060 may transmit the received electrical signal after audio data conversion to speaker 1061 for conversion by speaker 1061 into an audio signal output; on the other hand, microphone 1062 converts the collected sound signals into electrical signals, which are received by audio circuit 1060 and converted into audio data, which are processed by audio data output processor 1080 for transmission to, for example, another cell phone via RF circuit 1010 or for output to memory 1020 for further processing.

Wi-Fi belongs to a short-distance wireless transmission technology, and a mobile phone can help a user to send and receive e-mails, browse web pages, access streaming media and the like through a Wi-Fi module 1070, so that wireless broadband Internet access is provided for the user. Although fig. 5 shows Wi-Fi module 1070, it is understood that it does not belong to the necessary constitution of the handset, and can be omitted entirely as required within the scope of not changing the essence of the invention.

Processor 1080 is the control center of the handset, connects the various parts of the entire handset using various interfaces and lines, and performs various functions and processes of the handset by running or executing software programs and/or modules stored in memory 1020, and invoking data stored in memory 1020, thereby performing overall monitoring of the handset. Optionally, processor 1080 may include one or more processing units; alternatively, processor 1080 may integrate an application processor primarily handling operating systems, user interfaces, applications, etc., with a modem processor primarily handling wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 1080.

The handset further includes a power source 1090 (e.g., a battery) for powering the various components, optionally in logical communication with the processor 1080 via a power management system, such as for managing charge, discharge, and power consumption by the power management system.

Although not shown, the mobile phone may further include a camera, a bluetooth module, etc., which will not be described herein.

In the embodiment of the present application, the processor 1080 included in the mobile phone further has a control unit for executing the above model training method executed by the challenge sample generating device.

Referring to fig. 6, fig. 6 is a schematic diagram of a server structure according to an embodiment of the present application, where the server 1100 may have a relatively large difference due to different configurations or performances, and may include one or more central processing units (in english: central processing units, in english: CPU) 1122 (for example, one or more processors) and a memory 1132, and one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Wherein the memory 1132 and the storage medium 1130 may be transitory or persistent. The program stored on the storage medium 1130 may include one or more modules (not shown), each of which may include a series of instruction operations on a server. Still further, the central processor 1122 may be provided in communication with a storage medium 1130, executing a series of instruction operations in the storage medium 1130 on the server 1100.

The Server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input-output interfaces 1158, and/or one or more operating systems 1141, such as Windows Server, mac OS X, unix, linux, freeBSD, and the like.

The steps in the model training method in the above embodiment may be based on the structure of the server 1100 shown in fig. 6. For example, the CPU 1122 may perform the following operations by calling instructions in the memory 1132:

acquiring an initial challenge sample and a target attack category;

In one embodiment, the step of constructing the objective function of the model to be trained from the initial challenge sample and the objective attack class by gaussian process and prior distribution in bayesian optimization may also be implemented by the central processor 1122 by invoking instructions in the memory 1132:

In one embodiment, the step of determining the objective loss function of the model to be trained from the objective function may also be implemented by the central processor 1122 by calling instructions in the memory 1132:

In one embodiment, the step of searching a set of most harmful transform functions in the vicinity of the initial transform function by the cpu 1122 through the instruction in the call memory 1132 may be further implemented to obtain a set of most harmful transform functions corresponding to the initial transform function:

In one embodiment, the step of determining the target loss function of the model to be trained from the initial transformation function and the most harmful transformation function may also be implemented by the central processor 1122 by invoking instructions in the memory 1132:

In one embodiment, the step of determining a first penalty function that minimizes classification of the countered samples into target attack categories from the initial transform function and the most harmful transform function may also be implemented by the central processor 1122 by invoking instructions in the memory 1132:

In one embodiment, the step of obtaining the distance in the norm of the challenge sample and the original sample may also be accomplished by the central processor 1122 by invoking instructions in the memory 1132:

acquiring a weight parameter of the initial distance;

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the systems, apparatuses and modules described above may refer to the corresponding processes in the foregoing method embodiments, which are not repeated herein.

In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.

The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed over a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the flow or functions described in accordance with embodiments of the present application are fully or partially produced. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.

The foregoing describes in detail the technical solution provided by the embodiments of the present application, in which specific examples are applied to illustrate the principles and implementations of the embodiments of the present application, where the foregoing description of the embodiments is only used to help understand the methods and core ideas of the embodiments of the present application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope according to the ideas of the embodiments of the present application, the present disclosure should not be construed as limiting the embodiments of the present application in view of the above.

Claims

1. A method of challenge sample generation, the method comprising:

acquiring an initial challenge sample and a target attack category;

2. The challenge sample generation method of claim 1, wherein the constructing an objective function of a model to be trained from the initial challenge sample and the objective attack class by gaussian process and prior distribution in bayesian optimization comprises:

3. The method of claim 1, wherein determining an objective loss function of the model to be trained from the objective function comprises:

4. A challenge sample generation method according to claim 3, wherein searching a set of most harmful transform functions in the vicinity of the initial transform function to obtain a set of most harmful transform functions corresponding to the initial transform function comprises:

5. A challenge sample generation method according to claim 3, wherein said determining an objective loss function of the model to be trained from the initial transform function and the most detrimental transform function comprises:

6. The challenge sample generation method of claim 5, wherein determining a first loss function that minimizes classification of a challenge sample into a target attack class based on the initial transform function and the most deleterious transform function comprises:

7. The challenge sample generation method of claim 5, wherein the obtaining the distance between the challenge sample and the original sample in the norm sense comprises:

acquiring a weight parameter of the initial distance;

8. A method of model evaluation, the method comprising:

the robustness of the target model is evaluated using a plurality of challenge samples generated by the method for generating challenge samples according to any one of claims 1 to 7.

9. An challenge sample generating device, the device comprising:

10. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of any of claims 1 to 7.