WO2022077907A1 - 对抗攻击的检测方法、系统、设备、计算机可读存储介质 - Google Patents

对抗攻击的检测方法、系统、设备、计算机可读存储介质 Download PDF

Info

Publication number
WO2022077907A1
WO2022077907A1 PCT/CN2021/095175 CN2021095175W WO2022077907A1 WO 2022077907 A1 WO2022077907 A1 WO 2022077907A1 CN 2021095175 W CN2021095175 W CN 2021095175W WO 2022077907 A1 WO2022077907 A1 WO 2022077907A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
detected
category
activation
path corresponding
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/095175
Other languages
English (en)
French (fr)
Inventor
李照冬
冷静文
邱宇贤
过敏意
朱禺皓
陈�全
李超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jiao Tong University
Original Assignee
Shanghai Jiao Tong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Jiao Tong University filed Critical Shanghai Jiao Tong University
Publication of WO2022077907A1 publication Critical patent/WO2022077907A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models

Definitions

  • the present invention belongs to the technical field of neural networks, and relates to a detection method, system, and device, and in particular, to a detection method, system, device, and computer-readable storage medium against attacks.
  • Adversarial training is an important way to enhance the robustness of neural networks.
  • the samples will be mixed with some tiny disturbances, and the neural network can adapt to this disturbance through training, thereby enhancing the robustness of the neural network to adversarial samples.
  • the second prior art by constructing a multi-module redundant system.
  • the second defect of the prior art the overhead is too large, making it impossible to detect the adversarial attack online during the reasoning process.
  • the existing adversarial attack detection methods have many problems: on the one hand, these detection methods cannot realize online adversarial attack detection during the neural network inference process; on the other hand, these detection methods also bring a lot of extra overhead . Adversarial attacks will seriously endanger the reliability and privacy of neural network systems. In mission-critical scenarios such as autonomous driving, it is particularly important to achieve online adversarial attack detection.
  • the purpose of the present invention is to provide a detection method, system, device, and computer-readable storage medium for counteracting attacks, so as to solve the problem that the prior art cannot implement online inference in the neural network inference process.
  • Adversarial attack detection will bring additional overhead, which will seriously endanger the reliability and privacy of neural network systems.
  • one aspect of the present invention provides a detection method for confrontation attack, which is applied to a neural network; the detection method for confrontation attack includes: receiving training data, and extracting an activation path corresponding to the training data; Integrate several activation paths to form a category path corresponding to the training data of the entire category; receive the data to be detected, and extract the activation path corresponding to the data to be detected; calculate the category path corresponding to the data to be detected. Similarity between activation paths; according to the similarity, it is determined whether the data to be detected is an adversarial sample.
  • the step of extracting the activation path corresponding to the training data includes: according to the important neurons in the last layer of the neural network corresponding to the training data, reversely extract the important neurons of each layer layer by layer. neurons to obtain all important neurons for a class label in the neural network; several activation paths are formed through the determined important neurons in each layer.
  • the method of extracting the important neurons of each layer includes one or any combination of forward extraction, threshold mechanism and selective extraction.
  • the step of integrating several activation paths to form a category path corresponding to samples of the entire category includes: bitwise ORing the activation paths corresponding to the training data belonging to the same category through a bitwise OR operation. Calculate the category path corresponding to the entire category.
  • the step of extracting the activation path corresponding to the data to be detected includes: according to the important neurons in the last layer of the neural network corresponding to the data to be detected, reversely extract each layer by layer.
  • the important neurons of one layer are used to obtain all the important neurons for a class label in the neural network; the activation paths corresponding to the data to be detected are formed by the determined important neurons in each layer.
  • the step of judging whether the data to be detected is an adversarial sample according to the similarity includes: using a pre-stored classifier to associate the category path with the activation path corresponding to the data to be detected. The similarity between them is classified to determine whether the data to be detected is an adversarial sample.
  • the detection system against attacks includes: a data receiving module for receiving training data and data to be detected; a first extraction module for extracting an activation path corresponding to the training data; an integration module for integrating several activation paths to form a category path corresponding to the training data of the entire category; a second extraction module for extracting an activation path corresponding to the data to be detected ; a processing module, configured to calculate the similarity between the category path and the activation path corresponding to the data to be detected; according to the similarity, determine whether the data to be detected is an adversarial sample.
  • Yet another aspect of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method for detecting an anti-attack.
  • a final aspect of the present invention provides a detection device, comprising: a processor and a memory; the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory, so that the detection device executes the Adversarial attack detection methods.
  • the processor includes a neural network accelerator, a path builder, and a controller.
  • the detection method, system, device, and computer-readable storage medium of the present invention against attack have the following beneficial effects:
  • the method, system, device, and computer-readable storage medium for detecting an adversarial attack of the present invention can realize the detection of an online adversarial attack during the reasoning process of the neural network, so as to detect the abnormality of the neural network model; and the detection described in this embodiment
  • the method can achieve high detection accuracy and low overhead, which provides support for the robustness of deep learning systems.
  • FIG. 1 is a schematic flowchart of an embodiment of the method for detecting an adversarial attack of the present invention.
  • FIG. 2 is a schematic diagram showing the calculation of important neurons on the fully connected layer of the neural network of the present invention.
  • FIG. 3 is a schematic diagram showing the calculation of important neurons on the convolutional layer of the neural network of the present invention.
  • FIG. 4 is a schematic diagram showing the activation path structure of the neural network of the present invention.
  • FIG. 5 is a schematic diagram showing the principle structure of the detection system against attack of the present invention in an embodiment.
  • FIG. 6 is a schematic diagram showing the principle structure of the detection device of the present invention in an embodiment.
  • each input of the neural network will activate a number of neurons inside the neural network (the activated neurons refer to the neurons that can have a greater impact on the inference results, and are also referred to as Theoretical studies have shown that the neurons activated by inputs with the same class label tend to be similar, while the neurons activated by inputs belonging to different classes tend to be quite different. In other words, in the inference process of the neural network, there will be a large difference between the neurons activated by the normal input and the neurons activated by the adversarial example. Therefore, we can determine whether the input is a normal input or an adversarial sample by comparing the neurons activated by an input with those activated by a normal sample, thereby realizing adversarial attack detection.
  • An important neuron refers to a neuron that, for a certain input, will have a greater impact on the inference result during the inference process, that is, the neuron activated by the input mentioned above; Important neurons, enabling adversarial attack detection during inference.
  • This embodiment provides a detection method for confrontation attack, which is applied to a neural network; the detection method for confrontation attack includes:
  • FIG. 1 is a schematic flowchart of a method for detecting an adversarial attack in an embodiment.
  • the detection method for an adversarial attack specifically includes the following steps:
  • the S11 specifically includes:
  • the important neurons of the last layer in the neural network corresponding to the training data are reversely extracted layer by layer, so as to obtain all the important neurons for a class label in the neural network.
  • the important neuron in the latter layer is the second neuron (with a value of 0.46)
  • 0.46 0.1x2.1+1.0x0.09+0.4x0.2+0.3x0.2+ 0.2x0.1 (each item is arranged in descending order of value); determine a proportional coefficient ⁇ , ⁇ controls the proportion of the important neurons in the previous layer in the important neurons in the latter layer, and then controls the important neurons in the previous layer.
  • Each neuron is the important neuron in the previous layer; if there are other important neurons in the previous layer, it is calculated in the same way. Initially, we artificially set the neuron with the largest value in the last layer of the neural network (that is, the neuron corresponding to the output category) as the only important neuron in the last layer.
  • the method for extracting the activation path is based on the two aspects of the detection accuracy of the adversarial attack and the overhead, and the influence of the two is weighed.
  • the method of extracting important neurons of each layer includes one or any combination of forward extraction, threshold mechanism and selective extraction.
  • the extraction of important neurons is carried out in reverse, which makes the two processes of inference and extraction have a serial relationship.
  • the present invention proposes a forward extraction method: when the inference proceeds to the Li+1 layer, the neuron with the largest value can be directly selected from the Li+1 layer as the important neuron of this layer, and then use it to calculate Important neurons in layer Li.
  • the reasoning of the Li+1 layer and the extraction of the Li layer can be carried out at the same time, which will sacrifice part of the accuracy of the adversarial detection, but greatly reduce the time overhead required for the extraction of the entire activation path.
  • the extraction of important neurons is to sort each item in the sum formula, and then select the important neurons in turn.
  • this method has a large computational cost and storage cost. Therefore, the present invention proposes a threshold mechanism: the Lith layer in the neural network can determine a threshold All values in this layer are greater than the threshold All neurons are determined as important neurons, which greatly simplifies the process of extracting important neurons and reduces computational and storage overhead.
  • This embodiment adopts an application programming interface (API), so that a user can implement a combination of the above-mentioned different optimization schemes by calling the API, so as to adapt to different accuracy requirements and overhead requirements in different application scenarios. For example, forward extraction, employing a thresholding mechanism, and extracting only the last three layers of significant neurons can be combined to extract activation paths.
  • the combination of different optimization schemes greatly enhances the flexibility of the present invention.
  • a set of instructions is designed and a corresponding compiler optimization is performed to further improve the performance of the system, aiming at the calculation characteristics of the adversarial attack detection process.
  • the important neurons of each layer are reversely calculated layer by layer, and finally all the important neurons of the entire neural network for a certain input data are obtained. These important neurons have an important influence on the input of the neural network. , which constitute the activation path of the entire neural network, as shown in Figure 4.
  • a bitmask is used to represent the activation path.
  • the activation path in the inference process is assumed to be P(x), then m i,j in P(x)
  • the meaning of the bit is whether the jth neuron of the i-th layer of the neural network is an important neuron; the activation paths corresponding to input data with the same category label are often similar, while the activation paths corresponding to inputs belonging to different categories often exist larger difference.
  • the category path refers to belonging to the same category
  • the path obtained by the bitwise OR of the activation paths corresponding to all normal inputs of , if the activation path and the category path are similar, the input can be considered as a normal input.
  • S12 Integrate several activation paths to form a category path corresponding to the training data of the entire category.
  • S12 includes calculating the category path corresponding to the entire category through the bitwise OR operation of the activation paths corresponding to the training data belonging to the same category, that is, sequentially ORing each bit of the activation path represented by the bit mask ( OR) operation, and finally obtain a category path corresponding to the entire category represented by the bit mask; for example, the bit masks of the two activation paths are 0011 and 0101 respectively, then the bit mask of the calculated category path is 0111.
  • S13 Store the category path, so that the stored category path can be directly read during inference.
  • S14 Receive the data to be detected, and extract the activation path corresponding to the data to be detected.
  • S14 includes:
  • the important neurons of the last layer in the neural network corresponding to the data to be detected are reversely extracted layer by layer, so as to obtain all the important neurons for a class label in the neural network.
  • An activation path corresponding to the data to be detected is formed by the important neurons determined in each layer.
  • the method of extracting the activation path corresponding to the data to be detected is the same as the method of extracting the activation path of the training data, which will not be repeated here.
  • the above S14 and S11 may be executed simultaneously, or may be executed sequentially. In this embodiment, the S14 and the S11 are executed simultaneously.
  • the calculation formula for calculating the similarity between the category path and the activation path corresponding to the data to be detected is as follows:
  • S represents the activation path corresponding to the category path and the data to be detected
  • P(x) represents the activation path corresponding to the data to be detected
  • 1 represents the activation path corresponding to the data to be detected.
  • the number of ; & represents the bitwise OR operation
  • Pc represents the category path corresponding to the training data of the entire category, and c is the category.
  • a pre-stored classifier is used to classify the similarity between the category path and the activation path corresponding to the data to be detected, so as to determine whether the data to be detected is an adversarial sample.
  • the classifier adopts the random forest method in the field of machine learning.
  • S17 output the judgment result of S16, that is, the data to be detected is an adversarial sample or a non-confrontational sample.
  • the detection method for adversarial attacks described in this embodiment can realize the detection of online adversarial attacks during the reasoning process of the neural network, so as to detect the abnormality of the neural network model; and the detection method described in this embodiment can achieve a higher detection accuracy. As well as lower overhead, it supports the robustness of deep learning systems.
  • This embodiment also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the detection method against an attack.
  • the computer-readable storage medium means that all or part of the steps of implementing the above method embodiments can be completed by hardware related to computer programs.
  • the aforementioned computer program may be stored in a computer-readable storage medium.
  • the steps including the above method embodiments are executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.
  • This embodiment provides a detection system against attacks, which is applied to a neural network; the detection system against attacks includes:
  • a data receiving module for receiving training data and data to be detected
  • the first extraction module is used to extract the activation path corresponding to the training data
  • the integration module is used to integrate several activation paths to form a category path corresponding to the training data of the entire category;
  • the second extraction module is used to extract the activation path corresponding to the data to be detected
  • a processing module configured to calculate the similarity between the category path and the activation path corresponding to the data to be detected; according to the similarity, determine whether the data to be detected is an adversarial sample.
  • FIG. 5 is a schematic diagram showing the principle structure of the detection system against attack in one embodiment.
  • the anti-attack detection system 5 includes a data receiving module 51 , a first extraction module 52 , an integration module 53 , a storage module 54 , a second extraction module 55 and a processing module 56 .
  • the data receiving module 51 is used for receiving training data and data to be detected.
  • the first extraction module 52 connected to the data receiving module 51 is used to extract the activation path corresponding to the training data.
  • the first extraction module 52 reversely extracts the important neurons of each layer layer by layer according to the important neurons of the last layer in the neural network corresponding to the training data, so as to obtain the information in the neural network. All significant neurons for a class label. Several of the activation paths are formed through the identified important neurons in each layer.
  • the first extraction module 52 extracts the important neurons of each layer in one or any combination of forward extraction, threshold mechanism and selective extraction.
  • the specific extraction process is the same as the extraction scheme in the above method, and is not repeated here.
  • the integration module 53 connected to the first extraction module 52 is configured to calculate the category path corresponding to the entire category through the bitwise OR operation of the activation paths corresponding to the training data belonging to the same category.
  • the storage module 54 connected with the integration module 53 is used for storing the category path, so that the processing module 56 can directly read the stored category path during inference.
  • the second extraction module 55 connected with the data receiving module 51 is used to extract the important neurons of each layer in reverse layer by layer according to the important neurons of the last layer in the neural network corresponding to the data to be detected, so as to obtain the obtained data. All important neurons in the neural network for a class label.
  • the process of extracting the activation path corresponding to the data to be detected by the second extraction module 55 is the same as the process of extraction by the first extraction module 52 , and details are not repeated here.
  • the processing module 56 connected with the storage module 54 and the second extraction module 55 is used to calculate the similarity between the category path and the activation path corresponding to the data to be detected; Whether the data to be detected is an adversarial sample or not, and output the judgment result.
  • the judgment result includes whether the data to be detected is an adversarial sample or a non-confrontational sample.
  • S represents the activation path corresponding to the category path and the data to be detected
  • P(x) represents the activation path corresponding to the data to be detected
  • 1 represents the activation path corresponding to the data to be detected.
  • the number of ; & represents the bitwise OR operation
  • Pc represents the category path corresponding to the training data of the entire category, and c is the category.
  • each module of the above system is only a division of logical functions, and may be fully or partially integrated into a physical entity in actual implementation, or may be physically separated.
  • these modules can all be implemented in the form of software calling through processing elements, or all can be implemented in hardware, and some modules can be implemented in the form of calling software through processing elements, and some modules can be implemented in hardware.
  • the x module may be a separately established processing element, or may be integrated in a certain chip of the above-mentioned system to be implemented.
  • the x module can also be stored in the memory of the above-mentioned system in the form of program code, and is called by a certain processing element of the above-mentioned system to execute the function of the above x-module.
  • the implementation of other modules is similar. All or part of these modules can be integrated together or implemented independently.
  • the processing element described here may be an integrated circuit with signal processing capability.
  • each step of the above-mentioned method or each of the above-mentioned modules can be completed by an integrated logic circuit of hardware in the processor element or an instruction in the form of software.
  • the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), one or more microprocessors (Digital Singnal Processor, DSP for short), one or more Field Programmable Gate Arrays (FPGA for short), etc.
  • ASIC Application Specific Integrated Circuit
  • DSP Digital Singnal Processor
  • FPGA Field Programmable Gate Arrays
  • the processing element may be a general-purpose processor, such as a central processing unit (Central Processing Unit, CPU for short) or other processors that can call program codes.
  • CPU Central Processing Unit
  • These modules can be integrated together and implemented in the form of a System-on-a-chip (SOC for short).
  • SOC System-on-a-chip
  • FIG. 6 is a schematic diagram of the principle structure of the detection device in an embodiment.
  • the detection device 6 includes a memory 61 and a processor 62 .
  • the memory 61 is used to store a computer program
  • the processor 62 is used to execute the computer program stored in the memory 61, so that the detection device executes the detection method against attacks described in Embodiment 1 .
  • the memory 61 includes SRAM and DRAM.
  • the processor 62 includes a neural network accelerator 621 , a path builder 622 and a controller 623 .
  • the above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; may also be a digital signal processor (Digital Signal Processing, referred to as DSP) , Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • CPU Central Processing Unit
  • NP Network Processor
  • DSP Digital Signal Processing
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • the neural network accelerator 621 is used for inference calculation and partial sum recalculation during inference.
  • the path builder 622 is used for extracting important neurons during neural network inference, constructing activation paths, and calculating the similarity between paths.
  • the controller 623 is used for scheduling instructions and running the random forest classifier.
  • the neural network accelerator 621, the path constructor 622 and the SRAM of the controller 623 all adopt double buffering, and perform data transmission through the bus and the DRAM.
  • the protection scope of the anti-attack detection method of the present invention is not limited to the execution sequence of the steps enumerated in this embodiment. within the protection scope of the present invention.
  • the present invention also provides a detection system against attacks, the detection system against attacks can implement the method for detecting against attacks according to the present invention, but the implementation device for the method for detecting against attacks according to the present invention includes but is not limited to The structure of the attack-resistant detection system enumerated in this embodiment, all structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.
  • the method, system, device, and computer-readable storage medium for detecting an adversarial attack can realize the detection of an online adversarial attack during the reasoning process of the neural network, thereby detecting the abnormality of the neural network model;
  • the detection method described in the embodiment can achieve higher detection accuracy and lower overhead, which provides support for the robustness of the deep learning system.
  • the invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Image Analysis (AREA)

Abstract

一种对抗攻击的检测方法、系统、设备、计算机可读存储介质，所述对抗攻击的检测方法包括：接收训练数据，并提取与训练数据对应的激活路径（S11）；将若干激活路径进行整合，以形成整个类别的训练数据对应的类别路径（S12）；接收待检测数据，并提取与所述待检测数据对应的激活路径（S14）；计算所述类别路径与所述待检测数据对应的激活路径之间的相似度（S15）；根据所述类别路径与所述待检测数据对应的激活路径之间的相似度，判断所述待检测数据是否为对抗样本（S16）。能够在神经网络的推理过程中实现在线对抗攻击的检测，从而检测出神经网络模型的异常；且检测方法可以达到较高的检测准确率以及较低的开销，为深度学习系统的鲁棒性提供了支持。

Description

对抗攻击的检测方法、系统、设备、计算机可读存储介质 技术领域
本发明属于神经网络技术领域，涉及一种检测方法、系统、设备，特别是涉及一种对抗攻击的检测方法、系统、设备、计算机可读存储介质。
背景技术
近年来，以深度学习为代表的新一代人工智能算法，在图像处理、语音识别、机器翻译等诸多领域取得一系列重要突破，极大地提高了当前人工智能算法的性能。然而，这类模型通常鲁棒性不足，很容易受到对抗攻击的影响，一个精心设计的对抗样本可能会使原本正常的模型产生错误的结果；对抗攻击对深度学习系统的可靠性、隐私性产生了较大的危害，如何提高模型的鲁棒性，是当前以及未来人工智能领域的研究热点和难点。
以下简单介绍一下现有技术：
现有技术一：对抗训练(adversarial training)是增强神经网络鲁棒性的重要方式。在对抗训练的过程中，样本会被混合一些微小的扰动，通过训练使得神经网络适应这种扰动，从而增强了神经网络对于对抗样本的鲁棒性。
现有技术一的缺陷：现这种通过训练来防御对抗攻击的方式不适合在推理过程中在线检测对抗攻击，并且构建训练数据和进行训练也需要较大的工作量。
现有技术二：通过构建多模块冗余系统。
现有技术二的缺陷：开销过大，使得无法实现在推理过程中在线检测对抗攻击。
所以，当前已有的对抗攻击检测方法存在诸多问题：一方面，这些检测方法无法实现在神经网络推理过程中在线进行对抗攻击检测；另一方面，这些检测方法也会带来很大的额外开销。对抗攻击会严重危害神经网络系统的可靠性、隐私性。在诸如自动驾驶等任务关键型场景下，实现在线的对抗攻击检测尤为重要。
因此，如何提供一种对抗攻击的检测方法、系统、设备、计算机可读存储介质，以解决现有技术无法实现在神经网络推理过程中在线进行对抗攻击检测，且会带来额外开销，导致神经网络系统的可靠性、隐私性受到严重危害等缺陷，实已成为本领域技术人员亟待解决的技术问题。
发明内容
鉴于以上所述现有技术的缺点，本发明的目的在于提供一种对抗攻击的检测方法、系统、 设备、计算机可读存储介质，用于解决现有技术无法实现在神经网络推理过程中在线进行对抗攻击检测，且会带来额外开销，导致神经网络系统的可靠性、隐私性受到严重危害的问题。
为实现上述目的及其他相关目的，本发明一方面提供一种对抗攻击的检测方法，应用于神经网络；所述对抗攻击的检测方法包括：接收训练数据，并提取与训练数据对应的激活路径；将若干激活路径进行整合，以形成整个类别的训练数据对应的类别路径；接收待检测数据，并提取与所述待检测数据对应的激活路径；计算所述类别路径与所述待检测数据对应的激活路径之间的相似度；根据所述相似度，判断所述待检测数据是否为对抗样本。
于本发明的一实施例中，所述提取与训练数据对应的激活路径的步骤包括：根据与训练数据对应的神经网络中最后一层的重要神经元，逐层反向提取每一层的重要神经元，以获取所述神经网络中针对一类别标签的所有重要神经元；通过每一层中确定的重要神经元形成若干所述激活路径。
于本发明的一实施例中，提取每一层的重要神经元的方式包括前向提取、阈值机制及选择性提取中的一种或任意组合方式。
于本发明的一实施例中，所述将若干激活路径进行整合，以形成整个类别的样本对应的类别路径的步骤包括：将属于同一类别的训练数据所对应的激活路径通过按位取或操作计算出整个类别对应的类别路径。
于本发明的一实施例中，所述提取与所述待检测数据对应的激活路径的步骤包括：根据与待检测数据对应的神经网络中最后一层的重要神经元，逐层反向提取每一层的重要神经元，以获取所述神经网络中针对一类别标签的所有重要神经元；通过每一层中确定的重要神经元形成与所述待检测数据对应的激活路径。
于本发明的一实施例中，计算所述类别路径与所述待检测数据对应的激活路径之间的相似度的计算公式如下：S＝||P(x)&Pc|| 1/||P(x)|| 1其中，S表示类别路径与待检测数据对应的激活路径；P(x)表示与待检测数据对应的激活路径；||P(x)|| 1表示与待检测数据对应的激活路径中1的个数；&表示按位取或操作；Pc表示整个类别的训练数据对应的类别路径，c为类别。
于本发明的一实施例中，所述根据所述相似度，判断所述待检测数据是否为对抗样本的步骤包括：通过预存分类器将所述类别路径与所述待检测数据对应的激活路径之间的相似度进行分类，以判断所述待检测数据是否为对抗样本。
本发明另一方面提供一种对抗攻击的检测系统，应用于神经网络；所述对抗攻击的检测系统包括：数据接收模块，用于接收训练数据和待检测数据；第一提取模块，用于提取与训练数据对应的激活路径；整合模块，用于将若干激活路径进行整合，以形成整个类别的训练 数据对应的类别路径；第二提取模块，用于提取与所述待检测数据对应的激活路径；处理模块，用于计算所述类别路径与所述待检测数据对应的激活路径之间的相似度；根据所述相似度，判断所述待检测数据是否为对抗样本。
本发明又一方面提供一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现所述对抗攻击的检测方法。
本发明最后一方面提供一种检测设备，包括：处理器及存储器；所述存储器用于存储计算机程序，所述处理器用于执行所述存储器存储的计算机程序，以使所述检测设备执行所述对抗攻击的检测方法。
于本发明的一实施例中，所述处理器包括神经网络加速器、路径构造器及控制器。
如上所述，本发明所述的对抗攻击的检测方法、系统、设备、计算机可读存储介质，具有以下有益效果：
本发明所述对抗攻击的检测方法、系统、设备、计算机可读存储介质能够在神经网络的推理过程中实现在线对抗攻击的检测，从而检测出神经网络模型的异常；且本实施例所述检测方法可以达到较高的检测准确率以及较低的开销，为深度学习系统的鲁棒性提供了支持。
附图说明
图1显示为本发明的对抗攻击的检测方法于一实施例中的流程示意图。
图2显示为本发明的神经网络的全连接层上重要神经元的计算示意图。
图3显示为本发明的神经网络的卷积层上重要神经元的计算示意图。
图4显示为本发明的神经网络的激活路径结构示意图。
图5显示为本发明的对抗攻击的检测系统于一实施例中的原理结构示意图。
图6显示为本发明的检测设备于一实施例中的原理结构示意图。
元件标号说明
5        对抗攻击的检测系统
51       数据接收模块
52       第一提取模块
53       整合模块
54       存储模块
55       第二提取模块
56        处理模块
6         检测设备
61        存储器
62        处理器
621       神经网络加速器
622       路径构造器
622       控制器
S11～S17  步骤
具体实施方式
以下通过特定的具体实例说明本发明的实施方式，本领域技术人员可由本说明书所揭露的内容轻易地了解本发明的其他优点与功效。本发明还可以通过另外不同的具体实施方式加以实施或应用，本说明书中的各项细节也可以基于不同观点与应用，在没有背离本发明的精神下进行各种修饰或改变。需说明的是，在不冲突的情况下，以下实施例及实施例中的特征可以相互组合。
需要说明的是，以下实施例中所提供的图示仅以示意方式说明本发明的基本构想，遂图式中仅显示与本发明中有关的组件而非按照实际实施时的组件数目、形状及尺寸绘制，其实际实施时各组件的型态、数量及比例可为一种随意的改变，且其组件布局型态也可能更为复杂。
本发明所述对抗攻击的检测方法、系统、设备、计算机可读存储介质的基本原理如下：
正常图片在经过扰动处理后会产生对抗样本。从肉眼观察上两者似乎没有区别，但深度神经网络却无法识别扰动处理后的图片。
在神经网络的推理过程中，神经网络的每个输入都会激活神经网络内部的若干神经元(这些被激活的神经元是指能对推理结果产生较大影响的神经元，后文中也将其称为“重要神经元”)；理论研究表明，具有同一类别标签的输入所激活的神经元往往比较类似，而属于不同类别的输入所激活的神经元往往有较大的差异。换言之，在神经网络的推理过程中，正常的输入所激活的神经元和对抗样本所激活的神经元之间会存在较大的差异。因此我们可以通过将一个输入所激活的神经元和正常样本所激活的神经元进行对比，来确定该输入是正常输入还是对抗样本，从而实现了对抗攻击检测。
基于上述思想，本发明利用重要神经元的概念及其计算方法。重要神经元是指针对某个 输入，在推理过程中对推理结果会产生较大影响的神经元，即上文中提到的被输入激活的神经元；通过比较在推理过程中不同输入所对应的重要神经元，能够实现在推理过程中的对抗攻击检测。
实施例一
本实施例提供一种对抗攻击的检测方法，应用于神经网络；所述对抗攻击的检测方法包括：
接收训练数据，并提取与训练数据对应的激活路径；
将若干激活路径进行整合，以形成整个类别的训练数据对应的类别路径；
接收待检测数据，并提取与所述待检测数据对应的激活路径；
计算所述类别路径与所述待检测数据对应的激活路径之间的相似度；
根据所述相似度，判断所述待检测数据是否为对抗样本。
以下将结合图示对本实施例所提供的对抗攻击的检测方法进行详细说明。请参阅图1，显示为对抗攻击的检测方法于一实施例中的流程示意图。如图1所示，所述对抗攻击的检测方法具体包括以下步骤：
S11，接收训练数据，并提取与训练数据对应的激活路径。
在本实施例中，所述S11具体包括：
根据与训练数据对应的神经网络中最后一层的重要神经元，逐层反向提取每一层的重要神经元，以获取所述神经网络中针对一类别标签的所有重要神经元。
通过每一层中确定的重要神经元形成若干所述激活路径。
具体地，以图2所示，说明神经网络的全连接层上重要神经元的计算方法：
假设后一层中的重要神经元为第二个神经元(值为0.46)，根据全连接层的定义，0.46＝0.1x2.1+1.0x0.09+0.4x0.2+0.3x0.2+0.2x0.1(每项按值的降序排列)；确定一个比例系数θ，θ控制着前一层重要神经元在后一层重要神经元中所占的比例，进而控制了前一层重要神经元的个数；例如当θ＝0.6时，有0.1x2.1+1.0x0.09>0.6x0.46成立，这表明最少仅需要前一层中第四和第五两个神经元(值分别为1.0、0.1)即可超过后一层的重要神经元的值的60％，因此可认为前一层中的第四个神经元和第五个神经元对后一层的影响最大，这两个神经元即为前一层中的重要神经元；若前一层还有其他的重要神经元，按同样的方式计算。初始时，我们人为设定神经网络最后一层中值最大的神经元(即对应输出类别的神经元)为最后一层唯一的重要神经元。
具体地，以图3所示，说明神经网络的卷积层上重要神经元的计算方法：
在卷积层中，首先根据后一层的重要神经元确定其在前一层的接受域，例如5.47＝2.0x0.7+1.4x0.9+1.5x0.8+1.0x0.9+1.1x0.2+1.0x0.2+1.9x0.1+0.2x0.3+0.2x0.2(每项按值的降序排列)；然后根据θ的值找到前一层的重要神经元，例如当θ＝0.6时，有2.0x0.7+1.4x0.9+1.5x0.8>0.6x5.47成立，这表明最少仅需要接受域中值为2.0、1.5、1.4这三个神经元即可超过后一层的重要神经元的值的60％，这三个神经元即为前一层的重要神经元，若前一层还有其他的重要神经元，按同样的方式计算。
本实施例从对抗攻击检测准确率以及开销两方面出发，权衡两者的影响，对激活路径的提取方法。
在本实施例中，提取每一层的重要神经元的方式包括前向提取、阈值机制及选择性提取中的一种或任意组合方式。
前向提取重要神经元：
在全连接层上重要神经元的计算方法和卷积层上重要神经元的计算方法中，重要神经元的提取是反向进行的，该方法使得推理和提取两个过程之间是串行关系，虽然重要神经元的计算非常准确，却带来了较大的时间开销。因此本发明提出了前向提取法：当推理进行到第Li+1层时，可直接从第Li+1层中挑选值最大的神经元作为该层的重要神经元，然后再用它来计算第Li层的重要神经元。这样第Li+1层的推理和第Li层的提取可以同时进行，这虽然会牺牲一部分对抗检测的准确率，但大大减少了整个激活路径的提取所需的时间开销。
阈值机制
在全连接层上重要神经元的计算方法和卷积层上重要神经元的计算方法中，重要神经元的提取是通过将和式中的每一项进行排序，再从中依次挑选重要神经元，以使得前一层的部分和在后一层重要神经元中所占比例不低于θ，这种方法有着较大的计算开销以及存储开销。因此本发明提出了阈值机制：在神经网络中的第Li层可确定一个阈值
Figure PCTCN2021095175-appb-000001
该层中的所有值大于阈值
Figure PCTCN2021095175-appb-000002
的神经元都被判定为重要神经元，这大大简化了提取重要神经元的过程，减少了计算和存储开销。
选择性提取：
在全连接层上重要神经元的计算方法和卷积层上重要神经元的计算方法中，重要神经元的提取是将整个网络的所有层的重要神经元都进行提取，然而理论研究表明[1]，在许多神经网络中，靠后的层往往比前边的层对推理结果有着更大的影响。因此，本申请提出仅对神经网络后面的某些层提取重要神经元，前面层的重要神经元可以直接忽略。选择性提取策略进一步降低了提取重要神经元的开销。
本实施例采用应用程序接口(API)，使得用户可通过调用API来实现上述不同优化方案的组合，以适应不同应用场景下不同的准确率要求和开销要求。例如，可将前向提取、采用阈值机制以及仅提取后三层重要神经元这三者进行组合来提取激活路径。不同优化方案的组合使得本发明的灵活性大大增强。同时本实施例针对对抗攻击检测过程的计算特性，设计了一套指令集并进行了相应的编译器优化，以进一步提升系统的性能。
根据上述重要神经元计算方法，逐层反向计算每一层的重要神经元，最终得到整个神经网络针对某个输入数据的所有重要神经元,这些重要神经元对神经网络的输入有着重要的影响，它们构成了整个神经网络的激活路径，如图4所示。
具体地，本实施例采用比特掩码(bitmask)表示激活路径，对于输入的训练数据x，设其在推理过程中的激活路径为P(x)，则P(x)中的m i,j位的含义为神经网络第i层的第j个神经元是否为重要神经元；具有同一类别标签的输入数据所对应的激活路径往往比较类似，而属于不同类别的输入所对应的激活路径往往存在较大的差异。因此在推导过程中，通过计算当前输入所对应的激活路径，再将该激活路径与类别路径相比较，就能在推理过程中检测对抗攻击；在本实施例中，类别路径是指属于同一类别的所有正常输入所对应的激活路径按位取或得到的路径，若激活路径和类别路径比较相似，则可认为输入是正常输入。
S12，将若干激活路径进行整合，以形成整个类别的训练数据对应的类别路径。
具体地，S12包括将属于同一类别的训练数据所对应的激活路径通过按位或操作计算出整个类别对应的类别路径，即把用比特掩码表示的激活路径的每一位，依次进行或(OR)运算，最后得到一个用比特掩码表示的整个类别对应的类别路径；例如，两个激活路径的比特掩码分别为0011和0101，则计算出的类别路径的比特掩码为0111。
S13，存储所述类别路径，以便在推理时，直接读取已存储的类别路径。
S14，接收待检测数据，并提取与所述待检测数据对应的激活路径。
具体地，S14包括：
根据与待检测数据对应的神经网络中最后一层的重要神经元，逐层反向提取每一层的重要神经元，以获取所述神经网络中针对一类别标签的所有重要神经元。
通过每一层中确定的重要神经元形成与所述待检测数据对应的激活路径。
在本实施例中，所述待检测数据对应的激活路径的提取方法与所述训练数据的激活路径的提取方法一样，此处不再赘述。
以上S14与S11可以同时执行，也可以分先后顺序执行。在本实施例中，所述S14与所述S11为同时执行。
S15，计算所述类别路径与所述待检测数据对应的激活路径之间的相似度。
于本实施例中，计算所述类别路径与所述待检测数据对应的激活路径之间的相似度的计算公式如下：
S＝||P(x)&Pc|| 1/||P(x)|| 1
其中，S表示类别路径与待检测数据对应的激活路径；P(x)表示与待检测数据对应的激活路径；||P(x)|| 1表示与待检测数据对应的激活路径中年1的个数；&表示按位取或操作；Pc表示整个类别的训练数据对应的类别路径，c为类别。
S16，根据所述类别路径与所述待检测数据对应的激活路径之间的相似度，判断所述待检测数据是否为对抗样本。
具体地，通过预存分类器将所述类别路径与所述待检测数据对应的激活路径之间的相似度进行分类，以判断所述待检测数据是否为对抗样本。
于本实施例中，所述分类器采用机器学习领域中的随机森林方法。
S17，输出所述S16的判断结果，即所述待检测数据为对抗样本或为非对抗样本。
本实施例所述对抗攻击的检测方法能够在神经网络的推理过程中实现在线对抗攻击的检测，从而检测出神经网络模型的异常；且本实施例所述检测方法可以达到较高的检测准确率以及较低的开销，为深度学习系统的鲁棒性提供了支持。
本实施例还提供一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现所述对抗攻击的检测方法。
本领域普通技术人员可以理解计算机可读存储介质为：实现上述各方法实施例的全部或部分步骤可以通过计算机程序相关的硬件来完成。前述的计算机程序可以存储于一计算机可读存储介质中。该程序在执行时，执行包括上述各方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
实施例二
本实施例提供一种对抗攻击的检测系统，应用于神经网络；所述对抗攻击的检测系统包括：
数据接收模块，用于接收训练数据和待检测数据；
第一提取模块，用于提取与训练数据对应的激活路径；
整合模块，用于将若干激活路径进行整合，以形成整个类别的训练数据对应的类别路径；
第二提取模块，用于提取与所述待检测数据对应的激活路径；
处理模块，用于计算所述类别路径与所述待检测数据对应的激活路径之间的相似度；根 据所述相似度，判断所述待检测数据是否为对抗样本。
以下将结合图示对本实施例所提供的对抗攻击的检测系统进行详细描述。请参阅图5，显示为对抗攻击的检测系统于一实施例中的原理结构示意图。如图5所示，所述对抗攻击的检测系统5包括数据接收模块51、第一提取模块52、整合模块53、存储模块54、第二提取模块55及处理模块56。
所述数据接收模块51用于接收训练数据和待检测数据。
与所述数据接收模块51连接的所述第一提取模块52用于提取与训练数据对应的激活路径。
在本实施例中，所述第一提取模块52根据与训练数据对应的神经网络中最后一层的重要神经元，逐层反向提取每一层的重要神经元，以获取所述神经网络中针对一类别标签的所有重要神经元。通过每一层中确定的重要神经元形成若干所述激活路径。
在本实施例中，所述第一提取模块52提取每一层的重要神经元的方式包括前向提取、阈值机制及选择性提取中的一种或任意组合方式。具体提取过程与上述方法中的提取方案相同，此处不再赘述。
与所述第一提取模块52连接的所述整合模块53用于将属于同一类别的训练数据所对应的激活路径通过按位取或操作计算出整个类别对应的类别路径。
与所述整合模块53连接的存储模块54用于存储所述类别路径，以便所述处理模块56在推理时，直接读取已存储的类别路径。
与所述数据接收模块51连接的第二提取模块55用于根据与待检测数据对应的神经网络中最后一层的重要神经元，逐层反向提取每一层的重要神经元，以获取所述神经网络中针对一类别标签的所有重要神经元。
所述第二提取模块55提取与所述待检测数据对应的激活路径的过程与第一提取模块52提取的过程相同，此处不再赘述。
与所述存储模块54和所述第二提取模块55连接的处理模块56用于计算所述类别路径与所述待检测数据对应的激活路径之间的相似度；根据所述相似度，判断所述待检测数据是否为对抗样本，并输出判断结果。所述判断结果包括所述待检测数据为对抗样本或为非对抗样本。
具体地，所述处理模块56通过公式S＝||P(x)&Pc|| 1/||P(x)|| 1计算所述类别路径与所述待检测数据对应的激活路径之间的相似度。其中，S表示类别路径与待检测数据对应的激活路径；P(x)表示与待检测数据对应的激活路径；||P(x)|| 1表示与待检测数据对应的激活路径中年 1的个数；&表示按位取或操作；Pc表示整个类别的训练数据对应的类别路径，c为类别。
需要说明的是，应理解以上系统的各个模块的划分仅仅是一种逻辑功能的划分，实际实现时可以全部或部分集成到一个物理实体上，也可以物理上分开。且这些模块可以全部以软件通过处理元件调用的形式实现，也可以全部以硬件的形式实现，还可以部分模块通过处理元件调用软件的形式实现，部分模块通过硬件的形式实现。例如：x模块可以为单独设立的处理元件，也可以集成在上述系统的某一个芯片中实现。此外，x模块也可以以程序代码的形式存储于上述系统的存储器中，由上述系统的某一个处理元件调用并执行以上x模块的功能。其它模块的实现与之类似。这些模块全部或部分可以集成在一起，也可以独立实现。这里所述的处理元件可以是一种集成电路，具有信号的处理能力。在实现过程中，上述方法的各步骤或以上各个模块可以通过处理器元件中的硬件的集成逻辑电路或者软件形式的指令完成。以上这些模块可以是被配置成实施以上方法的一个或多个集成电路，例如：一个或多个特定集成电路(Application Specific Integrated Circuit，简称ASIC)，一个或多个微处理器(Digital Singnal Processor，简称DSP)，一个或者多个现场可编程门阵列(Field Programmable Gate Array，简称FPGA)等。当以上某个模块通过处理元件调度程序代码的形式实现时，该处理元件可以是通用处理器，如中央处理器(Central Processing Unit，简称CPU)或其它可以调用程序代码的处理器。这些模块可以集成在一起，以片上系统(System-on-a-chip，简称SOC)的形式实现。
实施例三
本实施例提供一种检测设备，请参阅图6，显示为检测设备于一实施例中的原理结构示意图。如图6所示，所述检测设备6包括存储器61和处理器62。
在本实施例中，所述存储器61用于存储计算机程序，所述处理器62用于执行所述存储器61存储的计算机程序，以使所述检测设备执行实施例一所述对抗攻击的检测方法。
在本实施例中，所述存储器61包括SRAM和DRAM。
在本实施例中，所述处理器62包括神经网络加速器621、路径构造器622及控制器623。
上述的处理器可以是通用处理器，包括中央处理器(Central Processing Unit，简称CPU)、网络处理器(Network Processor，简称NP)等；还可以是数字信号处理器(Digital Signal Processing，简称DSP)、专用集成电路(Application Specific Integrated Circuit，简称ASIC)、现场可编程门阵列(Field Programmable Gate Array，简称FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。
于本实施例中，所述神经网络加速器621用于推理计算以及推理时部分和的重计算。
所述路径构造器622用于提取神经网络推理时的重要神经元，构造激活路径，以及计算路径之间的相似度。所述控制器623用于指令的调度以及运行随机森林分类器。
其中，神经网络加速器621、路径构造器622和控制器623的SRAM均采用双缓冲，通过总线和DRAM进行数据传输。
本发明所述的对抗攻击的检测方法的保护范围不限于本实施例列举的步骤执行顺序，凡是根据本发明的原理所做的现有技术的步骤增减、步骤替换所实现的方案都包括在本发明的保护范围内。
本发明还提供一种对抗攻击的检测系统，所述对抗攻击的检测系统可以实现本发明所述的对抗攻击的检测方法，但本发明所述的对抗攻击的检测方法的实现装置包括但不限于本实施例列举的对抗攻击的检测系统的结构，凡是根据本发明的原理所做的现有技术的结构变形和替换，都包括在本发明的保护范围内。
综上所述，本发明所述对抗攻击的检测方法、系统、设备、计算机可读存储介质能够在神经网络的推理过程中实现在线对抗攻击的检测，从而检测出神经网络模型的异常；且本实施例所述检测方法可以达到较高的检测准确率以及较低的开销，为深度学习系统的鲁棒性提供了支持。本发明有效克服了现有技术中的种种缺点而具高度产业利用价值。
上述实施例仅例示性说明本发明的原理及其功效，而非用于限制本发明。任何熟悉此技术的人士皆可在不违背本发明的精神及范畴下，对上述实施例进行修饰或改变。因此，举凡所属技术领域中具有通常知识者在未脱离本发明所揭示的精神与技术思想下所完成的一切等效修饰或改变，仍应由本发明的权利要求所涵盖。

Claims (11)

  1. 一种对抗攻击的检测方法，其特征在于，应用于神经网络；所述对抗攻击的检测方法包括：
    接收训练数据，并提取与训练数据对应的激活路径；
    将若干激活路径进行整合，以形成整个类别的训练数据对应的类别路径；
    接收待检测数据，并提取与所述待检测数据对应的激活路径；
    计算所述类别路径与所述待检测数据对应的激活路径之间的相似度；
    根据所述相似度，判断所述待检测数据是否为对抗样本。
  2. 根据权利要求1所述的对抗攻击的检测方法，其特征在于，所述提取与训练数据对应的激活路径的步骤包括：
    根据与训练数据对应的神经网络中最后一层的重要神经元，逐层反向提取每一层的重要神经元，以获取所述神经网络中针对一类别标签的所有重要神经元；
    通过每一层中确定的重要神经元形成若干所述激活路径。
  3. 根据权利要求2所述的对抗攻击的检测方法，其特征在于，提取每一层的重要神经元的方式包括前向提取、阈值机制及选择性提取中的一种或任意组合方式。
  4. 根据权利要求2所述的对抗攻击的检测方法，其特征在于，所述将若干激活路径进行整合，以形成整个类别的样本对应的类别路径的步骤包括：
    将属于同一类别的训练数据所对应的激活路径通过按位取或操作计算出整个类别对应的类别路径。
  5. 根据权利要求2所述的对抗攻击的检测方法，其特征在于，所述提取与所述待检测数据对应的激活路径的步骤包括：
    根据与待检测数据对应的神经网络中最后一层的重要神经元，逐层反向提取每一层的重要神经元，以获取所述神经网络中针对一类别标签的所有重要神经元；
    通过每一层中确定的重要神经元形成与所述待检测数据对应的激活路径。
  6. 根据权利要求5所述的对抗攻击的检测方法，其特征在于，计算所述类别路径与所述待检测数据对应的激活路径之间的相似度的计算公式如下：
    S＝||P(x)&Pc|| 1/||P(x)|| 1
    其中，S表示类别路径与待检测数据对应的激活路径；P(x)表示与待检测数据对应的激 活路径；||P(x)|| 1表示与待检测数据对应的激活路径中1的个数；&表示按位取或操作；Pc表示整个类别的训练数据对应的类别路径，c为类别。
  7. 根据权利要求1或6所述的对抗攻击的检测方法，其特征在于，所述根据所述相似度，判断所述待检测数据是否为对抗样本的步骤包括：
    通过预存分类器将所述类别路径与所述待检测数据对应的激活路径之间的相似度进行分类，以判断所述待检测数据是否为对抗样本。
  8. 一种对抗攻击的检测系统，其特征在于，应用于神经网络；所述对抗攻击的检测系统包括：
    数据接收模块，用于接收训练数据和待检测数据；
    第一提取模块，用于提取与训练数据对应的激活路径；
    整合模块，用于将若干激活路径进行整合，以形成整个类别的训练数据对应的类别路径；
    第二提取模块，用于提取与所述待检测数据对应的激活路径；
    处理模块，用于计算所述类别路径与所述待检测数据对应的激活路径之间的相似度；根据所述相似度，判断所述待检测数据是否为对抗样本。
  9. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该程序被处理器执行时实现权利要求1至7中任一项所述对抗攻击的检测方法。
  10. 一种检测设备，其特征在于，包括：处理器及存储器；
    所述存储器用于存储计算机程序，所述处理器用于执行所述存储器存储的计算机程序，以使所述检测设备执行如权利要求1至7中任一项所述对抗攻击的检测方法。
  11. 根据权利要求11所述的检测设备，其特征在于，所述处理器包括神经网络加速器、路径构造器及控制器。
PCT/CN2021/095175 2020-10-14 2021-05-21 对抗攻击的检测方法、系统、设备、计算机可读存储介质 Ceased WO2022077907A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011094465.X 2020-10-14
CN202011094465.XA CN112163670A (zh) 2020-10-14 2020-10-14 对抗攻击的检测方法、系统、设备、计算机可读存储介质

Publications (1)

Publication Number Publication Date
WO2022077907A1 true WO2022077907A1 (zh) 2022-04-21

Family

ID=73866827

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/095175 Ceased WO2022077907A1 (zh) 2020-10-14 2021-05-21 对抗攻击的检测方法、系统、设备、计算机可读存储介质

Country Status (2)

Country Link
CN (1) CN112163670A (zh)
WO (1) WO2022077907A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115776401A (zh) * 2022-11-23 2023-03-10 中国人民解放军国防科技大学 基于少样本学习对网络攻击事件进行溯源的方法、装置
CN116523026A (zh) * 2023-05-08 2023-08-01 中移动信息技术有限公司 模型训练方法、装置、设备及计算机存储介质
CN117764120A (zh) * 2024-02-22 2024-03-26 天津普智芯网络测控技术有限公司 一种可降低单粒子故障影响的图片识别架构

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112163670A (zh) * 2020-10-14 2021-01-01 上海交通大学 对抗攻击的检测方法、系统、设备、计算机可读存储介质
CN113850382A (zh) * 2021-09-24 2021-12-28 上海数熙传媒科技有限公司 一种基于智能数字屏的内容审查硬件系统
CN114492832B (zh) * 2021-12-24 2024-10-01 北京航空航天大学 一种基于联想学习的选择性攻击方法及装置
CN117787352B (zh) * 2023-12-01 2024-09-03 北京天融信网络安全技术有限公司 用于神经网络模型的对抗样本检测的方法及处理器

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190220605A1 (en) * 2019-03-22 2019-07-18 Intel Corporation Adversarial training of neural networks using information about activation path differentials
CN110046712A (zh) * 2019-04-04 2019-07-23 天津科技大学 基于生成模型的隐空间模型化策略搜索学习方法
CN111079930A (zh) * 2019-12-23 2020-04-28 深圳市商汤科技有限公司 数据集质量参数的确定方法、装置及电子设备
CN111651765A (zh) * 2020-05-27 2020-09-11 上海交通大学 基于生成式对抗网络的程序执行路径生成方法
CN112163670A (zh) * 2020-10-14 2021-01-01 上海交通大学 对抗攻击的检测方法、系统、设备、计算机可读存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190220605A1 (en) * 2019-03-22 2019-07-18 Intel Corporation Adversarial training of neural networks using information about activation path differentials
CN111723927A (zh) * 2019-03-22 2020-09-29 英特尔公司 使用与激活路径差分有关的信息对神经网络的对抗训练
CN110046712A (zh) * 2019-04-04 2019-07-23 天津科技大学 基于生成模型的隐空间模型化策略搜索学习方法
CN111079930A (zh) * 2019-12-23 2020-04-28 深圳市商汤科技有限公司 数据集质量参数的确定方法、装置及电子设备
CN111651765A (zh) * 2020-05-27 2020-09-11 上海交通大学 基于生成式对抗网络的程序执行路径生成方法
CN112163670A (zh) * 2020-10-14 2021-01-01 上海交通大学 对抗攻击的检测方法、系统、设备、计算机可读存储介质

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115776401A (zh) * 2022-11-23 2023-03-10 中国人民解放军国防科技大学 基于少样本学习对网络攻击事件进行溯源的方法、装置
CN115776401B (zh) * 2022-11-23 2024-04-19 中国人民解放军国防科技大学 基于少样本学习对网络攻击事件进行溯源的方法、装置
CN116523026A (zh) * 2023-05-08 2023-08-01 中移动信息技术有限公司 模型训练方法、装置、设备及计算机存储介质
CN116523026B (zh) * 2023-05-08 2026-01-09 中移动信息技术有限公司 模型训练方法、装置、设备及计算机存储介质
CN117764120A (zh) * 2024-02-22 2024-03-26 天津普智芯网络测控技术有限公司 一种可降低单粒子故障影响的图片识别架构

Also Published As

Publication number Publication date
CN112163670A (zh) 2021-01-01

Similar Documents

Publication Publication Date Title
WO2022077907A1 (zh) 对抗攻击的检测方法、系统、设备、计算机可读存储介质
Schorn et al. Efficient on-line error detection and mitigation for deep neural network accelerators
Zhao et al. Software defect prediction via cost-sensitive Siamese parallel fully-connected neural networks
CN105912990B (zh) 人脸检测的方法及装置
CN108027899B (zh) 用于提高经训练的机器学习模型的性能的方法
US20240020514A1 (en) Improper neural network input detection and handling
US20210203605A1 (en) Method and apparatus for detecting abnormal traffic pattern
CN113590472B (zh) 一种回归测试中测试用例优先级排序方法
US11429855B2 (en) Acceleration of neural networks using depth-first processing
US10747961B2 (en) Method and device for identifying a sentence
CN113541985B (zh) 物联网故障诊断方法、模型的训练方法及相关装置
Meng et al. Moead: A parameter-efficient model for multi-class anomaly detection
CN112906865A (zh) 神经网络架构搜索方法、装置、电子设备及存储介质
CN109740348B (zh) 一种基于机器学习的硬件木马定位方法
CN111582229A (zh) 一种网络自适应半精度量化的图像处理方法和系统
KR102886499B1 (ko) 신경망 네트워크의 셀프 어텐션 연산을 가속하는 장치
US20210049469A1 (en) Memory remapping for sparse neural networks
CN115629942A (zh) 信创环境下基于大数据和机器学习的运维数据异常检测处理方法、装置、处理器及存储介质
CN113254939B (zh) 基于多注意力机制和自适应学习的智能合约漏洞检测方法
CN118608823A (zh) 基于改进yolov8的无序零件检测方法及系统
CN117272195A (zh) 基于图卷积注意力网络的区块链异常节点检测方法及系统
CN114743044B (zh) 基于部分域适应的故障诊断方法和装置、存储介质和终端
CN114511015A (zh) 基于指令特征重要性的sdc脆弱性预测方法
CN115687118A (zh) 深度神经网络模型测试方法、系统、设备及存储介质
Chai et al. FDW-YOLOv8: A Lightweight Unmanned Aerial Vehicle Small Target Detection Algorithm Based on Enhanced YOLOv8

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21878961

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21878961

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21878961

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 15.01.2024)

122 Ep: pct application non-entry in european phase

Ref document number: 21878961

Country of ref document: EP

Kind code of ref document: A1