CN113987477A - Distributed AI system poison-throwing prevention method and system - Google Patents
Distributed AI system poison-throwing prevention method and system Download PDFInfo
- Publication number
- CN113987477A CN113987477A CN202111247976.5A CN202111247976A CN113987477A CN 113987477 A CN113987477 A CN 113987477A CN 202111247976 A CN202111247976 A CN 202111247976A CN 113987477 A CN113987477 A CN 113987477A
- Authority
- CN
- China
- Prior art keywords
- risk
- data
- nodes
- node
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 230000002265 prevention Effects 0.000 title claims abstract description 17
- 231100000572 poisoning Toxicity 0.000 claims abstract description 52
- 230000000607 poisoning effect Effects 0.000 claims abstract description 52
- 238000012545 processing Methods 0.000 claims abstract description 32
- 241000700605 Viruses Species 0.000 claims abstract description 14
- 238000007689 inspection Methods 0.000 claims description 27
- 238000012360 testing method Methods 0.000 claims description 12
- 230000001186 cumulative effect Effects 0.000 claims description 7
- 239000002574 poison Substances 0.000 claims description 7
- 231100000614 poison Toxicity 0.000 claims description 7
- 238000012163 sequencing technique Methods 0.000 claims description 6
- 230000002155 anti-virotic effect Effects 0.000 claims description 4
- 230000001147 anti-toxic effect Effects 0.000 claims 1
- 208000005374 Poisoning Diseases 0.000 description 42
- 238000013473 artificial intelligence Methods 0.000 description 19
- 230000007123 defense Effects 0.000 description 11
- 230000009286 beneficial effect Effects 0.000 description 6
- 230000002159 abnormal effect Effects 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000001066 destructive effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000005059 dormancy Effects 0.000 description 1
- 238000005242 forging Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 231100000252 nontoxic Toxicity 0.000 description 1
- 230000003000 nontoxic effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 231100000331 toxic Toxicity 0.000 description 1
- 230000002588 toxic effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/554—Detecting local intrusion or implementing counter-measures involving event detection and direct action
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A virus exposure prevention method and a virus exposure prevention system for a distributed AI system are provided, wherein the method comprises the following steps: receiving submitted data sent by each client node, updating a center model based on the submitted data, judging whether the current system has a poisoning sign according to the difference between the error of the center model and the expected error, if not, mirroring the center model according to a preset strategy, informing each client node to empty cache and re-cache data, wherein the cache data comprises original data, a processing algorithm and intermediate data of the client node; otherwise, acquiring all client nodes submitting data from the last model mirror image to the current moment, taking the client nodes as risk nodes, calculating the risk probability of the risk nodes according to the difference between the error of the central model and the expected error, acquiring the accumulated risk probability of the risk nodes based on the risk probability, inspecting the risk nodes according to the accumulated risk probability, taking the poisoned nodes off line if the poisoned nodes exist, and recovering the central model to be the model of the last mirror image.
Description
Technical Field
The invention relates to the technical field of distributed AI (artificial intelligence) systems, in particular to a virus exposure prevention system method and a virus exposure prevention system for a distributed AI system.
Background
The security of distributed AI systems is always a concern. Because of the frequent data transmission as a large distributed network system, the data with terminal characteristics, adjusted or partially adjusted models are all useful information for frequent synchronization in real-time operation in the network. But also becomes a viable means for an attacker by destroying the data and forging the data. Taking the challenge of poison (poison) as an example: in the distributed AI system, since each client can be exposed to model parameters and training data, some malicious clients are likely to send tampered data or weights to the server, thereby affecting the global model. The virus-inflicted attacks can be generally classified into three categories, Data Poisoning (Data Poisoning), Model Poisoning (Model Poisoning), and Data Modification (Data Modification).
Traditional defense methods fall into two broad categories, active defense (reactive defense) and reactive defense (reactive defense). Active defense is a method of guessing which threats will be met and deploying efficient defense techniques, while reactive defense is an operation that is performed when an attack is discovered. However, these two major defenses are more ideas than actual methods, for example, it is difficult to determine which are threats of attackers, and even a data packet transmitted normally determines that the attacker is data poisoning and model poisoning according to what kind of characteristics. If the judgment standard of the abnormal threat data is reduced, a large amount of data needs to be identified or discarded, so that the running efficiency of the system is low, and if the judgment standard of the abnormal threat data is improved, a large amount of abnormal data can be mixed into a normal business flow, so that the abnormal model in the running of the system causes the failure of a business target. Therefore, the traditional anti-virus method of the distributed AI system has low recognition efficiency and accuracy.
Disclosure of Invention
In view of the foregoing analysis, embodiments of the present invention provide a virus exposure prevention method and system for a distributed AI system, so as to solve the problem that the existing virus exposure prevention method for a distributed AI system has low recognition efficiency and accuracy.
On one hand, the embodiment of the invention provides a virus exposure prevention method for a distributed AI system, which comprises the following steps:
receiving submitted data sent by each client node, updating a center model based on the submitted data, judging whether the current system has a poisoning sign according to the difference between the error of the center model and the expected error, if not, mirroring the center model according to a preset strategy, informing each client node to empty cache and re-cache data, wherein the cache data comprises the original data, the processing algorithm and the intermediate data of the client node;
otherwise, acquiring all client nodes submitting data from the last model mirror image to the current moment, taking the client nodes as risk nodes, calculating the risk probability of the risk nodes according to the difference between the error of the central model and the expected error, acquiring the accumulated risk probability of the risk nodes based on the risk probability, inspecting the risk nodes according to the accumulated risk probability, taking the poisoned nodes off line if the poisoned nodes exist, and recovering the central model to the model of the last mirror image.
Based on the further improvement of the method, whether the current system has the poisoning sign or not is judged according to the difference between the error of the center model and the expected error, the method comprises the steps of testing the center model by adopting test data, and if the error of the center model is larger than the expected error and the difference between the error of the center model and the expected error is larger than a first threshold value, judging that the current system has the poisoning sign.
The beneficial effects of the above technical scheme are as follows: whether the system has poisoning signs or not is judged according to errors of the central model and expected errors, if not, the model is mirrored according to a preset strategy, if yes, the risk node is patrolled according to the accumulated risk probability, and when the risk node is judged to be the poisoning node through patrolling and examining, the poisoning node is forcibly offline, and the model is restored to the last mirroring, so that the poisoning attack is accurately identified through fast reading, the subsequent normal operation of the model is not influenced, and the defense efficiency is improved.
Further, according to the accumulative risk probability, the risk node is patrolled and examined, including:
acquiring all risk nodes with the accumulated risk probability larger than a second threshold value, and sequencing the risk nodes from large to small according to the accumulated risk probability;
sequentially polling each risk node;
and if the ith risk node is judged to be a poisoned node through inspection, forcibly offline the risk node, and continuously inspecting the (i + 1) th to nth risk nodes until encountering a first non-poisoned risk node, and finishing inspection.
The beneficial effects of the above technical scheme are as follows: when the routing inspection is carried out, the risk probability is not only considered, but the previous risk probabilities are added through the accumulated risk probability, and the risk nodes needing risks are determined. By considering the historical risk probability accumulated risk probability of each node, the higher the accumulated risk probability is, the node is in a susceptible environment, the higher the possibility of virus exposure is, extra attention is needed, and therefore the accuracy and the efficiency of detection are improved.
Further, the polling content includes:
acquiring original data, intermediate data, a processing algorithm and submitted data cached by a risk node;
and processing the original data according to the processing algorithm to obtain actual intermediate data and result data, comparing the obtained actual intermediate data and result data with intermediate data and submitted data cached by the risk node, and if the intermediate data or the submitted data have data loss, data errors, data timestamps or data check marks which are not consistent, judging the risk node as a poisoned node.
The beneficial effects of the above technical scheme are as follows: considering the efficiency and comprehensiveness of deep inspection, starting from the original data and processing the data according to a processing algorithm from the lower part of a data chain to obtain actual intermediate data and result data, and comparing the obtained actual intermediate data and result data with intermediate data and submitted data cached by risk nodes to quickly and comprehensively check whether the nodes are poisoned. Meanwhile, the inspection can eliminate the poisoning interference of the client node, comprehensive data can be used for inspection, and the inspection rate is high.
Further, the offline time of the poisoned node is calculated by the following formula:
The beneficial effects of the above technical scheme are as follows: the higher the frequency of the risk nodes and the higher the degree of the accumulated risk probability, the longer the off-shelf time of the nodes is greatly delayed, so that the vulnerable nodes which are attacked for many times are ensured to be infinite forbidden nodes, the client side is informed to use personnel to update or kill viruses, and the network is allowed to be accessed again only after the off-shelf time is reached, thereby ensuring the normal operation of the system.
Further, calculating a risk probability of a risk node according to a difference between an error of the center model and an expected error, comprising:
determining a risk level value according to the difference between the error of the center model and an expected error;
the risk probability of each risk node is the risk level value divided by the number of risk nodes.
The embodiment of the invention provides an anti-virus system of a distributed AI system, which comprises the following modules:
the central model updating module is used for receiving submitted data sent by each client node, updating a central model based on the submitted data, judging whether the current system has a poisoning sign according to the difference between the error of the central model and the expected error, and if not, mirroring the central model according to a preset strategy and informing each client node to empty the cache and perform data caching again;
and the inspection module is used for acquiring all client nodes submitting data from the last model mirror image to the current moment when the central model updating module judges that the current system has the poisoning sign, taking the client nodes as risk nodes, calculating the risk probability of the risk nodes according to the difference between the error of the central model and the expected error, acquiring the accumulated risk probability of the risk nodes based on the risk probability, inspecting the risk nodes according to the accumulated risk probability, taking the poisoning nodes off line if the poisoning nodes exist, and recovering the central model into the model of the last mirror image.
Further, the central model updating module is configured to determine whether the current system has a sign of poisoning according to a difference between an error of the central model and an expected error, where the determining includes testing the central model with test data, and determining that the current system has the sign of poisoning if the error of the central model is greater than the expected error and a difference between the error of the central model and the expected error is greater than a first threshold.
Further, the module of patrolling and examining is used for patrolling and examining the risk node according to accumulative risk probability, includes:
acquiring all risk nodes with the accumulated risk probability larger than a second threshold value, and sequencing the risk nodes from large to small according to the accumulated risk probability;
sequentially polling each risk node;
and if the ith risk node is judged to be a poisoned node through inspection, forcibly offline the risk node, and continuously inspecting the (i + 1) th to nth risk nodes until encountering a first non-poisoned risk node, and finishing inspection.
Further, the module of patrolling and examining includes:
the data acquisition module is used for acquiring original data, intermediate data, a processing algorithm and submitted data cached by the risk node;
and the data comparison module is used for processing the original data according to the processing algorithm to obtain actual intermediate data and submitted data, comparing the obtained actual intermediate data and submitted data with intermediate data and submitted data cached by the risk node, and judging the risk node as a poisoned node if the intermediate data or the submitted data have data loss, data errors, data timestamps or data check marks which are not consistent.
In the invention, the technical schemes can be combined with each other to realize more preferable combination schemes. Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, wherein like reference numerals are used to designate like parts throughout.
Fig. 1 is a flowchart illustrating a virus exposure prevention method of a distributed AI system according to an embodiment of the present invention;
fig. 2 is a block diagram of an anti-virus system of the distributed AI system according to an embodiment of the present invention.
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
The distributed AI system virus attack has the following characteristics:
the counterfeit data and the model need a dedicated module and need to reside in 1 terminal. Otherwise, the overhead of frequent jump installation and deployment is too large, if the dormancy gap attack is carried out, the attack frequency and the data volume are too small, the attack cannot play a destructive role, and the cost is too high for an attacker.
The forged data and model can only be on one or some layers of data packets, and cannot run through the whole data link. For example, if the interactive data of the end user is adopted as the submitted data, only the interactive data of the user in certain interfaces or time periods is necessarily collected, and the data virus-throwing only modifies the last sent interactive information packet and cannot modify the original whole-course data due to the large data processing amount and too long forgery time.
Counterfeit data and models can cause later synchronized AI models to operate abnormally, and although specific data packets cannot be located, the confidence that adulterated data in submitted data between normal models and abnormal AI models is necessarily high.
Based on the above three features, an embodiment of the present invention discloses a poison-throwing prevention method for a distributed AI system, as shown in fig. 1, including the following steps:
s1, receiving submitted data sent by each client node, updating a center model based on the submitted data, judging whether the current system has a poisoning sign according to the difference between the error of the center model and the expected error, if not, mirroring the center model according to a preset strategy and informing each client node to empty the cache and perform data caching again, wherein the cached data comprises the original data, the processing algorithm and the intermediate data of the client node;
s2, otherwise, acquiring all client nodes which submit data from the last time of model mirroring to the current time, taking the client nodes as risk nodes, calculating risk probability of the risk nodes according to the difference between the error of the central model and the expected error, acquiring the cumulative risk probability of the risk nodes based on the risk probability, inspecting the risk nodes according to the cumulative risk probability, taking the poisoned nodes off line if the poisoned nodes exist, and recovering the central model to be the model of the last time of mirroring.
Whether the system has poisoning signs or not is judged according to errors of the central model and expected errors, if not, the model is mirrored according to a preset strategy, if yes, the risk nodes are patrolled according to accumulated risk probability, and when the risk nodes are judged to be poisoning nodes through patrolling and examining, the poisoning nodes are forcibly offline, and the model is restored to the previous mirroring, so that the poisoning attack is rapidly and accurately identified, and because the model of the previous mirroring is a non-poisoning model, the subsequent normal operation of the model is not influenced, and the defense efficiency is improved.
When the AI model is trained, the error is continuously reduced, and the loss is continuously reduced, so that whether the system has poisoning signs can be judged according to the training error and the expected error of the model. Specifically, judging whether the current system has a poisoning sign according to the difference between the error of the center model and the expected error, including testing the center model by using test data, and if the error of the center model is larger than the expected error and the difference between the error of the center model and the expected error is larger than a first threshold, judging that the current system has the poisoning sign. Wherein the expected error can be determined according to the past error change speed of the model.
And if the poisoning evidence does not exist, carrying out model mirroring on the central model according to a preset strategy. In practice, the predetermined strategy may be timed mirroring or model mirroring with a certain rate of error reduction. And after the model is mirrored, each client node is informed to empty the cache and restart the data cache. The client node stores the data of the last nontoxic mirror image at the current moment through data caching, and when the poisoning sign of the system is judged to appear, whether the node is poisoned can be quickly and accurately judged according to the data cached by the client node.
If the current system is judged to have the poisoning sign, whether a poisoning node exists needs to be determined, and the poisoning node is forced to be off-line, so that the accuracy of the model is ensured.
Firstly, calculating the risk probability of a risk node according to the difference between the error of the center model and the expected error, wherein the risk probability calculating step comprises the following steps:
determining a risk level value according to the difference between the error of the center model and an expected error;
specifically, according to the difference between the error of the central model and the expected error, the risk can be divided into different grade values, the specific grade value division can be divided according to actual needs, and the larger the difference is, the higher the risk of poisoning is, and the larger the risk probability of the corresponding node submitting data is. The beneficial effects of a batch of data trainings normally submitted by distributed client nodes on the model can be considered independent (this is also a distributed objective, if relevant, not so many clients are needed, since the data effects they submit are the same, distributed training is only necessary if independent effects are the same), then the beneficial effects of each batch of data trainings, such as the magnitude of improvement of the central model loss function, follow a normal distribution-e.g., so the risk level values can be divided according to the standard positive-too distribution: when the deviation between the error of the model and the expected error is more than or equal to 2.1 percent and less than 13.5 percent, the risk grade value is 1; when the deviation is more than or equal to 13.5% and less than 34%, the risk grade value is 2; if the deviation is greater than or equal to 34%, the risk grade value is divided into 3. And when the deviation is less than 2.1, judging that the current state is a normal state and no toxic sign exists.
The risk probability of each risk node is the risk level value divided by the number of risk nodes. Since it is not yet possible to determine which nodes are poisoned nodes, the risk probability is shared per risk node.
When the inspection is carried out, the risk probability is not only considered, but the previous risk probabilities are added through the accumulated risk probability, and the risk node needing the inspection is determined. By considering the historical risk probability accumulated risk probability of each node, the higher the accumulated risk probability is, the node is in a susceptible environment, the higher the possibility of virus exposure is, extra attention is needed, and therefore the accuracy and the efficiency of detection are improved.
Specifically, patrol and examine risk node according to accumulative risk probability, include:
and acquiring all risk nodes with the accumulated risk probability larger than a second threshold, and sequencing the risk nodes from large to small according to the accumulated risk probability. The second threshold value may be set according to actual needs, and may be set to 0.5, for example.
And sequentially polling each risk node.
And if the ith risk node is judged to be a poisoned node through inspection, forcibly offline the risk node, and continuously inspecting the (i + 1) th to nth risk nodes until encountering a first non-poisoned risk node, and finishing inspection.
Specifically, patrol and examine risk node includes:
acquiring original data, intermediate data, a processing algorithm and submitted data cached by a risk node;
and processing the original data according to the processing algorithm to obtain actual intermediate data and result data, comparing the obtained actual intermediate data and result data with intermediate data and submitted data cached by the risk node, and if the intermediate data or the submitted data have data loss, data errors, data timestamps or data check marks which are not consistent, judging the risk node as a poisoned node.
Specifically, the original data cached by the client node is the input data received by the client node, the processing algorithm is an algorithm for processing the original data, the intermediate data is generated when data processing is performed according to the processing algorithm, and the submitted data is data submitted to the server by the client node.
Considering the efficiency and comprehensiveness of deep inspection, starting from the original data and processing the data according to a processing algorithm from the lower part of a data chain to obtain actual intermediate data and result data, and comparing the obtained actual intermediate data and result data with intermediate data and submitted data cached by risk nodes to quickly and comprehensively check whether the nodes are poisoned. Meanwhile, the inspection can eliminate the poisoning interference of the client node, comprehensive data can be used for inspection, and the inspection rate is high.
When the air risk node is detected to be a poisoned node, the risk node can be forced to be offline. The poisoned node can be permanently off-line, and the off-line time can be set according to the times of the poisoned node as the risk node and the accumulated risk probability. Specifically, the offline time of the poisoned node may be calculated by the following formula:
Time of offlineofflineIs the base off-line TimebaselineThe geometric progression comprises the product of the number of times of risk nodes and the accumulated risk probability, obviously, the number of times of risk nodes is greater than 1, the higher the number of times of risk nodes and the accumulated risk probability, the great delay of the offline time of the node can be regulated and controlled by a system administrator, the vulnerable nodes attacked for many times can be rapidly turned into indefinite (for example, the offline time exceeds 10 years) blocking nodes, a client user is informed to update or kill viruses of the system, and the network is allowed to be accessed again only after the offline time is reached.
One embodiment of the present invention discloses a poison prevention system of a distributed AI system, as shown in fig. 2, including the following modules:
the central model updating module is used for receiving submitted data sent by each client node, updating the central model based on the submitted data, judging whether the current system has a poisoning sign according to the difference between the error of the central model and the expected error, if not, mirroring the central model according to a preset strategy, informing each client node to empty cache and re-cache data, wherein the cached data comprises the original data, the processing algorithm and the intermediate data of the client node;
and the inspection module is used for acquiring all client nodes which submit data from the last model mirror image to the current moment when the central model updating module judges whether the current system has the poisoning sign, taking the client nodes as risk nodes, calculating the risk probability of the risk nodes according to the difference between the error of the central model and the expected error, acquiring the accumulated risk probability of the risk nodes based on the risk probability, inspecting the risk nodes according to the accumulated risk probability, taking the poisoning nodes off line if the poisoning nodes exist, and recovering the central model into the model of the last mirror image.
Preferably, the central model updating module is configured to determine whether the current system has a sign of poisoning according to a difference between an error of the central model and an expected error, where the determining includes testing the central model by using test data, and determining that the current system has the sign of poisoning if the error of the central model is greater than the expected error and a difference between the error of the central model and the expected error is greater than a first threshold.
Preferably, the polling module is used for polling the risk nodes according to the accumulated risk probability, and comprises:
acquiring all risk nodes with the accumulated risk probability larger than a second threshold value, and sequencing the risk nodes from large to small according to the accumulated risk probability;
sequentially polling each risk node;
and if the ith risk node is judged to be a poisoned node through inspection, forcibly offline the risk node, and continuously inspecting the (i + 1) th to nth risk nodes until encountering a first non-poisoned risk node, and finishing inspection.
Preferably, the polling of the polling module includes:
the data acquisition module is used for acquiring original data, intermediate data, a processing algorithm and submitted data cached by the risk node;
and the data comparison module is used for processing the original data according to the processing algorithm to obtain actual intermediate data and submitted data, comparing the obtained actual intermediate data and submitted data with intermediate data and submitted data cached by the risk node, and judging the risk node as a poisoned node if the intermediate data or the submitted data have data loss, data errors, data timestamps or data check marks which are not consistent.
The method embodiment and the device embodiment are based on the same principle, and the related parts can be referenced mutually, and the same technical effect can be achieved. For a specific implementation process, reference is made to the method embodiment, which is not described herein again.
Those skilled in the art will appreciate that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program, which is stored in a computer readable storage medium, to instruct related hardware. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.
Claims (10)
1. A virus exposure prevention method for a distributed AI system is characterized by comprising the following steps:
receiving submitted data sent by each client node, updating a center model based on the submitted data, judging whether the current system has a poisoning sign according to the difference between the error of the center model and the expected error, if not, mirroring the center model according to a preset strategy, informing each client node to empty cache and re-cache data, wherein the cache data comprises the original data, the processing algorithm and the intermediate data of the client node;
otherwise, acquiring all client nodes submitting data from the last model mirror image to the current moment, taking the client nodes as risk nodes, calculating the risk probability of the risk nodes according to the difference between the error of the central model and the expected error, acquiring the accumulated risk probability of the risk nodes based on the risk probability, inspecting the risk nodes according to the accumulated risk probability, taking the poisoned nodes off line if the poisoned nodes exist, and recovering the central model to the model of the last mirror image.
2. The method of claim 1, wherein determining whether the current system has a sign of poisoning according to a difference between the error of the central model and an expected error comprises testing the central model with test data, and determining that the current system has a sign of poisoning if the error of the central model is greater than the expected error and the difference between the error of the central model and the expected error is greater than a first threshold.
3. The anti-virus-exposure method for the distributed AI system according to claim 1, wherein the patrol of the risk nodes according to the cumulative risk probability comprises:
acquiring all risk nodes with the accumulated risk probability larger than a second threshold value, and sequencing the risk nodes from large to small according to the accumulated risk probability;
sequentially polling each risk node;
and if the ith risk node is judged to be a poisoned node through inspection, forcibly offline the risk node, and continuously inspecting the (i + 1) th to nth risk nodes until encountering a first non-poisoned risk node, and finishing inspection.
4. The poison prevention method of the distributed AI system of claim 1, wherein the polling content includes:
acquiring original data, intermediate data, a processing algorithm and submitted data cached by a risk node;
and processing the original data according to the processing algorithm to obtain actual intermediate data and result data, comparing the obtained actual intermediate data and result data with intermediate data and submitted data cached by the risk node, and judging the risk node as a poisoned node if the intermediate data or the submitted data have data loss, data errors, data time stamps or data check marks which are not consistent.
5. The poison prevention method of the distributed AI system of claim 1, wherein the down time of the poisoned node is calculated by the following formula:
6. The poison prevention method of the distributed AI system of claim 1, wherein calculating the risk probability of a risk node according to the difference between the error of the central model and the expected error comprises:
determining a risk level value according to the difference between the error of the center model and an expected error;
the risk probability of each risk node is the risk level value divided by the number of risk nodes.
7. The utility model provides a distributed AI system's antitoxic system which characterized in that includes following module:
the central model updating module is used for receiving submitted data sent by each client node, updating a central model based on the submitted data, judging whether the current system has a poisoning sign according to the difference between the error of the central model and the expected error, and if not, mirroring the central model according to a preset strategy and informing each client node to empty the cache and perform data caching again;
and the inspection module is used for acquiring all client nodes submitting data from the last model mirror image to the current moment when the central model updating module judges that the current system has the poisoning sign, taking the client nodes as risk nodes, calculating the risk probability of the risk nodes according to the difference between the error of the central model and the expected error, acquiring the accumulated risk probability of the risk nodes based on the risk probability, inspecting the risk nodes according to the accumulated risk probability, taking the poisoning nodes off line if the poisoning nodes exist, and recovering the central model into the model of the last mirror image.
8. The poison prevention system of the distributed AI system of claim 7, wherein the central model update module is configured to determine whether the current system has a sign of poisoning according to a difference between an error of the central model and an expected error, and the determination includes testing the central model with test data, and determining that the current system has the sign of poisoning if the error of the central model is greater than the expected error and the difference between the error of the central model and the expected error is greater than a first threshold.
9. The system of claim 7, wherein the patrol module is configured to patrol the risk nodes according to the cumulative risk probability, and comprises:
acquiring all risk nodes with the accumulated risk probability larger than a second threshold value, and sequencing the risk nodes from large to small according to the accumulated risk probability;
sequentially polling each risk node;
and if the ith risk node is judged to be a poisoned node through inspection, forcibly offline the risk node, and continuously inspecting the (i + 1) th to nth risk nodes until encountering a first non-poisoned risk node, and finishing inspection.
10. The anti-virus exposure system of the distributed AI system of claim 7, wherein the routing inspection module includes:
the data acquisition module is used for acquiring original data, intermediate data, a processing algorithm and submitted data cached by the risk node;
and the data comparison module is used for processing the original data according to the processing algorithm to obtain actual intermediate data and submitted data, comparing the obtained actual intermediate data and submitted data with intermediate data and submitted data cached by the risk node, and judging the risk node as a poisoned node if the intermediate data or the submitted data have data loss, data errors, data timestamps or data check marks which are not consistent.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111247976.5A CN113987477B (en) | 2021-10-26 | 2021-10-26 | Anti-poisoning method and system for distributed AI system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111247976.5A CN113987477B (en) | 2021-10-26 | 2021-10-26 | Anti-poisoning method and system for distributed AI system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113987477A true CN113987477A (en) | 2022-01-28 |
| CN113987477B CN113987477B (en) | 2024-07-19 |
Family
ID=79741644
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111247976.5A Active CN113987477B (en) | 2021-10-26 | 2021-10-26 | Anti-poisoning method and system for distributed AI system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113987477B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118250094A (en) * | 2024-05-27 | 2024-06-25 | 广州优刻谷科技有限公司 | Malicious node identification method, system and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102934122A (en) * | 2010-05-07 | 2013-02-13 | 阿尔卡特朗讯公司 | Method for adapting security policies of an information system infrastructure |
| US20130081134A1 (en) * | 2011-09-24 | 2013-03-28 | Daniel A. Gerrity | Instruction set adapted for security risk monitoring |
| CN103942726A (en) * | 2014-03-26 | 2014-07-23 | 国家电网公司 | Intelligent inspection method for condition evaluation work of power grid equipment |
| CN108667799A (en) * | 2018-03-28 | 2018-10-16 | 中国科学院信息工程研究所 | It is a kind of to be directed to the defence method and system that browser rs cache is poisoned |
| US20190156426A1 (en) * | 2015-02-04 | 2019-05-23 | Riv Data Corp. | Systems and methods for collecting and processing alternative data sources for risk analysis and insurance |
| CN111027715A (en) * | 2019-12-11 | 2020-04-17 | 支付宝(杭州)信息技术有限公司 | Monte Carlo-based federated learning model training method and device |
| CN111914256A (en) * | 2020-07-17 | 2020-11-10 | 华中科技大学 | Defense method for machine learning training data under toxic attack |
| CN112163638A (en) * | 2020-10-20 | 2021-01-01 | 腾讯科技（深圳）有限公司 | Defense method, device, equipment and medium for image classification model backdoor attack |
| CN112487475A (en) * | 2020-11-30 | 2021-03-12 | 北京京航计算通讯研究所 | Method and system for risk analysis of secret-related carrier |
-
2021
- 2021-10-26 CN CN202111247976.5A patent/CN113987477B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102934122A (en) * | 2010-05-07 | 2013-02-13 | 阿尔卡特朗讯公司 | Method for adapting security policies of an information system infrastructure |
| US20130081134A1 (en) * | 2011-09-24 | 2013-03-28 | Daniel A. Gerrity | Instruction set adapted for security risk monitoring |
| CN103942726A (en) * | 2014-03-26 | 2014-07-23 | 国家电网公司 | Intelligent inspection method for condition evaluation work of power grid equipment |
| US20190156426A1 (en) * | 2015-02-04 | 2019-05-23 | Riv Data Corp. | Systems and methods for collecting and processing alternative data sources for risk analysis and insurance |
| CN108667799A (en) * | 2018-03-28 | 2018-10-16 | 中国科学院信息工程研究所 | It is a kind of to be directed to the defence method and system that browser rs cache is poisoned |
| CN111027715A (en) * | 2019-12-11 | 2020-04-17 | 支付宝(杭州)信息技术有限公司 | Monte Carlo-based federated learning model training method and device |
| CN111914256A (en) * | 2020-07-17 | 2020-11-10 | 华中科技大学 | Defense method for machine learning training data under toxic attack |
| CN112163638A (en) * | 2020-10-20 | 2021-01-01 | 腾讯科技（深圳）有限公司 | Defense method, device, equipment and medium for image classification model backdoor attack |
| CN112487475A (en) * | 2020-11-30 | 2021-03-12 | 北京京航计算通讯研究所 | Method and system for risk analysis of secret-related carrier |
Non-Patent Citations (3)
| Title |
|---|
| MICHAEL CHARY等: "Diagnosis of Acute Poisoning using explainable artificial intelligence", pages 1 - 6, Retrieved from the Internet <URL:《网页在线公开：https://www.sciencedirect.com/science/article/pii/S0010482521002638》> * |
| TONG ZHANG等: "A NFT verification method for distributed AI system", pages 1 - 6, Retrieved from the Internet <URL:《网页在线公开：A NFT verification method for distributed AI system》> * |
| 周纯毅等: "分布式深度学习隐私与安全攻击研究进展与挑战", 《计算机研究与发展》, vol. 58, no. 5, 1 July 2021 (2021-07-01), pages 927 - 943 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118250094A (en) * | 2024-05-27 | 2024-06-25 | 广州优刻谷科技有限公司 | Malicious node identification method, system and storage medium |
| CN118250094B (en) * | 2024-05-27 | 2024-08-27 | 广州优刻谷科技有限公司 | Malicious node identification method, system and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113987477B (en) | 2024-07-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11201882B2 (en) | Detection of malicious network activity | |
| CN108494746B (en) | Method and system for detecting abnormal flow of network port | |
| US7617170B2 (en) | Generated anomaly pattern for HTTP flood protection | |
| US7624084B2 (en) | Method of generating anomaly pattern for HTTP flood protection | |
| Chen et al. | An efficient network intrusion detection | |
| CN101841533B (en) | Method and device for detecting distributed denial-of-service attack | |
| WO2005048022A2 (en) | Method and system for addressing intrusion attacks on a computer system | |
| CN110351291B (en) | DDoS attack detection method and device based on multi-scale convolutional neural network | |
| KR100950079B1 (en) | Network abnormal state detection device using HMMHidden Markov Model and Method thereof | |
| CN113987477A (en) | Distributed AI system poison-throwing prevention method and system | |
| KR102433830B1 (en) | System and method for security threats anomaly detection based on artificial intelligence | |
| CN111510434A (en) | Network intrusion detection method, system and related equipment | |
| CN111628961A (en) | DNS (Domain name Server) anomaly detection method | |
| Kemp et al. | An approach to application-layer DoS detection | |
| von der Assen et al. | Ransomai: Ai-powered ransomware for stealthy encryption | |
| CN111131309A (en) | Distributed denial of service detection method and device and model creation method and device | |
| CN118300810A (en) | Attack detection method, apparatus, device, storage medium, and computer program product | |
| Ogawa et al. | Malware originated http traffic detection utilizing cluster appearance ratio | |
| CN116994167A (en) | Website security monitoring method based on machine learning algorithm | |
| CN115659351B (en) | Information security analysis method, system and equipment based on big data office | |
| Eswari et al. | DDoS Attacks in Traffic Flow Streams Using Ensemble Classifiers | |
| CN115632832B (en) | Big data attack processing method and system applied to cloud service | |
| Yanchun | The intrusion detection system based on fuzzy association rules mining | |
| Aung et al. | Software rejuvenation approach to security engineering | |
| CN118395439B (en) | Self-destruction method and device for distributed cloud system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |