CN111046379B - Anti-attack monitoring method and device - Google Patents
Anti-attack monitoring method and device Download PDFInfo
- Publication number
- CN111046379B CN111046379B CN201911242921.8A CN201911242921A CN111046379B CN 111046379 B CN111046379 B CN 111046379B CN 201911242921 A CN201911242921 A CN 201911242921A CN 111046379 B CN111046379 B CN 111046379B
- Authority
- CN
- China
- Prior art keywords
- input data
- attack
- target model
- sample space
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
Abstract
The specification discloses a method and apparatus for monitoring against attacks. The method comprises the following steps: obtaining a confrontation sample space of a target model; collecting input data for calling the target model; determining whether the input data falls within the countermeasure sample space; and calculating monitoring parameters of the input data falling into the antagonistic sample space in the monitoring period according to the judgment result, and determining to monitor the antagonistic attack facing the target model when the monitoring parameters meet preset attack conditions. According to the scheme, the attack resistance can be effectively monitored, the security risks such as privacy disclosure, fund loss and the like are effectively reduced, and the data security is ensured.
Description
Technical Field
The present disclosure relates to the field of artificial intelligence, and in particular, to a method and an apparatus for monitoring attack countermeasures.
Background
With the continuous development of artificial intelligence, machine learning models are more and more complex, and the accuracy is higher and higher. However, the more accurate the model, the less robust the model may be, i.e. the less robust the model is, which creates an opportunity for an attack.
Taking the counterattack as an example, an attacker slightly modifies the samples to form countersamples and inputs the countersamples into the model, so that the model outputs an incorrect prediction result. For example, for a scene that identity authentication is performed by face recognition, an attacker constructs a countermeasure sample and inputs the countermeasure sample into a face recognition model, and if the model recognizes the countermeasure sample as a legitimate user, the attacker can perform the identity authentication to bring security risks such as private data leakage and capital loss.
Disclosure of Invention
In view of the above, the present specification provides a method and apparatus for monitoring against attacks.
Specifically, the description is realized by the following technical scheme:
a method of monitoring against an attack, comprising:
obtaining a confrontation sample space of a target model;
collecting input data for calling the target model;
determining whether the input data falls within the countermeasure sample space;
and calculating monitoring parameters of the input data falling into the antagonistic sample space in the monitoring period according to the judgment result, and determining to monitor the antagonistic attack facing the target model when the monitoring parameters meet preset attack conditions.
A surveillance device for countering an attack, comprising:
an acquisition unit that acquires a countermeasure sample space of a target model;
the acquisition unit is used for acquiring input data for calling the target model;
a judging unit that judges whether the input data falls into the countermeasure sample space;
and the monitoring unit is used for calculating monitoring parameters of the input data falling into the antagonistic sample space in the monitoring period according to the judgment result, and determining to monitor the antagonistic attack facing the target model when the monitoring parameters meet the preset attack conditions.
A surveillance device for countering an attack, comprising:
a processor;
a memory for storing machine executable instructions;
wherein, by reading and executing machine-executable instructions stored by the memory that correspond to monitoring logic against an attack, the processor is caused to:
obtaining a confrontation sample space of a target model;
collecting input data for calling the target model;
determining whether the input data falls within the countermeasure sample space;
and calculating monitoring parameters of the input data falling into the antagonistic sample space in the monitoring period according to the judgment result, and determining to monitor the antagonistic attack facing the target model when the monitoring parameters meet preset attack conditions.
One embodiment of the specification realizes that input data of a calling target model is collected, whether the input data fall into an antagonistic sample space of the target model is judged, monitoring parameters of the input data falling into the antagonistic sample space in a monitoring period are calculated according to a judgment result, and if the monitoring parameters meet attack conditions, the monitoring of the antagonistic attack facing the target model is confirmed. The method does not influence the normal use of the target model, can monitor the counterattack in time, and effectively reduces the security risks of private data leakage, capital loss and the like.
Drawings
Fig. 1 is a flow chart illustrating a method of monitoring against an attack in an exemplary embodiment of the present description.
Fig. 2 is a flow chart of another monitoring method for fighting attacks according to an exemplary embodiment of the present disclosure.
Fig. 3 is a flowchart illustrating a method for obtaining a target model versus a sample space according to an exemplary embodiment of the present disclosure.
Fig. 4 is a flow chart illustrating another method of anti-attack monitoring according to an exemplary embodiment of the present disclosure.
Fig. 5 is a schematic structural diagram of a device for monitoring attack resistance according to an exemplary embodiment of the present disclosure.
Fig. 6 is a block diagram of a counter attack monitoring device according to an exemplary embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the specification, as detailed in the appended claims.
The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the description. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of the present specification. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
With the continuous development of artificial intelligence, researchers are continuously designing deeper and more complex machine learning models so that the models output more accurate prediction results. However, as the accuracy of the model is improved, the robustness of the model may be worse and worse, which makes the model vulnerable.
Taking the counterattack as an example, the counterattack sample is formed by slightly modifying the sample, and the counterattack sample is input into the model so that the model can output an incorrect prediction result. For example, in an image recognition model, such subtle modifications may be adding some disturbing noise to the image. After the modified image is input into the image recognition model, the image recognition model may recognize a picture of a puppy as a picture of a car, resulting in outputting a completely erroneous recognition result. Counterattacks may exist in the fields of image recognition, speech recognition, text recognition, etc.
In some scenarios, fighting an attack may pose a security risk. For example, for a scene of identity authentication by face recognition, an attacker constructs a countermeasure sample and inputs the countermeasure sample into a face recognition model, and if the face recognition model recognizes the countermeasure sample as a legal user, the attacker can pass the identity authentication, thereby bringing security risks such as private data leakage and capital loss.
The specification provides a method and a device for monitoring attack resistance.
Fig. 1 is a flow chart illustrating a method of monitoring against an attack in an exemplary embodiment of the present description.
The monitoring method for resisting attacks can be applied to electronic devices with processors and memories, such as servers or server clusters, and the like, and the description does not make specific limitation.
Referring to fig. 1, the method for monitoring attack defense may include the following steps:
step 101, obtaining a confrontation sample space of a target model.
In this specification, in the application scene dimension, the target model may be a speech recognition model, an image recognition model, a character recognition model, or the like; in the dimension of the model structure, the target model may be a model based on a neural network, and the like, and this specification does not particularly limit this.
In this specification, the confrontation sample space may be obtained through pre-calculation after the training of the target model is completed and before the target model is formally brought online. Of course, the confrontation sample space can also be calculated after the target model is on line, and the description does not specially limit this.
In the present specification, a challenge sample may be obtained through an attack test, and a challenge sample space may be generated from the challenge sample.
In one example, the attack test may be a black box test based on a boundary attack.
The boundary attack means that a countermeasure sample with large interference is constructed firstly to test a target model, the interference of the sample is continuously reduced on the premise of ensuring the countermeasure, and finally the countermeasure sample with small interference is obtained.
In practical applications, when a challenge sample is generated based on an original image, a more disturbing challenge sample may be generated first. For example, the pixel values of some pixel points on the original image may be randomly changed, the modified original image is input into the target model, and if the target model outputs a prediction result of misjudgment, the modified image is used as a countermeasure sample. After obtaining the countermeasure sample, randomly disturbing the countermeasure sample along a direction close to the original image by taking the countermeasure sample as a starting point in the space according to the space coordinate of the countermeasure sample and the space coordinate of the original image, and continuously reducing the distance between the disturbed countermeasure sample and the original image on the premise of ensuring the countermeasure of the countermeasure sample.
For example, the perturbed challenge sample may be input into the target model, and if the target model outputs a wrong prediction result, which indicates that the challenge sample still has the challenge, the challenge sample may be further perturbed randomly in the above direction, so that the challenge sample is closer to the original image, and finally the challenge sample closest to the original image is obtained, that is, the challenge sample with the smallest interference is obtained. By adopting the method, a plurality of confrontation samples of the target model can be obtained.
The present specification states that the challenge sample can be constructed by other methods, and the present specification does not specifically limit this.
In another example, the attack test may also be a white-box test based on a boundary attack. The steps of the white box test refer to the steps of the black box test described above, and are not described in detail herein.
It should be noted that the white box test requires to acquire a complete target model file in advance, and the target model file may include the structure and parameters of the target model. In this specification, a challenge sample space of the target model may be determined based on the challenge samples.
In one example, spatial coordinates of each challenge sample of the target model may be determined, and a challenge sample space of the target model may be determined based on the spatial coordinates.
Taking the target model as an image recognition model as an example, assuming that a pair of confrontation samples are color images with pixels of 64 × 64, the confrontation samples have 64 × 64 pixels, each pixel has 3 pixel values, and the confrontation samples have 64 × 3 — 12288 pixel values in total, the spatial coordinates of the confrontation samples of the image recognition model have 12288 dimensions, that is, the confrontation sample space has 12288 dimensions, and the value of each dimension is a certain pixel value of the pixel corresponding to the confrontation sample.
For example, the first dimension of the challenge sample space may represent the 1 st pixel value of the first pixel point of the challenge sample; the second dimension of the confrontation sample space can represent the 2 nd pixel value of the first pixel point of the confrontation sample; the third dimension of the confrontation sample space can represent the 3 rd pixel value of the first pixel point of the confrontation sample; the fourth dimension of the challenge sample space may represent the 1 st pixel value … … of the second pixel of the challenge sample, and so on.
Clustering the confrontation samples based on the space coordinates of the confrontation samples to obtain a plurality of confrontation sample clusters. The Clustering algorithm may be a K-Means algorithm, a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, etc., and the description does not specifically limit this.
In this example, the several antagonizing sample clusters can be taken as the antagonizing sample space.
In another example, after obtaining the countermeasure sample clusters, a corresponding convex hull may be generated for each countermeasure sample cluster, and the generated convex hulls are used as the countermeasure sample space. The convex envelope calculation method may be a Graham algorithm, a Melkman algorithm, an Andrew algorithm, and the like, and this specification does not limit this.
And 102, acquiring input data for calling the target model.
After the model is online, the target model may provide an Application Programming Interface (API) Interface for the caller, so that the caller may call the target model according to the API Interface. And collecting input data when the model calling party calls the model. For example, for an image recognition model, the input data may be an image; for a speech recognition model, the input data may be a segment of speech.
In one example, input data for the target model may be collected in real-time. For example, the calling of the target model can be monitored, and when the target model is monitored to be called, the input data input by the calling party is acquired.
In another example, historical input data of the target model may be collected periodically at preset time intervals, which may be the following monitoring periods against attacks.
It is worth mentioning that step 101 may also follow step 102. For example, if the historical input data of the target model is collected periodically in step 102, the confrontation sample space of the target model may be obtained after the historical input data of the target model is collected, and step 103 is performed.
Step 103, judging whether the input data falls into the confrontation sample space.
In one example, spatial coordinates of the input data may be determined, and a determination may be made as to whether the spatial coordinates fall within the antagonistic sample space of the target model.
In one example, the spatial coordinates may be input into a preset fitting function, and then it is determined whether the spatial coordinates fall into any one of the convex envelopes according to the output result. For example, if the spatial coordinate is x and the fitting function is F, x may be input into F to obtain F (x), and if F (x) <0, it is determined that the convex envelope is included, otherwise, it is determined that the convex envelope is not included. If the spatial coordinates fall within any one of the convex envelopes, the spatial coordinates fall within the antagonistic sample space of the target model.
In another example, the distance between the input data and each confrontation sample cluster can be calculated according to the space coordinates, and whether the distance between the input data and each confrontation sample cluster is smaller than a preset distance threshold value or not can be judged. For example, the distance of the input data from the center point of each antagonistic sample cluster may be calculated as the distance of the input data from the corresponding antagonistic sample cluster.
And if a confrontation sample cluster exists, and the distance between the input data and the confrontation sample cluster is smaller than the preset distance threshold, confirming that the input data falls into a confrontation sample space.
The distance threshold may be predetermined.
And 104, calculating monitoring parameters of the input data falling into the confrontation sample space in the monitoring period according to the judgment result, and determining to monitor the confrontation attack facing the target model when the monitoring parameters meet preset attack conditions.
In one example, the monitoring parameter is a quantity of input data falling within the challenge sample space, and the attack condition is that the quantity reaches a quantity threshold. In practical applications, it may be monitored whether the amount of the input data falling into the confrontation sample space reaches the amount threshold within a preset monitoring period. And if the quantity threshold is reached, determining that the counterattack facing the target model is monitored.
The number threshold may be determined in a manner that: and taking the average number of the input data falling into the confrontation sample space of the target model in a plurality of historical monitoring periods as a number threshold value.
For example, assuming that the monitoring period is 2 hours, and the average number of input data that fall into the challenge sample space in every two hours for the last 3 days of the target model is 200, 200 may be taken as the number threshold. It is noted that, considering that the calling requirements of the calling party to the target model may be different at different time periods in a day, the differentiated number threshold determination may also be performed on the monitoring period.
For another example, in consideration of the existence of the error, the number threshold may be multiplied by a preset error coefficient, and the calculated value may be used as the final number threshold.
For another example, the number threshold may also be set manually.
In another example, the monitoring parameter may also be a proportion of the input data falling within the challenge sample space, and the attack condition may be that the proportion reaches a proportion threshold.
In practical application, it may be monitored whether the ratio of the amount of the input data falling into the confrontation sample space to the amount of all the input data in the detection period reaches a ratio threshold value within a preset monitoring period. And if the proportion threshold is reached, confirming and monitoring the counterattack facing the target model.
The determination method of the proportional threshold refers to the number threshold, and is not described herein again.
As can be seen from the above description, in an embodiment of the present specification, an attack test may be performed on a target model to obtain a plurality of countermeasure samples of the target model, and the plurality of countermeasure samples may be calculated to obtain a countermeasure sample space.
When the target model is subjected to anti-attack monitoring, input data for calling the target model can be collected, whether the input data fall into a pre-calculated anti-sample space or not is judged, monitoring parameters of the input data falling into the anti-sample space in a monitoring period are calculated according to a judgment result, and if the monitoring parameters meet attack conditions, the anti-attack facing the target model is considered to be monitored. The method does not affect the normal use of the target model, and can monitor the counterattack.
Fig. 2 is a flow chart of another monitoring method for fighting attacks according to an exemplary embodiment of the present disclosure.
The monitoring method for resisting attacks can be applied to electronic devices with processors and memories, such as servers or server clusters, and the like, and the description does not make specific limitation.
Referring to fig. 2, the method for monitoring attack defense may include the following steps:
in step 201, a confrontation sample space of the target model is obtained.
Step 202, collecting input data for calling the target model.
Step 203, determine whether the input data falls into the confrontation sample space.
And 204, calculating monitoring parameters of the input data falling into the confrontation sample space in the monitoring period according to the judgment result, and determining to monitor the confrontation attack facing the target model when the monitoring parameters meet preset attack conditions.
Please refer to steps 101 to 104 in steps 201 to 204, which are not described herein.
Step 205, sending the alarm information.
When the monitoring parameters meet the preset attack conditions, after the counterattack facing the target model is determined to be monitored, alarm information can be sent.
In one example, the alert information may include the current monitoring period, the amount/proportion of input data that falls into the countermeasure space, and the like.
For example, the alarm information may be: "223 suspicious input data, suspected of being present against the attack, were monitored within 10 minutes". If the number of the input data falling into the countermeasure space still rises, the number/proportion of the suspicious input data can be updated, and the alarm is continuously given.
In another example, the alarm information may further include an identification of the target model caller corresponding to the input data, where the identification may be an ID, a name, an IP address, and the like of the caller.
For example, the alarm information may be: 223 suspicious inputs were monitored in "10 minutes, suspected to exist against the attack. Of these, 80% of the suspect input data comes from user a. The "caller identification information" can be obtained from the call log in the target model call process.
In another example, the alarm information may further include a prediction result of the target model on the input data falling into the challenge sample space to determine whether the challenge attack is successful.
For example, if an attacker attempts to input an image of an illegal user to which interference is added into a target model, and a prediction result output by the target model is a legal user, the alarm information may be: 223 suspicious inputs were monitored in "10 minutes, suspected to exist against the attack. Wherein, the output results of 220 input data are illegal users, and the output results of 2 input data are legal users. And judging whether the attack is successfully resisted according to the prediction result output by the target model.
As can be seen from the above description, in another embodiment of the present specification, after the existence of the target-oriented countermeasure attack is detected, an alarm message may be sent. The alarm information can show the attack times and the attack results of the counterattack, can also trace back to the attack source, and can subsequently take some measures to resist the counterattack according to the alarm information. For example, the call of a suspicious caller is intercepted, and the like, so that the security risks of private data leakage, fund loss and the like are effectively reduced.
The method for monitoring against attacks in the present specification is described below with reference to a specific embodiment.
The monitoring method for resisting attacks can be applied to a server.
Referring to fig. 3 and fig. 4, the method for monitoring attack includes two processes: carrying out attack test on the target model to obtain a confrontation sample space; input data of the target model is monitored to monitor for a counter attack.
Fig. 3 is a flowchart illustrating a method for obtaining a target model versus a sample space according to an exemplary embodiment of the present disclosure.
In this embodiment, the target model is a face recognition model for user identity authentication.
Step 301, a face recognition model is called.
In this embodiment, an explanatory document of a calling mode of the face recognition model and a calling interface need to be acquired.
Step 302, a black box test based on boundary attack is performed on the face recognition model to obtain a plurality of confrontation samples.
The method includes the steps of conducting attack testing on a face recognition model, wherein the attack testing is black box testing based on boundary attack, firstly constructing a face image with high interference as an antagonistic sample, inputting the face image into the face recognition model, and continuously reducing the interference of the antagonistic sample on the premise of ensuring the antagonism according to a result output by the face recognition model to finally obtain a plurality of antagonistic samples with low interference. In this embodiment, the interference of the countercheck sample may be noise increase on the face image, adjustment of a pixel value of a specific pixel point, or the like.
Determining the space coordinates of a plurality of confrontation samples, and clustering by using a K-Means algorithm based on the space coordinates to obtain a plurality of confrontation sample clusters. And generating a corresponding convex envelope for each confrontation sample cluster based on a Graham algorithm, and taking the generated convex envelopes as confrontation sample space of the face recognition model.
Fig. 4 is a flow chart illustrating another method of anti-attack monitoring according to an exemplary embodiment of the present disclosure.
Step 401, deploying a face recognition model.
Step 402, obtaining a confrontation sample space of the face recognition model.
And step 403, acquiring an input image for calling the face recognition model.
In this embodiment, an input image of the face recognition model is acquired in real time.
Step 404, determine whether the input image falls into the confrontation sample space.
In this embodiment, the coordinates of the input image are calculated, and whether the coordinates fall into any one convex envelope is determined based on a preset fitting function.
And 405, calculating the proportion of the input image falling into the confrontation sample space in the monitoring period according to the judgment result.
In this embodiment, in a preset monitoring period, the input images of the face recognition model are collected in real time, and when one input image is collected, step 404 is executed, if the input image falls into the countermeasure sample space as a result of the determination, the suspicious input image is counted by +1, and if the input image does not fall into the countermeasure sample space as a result of the determination, the safe input image is counted by + 1.
And 406, if the proportion reaches a proportion threshold value, determining to monitor the counter attack facing the face recognition model.
In this embodiment, the proportional threshold may be obtained according to historical input data of the face recognition model, for example, by statistics: the face recognition model has an average hourly input image falling convex envelope of 0.05 over the last 30 days. The ratio threshold is determined to be 0.05 with a monitoring period of 1 hour.
In the monitoring period, whether the proportion of the input image falling into the convex envelope is larger than the proportion threshold value 0.05 or not can be judged in real time. For example, the number of suspicious input images detected in step 405 may be divided by the sum of the number of suspicious input images and the number of security input images, to determine whether the ratio of the obtained suspicious input images is greater than 0.05, and if the ratio is greater than 0.05, it is determined that the anti-attack is detected.
Step 407, sending the alarm information.
Upon detection of a challenge attack, an alert message may be sent. In this embodiment, the alarm information may include the current monitoring period, the proportion of the input image falling into the convex envelope, the identifier of the caller of the face recognition model, and the like. The following table shows an example of the alarm information by way of example:
the table shows the proportion of the input images suspected to resist the attack, the calling party identification with more calling times and the corresponding calling times in the current monitoring period, and comprehensively reflects the attack condition of the face recognition model in the current monitoring period.
For example, the alarm information in the table is used, the suspicious input images input by the user a in the current monitoring period are the most, and in order to prevent counterattack, the call request of the user a may be subsequently intercepted, for example, the call request of the user a in a preset time period is intercepted.
From the above description, it can be seen that the counter attack monitoring method provided by the specification can be used for monitoring the counter attack of the face recognition model, and when the counter attack facing the face recognition model is confirmed and monitored, defense strategies such as interception and invocation can be adopted in time, so that the security risks such as private data leakage and capital loss are effectively reduced.
In correspondence with the aforementioned embodiments of the monitoring method against attacks, the present description also provides embodiments of a device for detection against attacks.
Embodiments of the apparatus for detection of attacks of the present description may be applied on a server. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. Taking a software implementation as an example, as a logical device, the device is formed by reading corresponding computer program instructions in the nonvolatile memory into the memory for operation through the processor of the server where the device is located. In terms of hardware, as shown in fig. 5, a hardware structure diagram of a server where the monitoring device for monitoring against attacks is located in this specification is shown, except for the processor, the memory, the network interface, and the nonvolatile memory shown in fig. 5, an electronic device where the device is located in the embodiment may also include other hardware according to an actual function of the server, which is not described again.
Fig. 6 is a block diagram of an apparatus for monitoring against an attack, according to an exemplary embodiment of the present disclosure.
Referring to fig. 6, the apparatus 600 for detecting an attack countermeasure can be applied to the server shown in fig. 5, and includes: the device comprises an acquisition unit 610, an acquisition unit 620, a judgment unit 630 and a monitoring unit 640.
The obtaining unit 610 obtains a confrontation sample space of the target model;
the acquisition unit 620 acquires input data for calling the target model;
a judging unit 630 that judges whether the input data falls into the countermeasure sample space;
and the monitoring unit 640 calculates monitoring parameters of the input data falling into the countermeasure sample space in the monitoring period according to the judgment result, and determines to monitor the countermeasure attack facing the target model when the monitoring parameters meet preset attack conditions.
Optionally, the determining unit 630:
determining spatial coordinates of the input data;
judging whether the space coordinate falls into any convex envelope or not;
if so, determining that the input data falls into the countermeasure sample space.
Optionally, the determining unit 630:
determining spatial coordinates of the input data;
judging whether the distance between the input data and the confrontation sample cluster is smaller than a threshold value or not according to the space coordinate;
if so, determining that the input data falls into the countermeasure sample space.
Optionally, the apparatus further includes an alarm unit 640, which sends alarm information.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution in the specification. One of ordinary skill in the art can understand and implement it without inventive effort.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
In correspondence with the foregoing embodiments of the monitoring method against attacks, the present specification also provides a monitoring device against attacks, the device including: a processor and a memory for storing machine executable instructions. Wherein the processor and the memory are typically interconnected by means of an internal bus. In other possible implementations, the device may also include an external interface to enable communication with other devices or components.
In this embodiment, the processor is caused to:
obtaining a confrontation sample space of a target model;
collecting input data for calling the target model;
determining whether the input data falls within the countermeasure sample space;
and calculating monitoring parameters of the input data falling into the antagonistic sample space in the monitoring period according to the judgment result, and determining to monitor the antagonistic attack facing the target model when the monitoring parameters meet preset attack conditions.
Optionally, in determining the confrontation sample space of the target model, the processor is caused to:
performing attack testing on the target model to obtain at least one countermeasure sample of the target model;
based on the challenge samples, a challenge sample space of the target model is determined.
Optionally, in performing the attack test, the processor is caused to:
carrying out black box test based on boundary attack; or
White-box testing based on boundary attacks is performed.
Optionally, in determining a challenge sample space of the target model based on the challenge samples, the processor is caused to:
determining spatial coordinates of each challenge sample;
clustering the confrontation samples based on the space coordinates to obtain a plurality of confrontation sample clusters;
generating a corresponding convex hull for each challenge sample cluster as the challenge sample space.
Optionally, in determining whether the input data falls within the countermeasure sample space, the processor is caused to:
determining spatial coordinates of the input data;
judging whether the space coordinate falls into any convex envelope or not;
if so, determining that the input data falls into the countermeasure sample space.
Optionally, in determining whether the input data falls within the confrontation sample space, the processor is caused to:
determining spatial coordinates of the input data;
judging whether the distance between the input data and any confrontation sample cluster is smaller than a distance threshold value or not according to the space coordinate;
if so, determining that the input data falls into the countermeasure sample space.
Optionally, after determining that a counter attack facing the target model is monitored, the processor is further caused to:
and sending alarm information.
In correspondence with the foregoing embodiments of the monitoring method against attacks, the present specification also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of:
obtaining a confrontation sample space of a target model;
collecting input data for calling the target model;
determining whether the input data falls within the countermeasure sample space;
and calculating monitoring parameters of the input data falling into the antagonistic sample space in the monitoring period according to the judgment result, and determining to monitor the antagonistic attack facing the target model when the monitoring parameters meet preset attack conditions.
Optionally, the determining manner of the confrontation sample space of the target model includes:
performing attack testing on the target model to obtain at least one countermeasure sample of the target model;
based on the challenge samples, a challenge sample space of the target model is determined.
Optionally, the attack test includes:
black box testing based on boundary attack; or
White-box testing based on boundary attacks.
Optionally, the determining a challenge sample space of the target model based on the challenge samples includes:
determining spatial coordinates of each challenge sample;
clustering the confrontation samples based on the space coordinates to obtain a plurality of confrontation sample clusters;
generating a corresponding convex hull for each challenge sample cluster as the challenge sample space.
Optionally, the determining whether the input data falls into the confrontation sample space includes:
determining spatial coordinates of the input data;
judging whether the space coordinate falls into any convex envelope or not;
if so, determining that the input data falls into the countermeasure sample space.
Optionally, the determining whether the input data falls into the confrontation sample space includes:
determining spatial coordinates of the input data;
judging whether the distance between the input data and any confrontation sample cluster is smaller than a distance threshold value or not according to the space coordinate;
if so, determining that the input data falls into the countermeasure sample space.
Optionally, the monitoring parameter is the amount of input data falling into the countermeasure sample space, and the attack condition is that the amount reaches an amount threshold.
Optionally, the monitoring parameter is a proportion of the input data falling into the countermeasure sample space, and the attack condition is that the proportion reaches a proportion threshold.
Optionally, after determining that the counter attack facing the target model is monitored, the method further includes:
and sending alarm information.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The above description is only a preferred embodiment of the present disclosure, and should not be taken as limiting the present disclosure, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (19)
1. A method of monitoring against an attack, comprising:
obtaining a confrontation sample space of a target model;
collecting input data for calling the target model;
determining whether the input data falls into the countermeasure sample space, the countermeasure sample space being derived based on a countermeasure sample;
and calculating monitoring parameters of the input data falling into the antagonistic sample space in the monitoring period according to the judgment result, and determining to monitor the antagonistic attack facing the target model when the monitoring parameters meet preset attack conditions.
2. The method of claim 1, wherein the determination of the confrontational sample space of the target model comprises:
performing attack testing on the target model to obtain at least one countermeasure sample of the target model;
based on the challenge samples, a challenge sample space of the target model is determined.
3. The method of claim 2, the attack testing, comprising:
black box testing based on boundary attack; or
White-box testing based on boundary attacks.
4. The method of claim 2, the determining a challenge sample space of the target model based on the challenge samples, comprising:
determining spatial coordinates of each challenge sample;
clustering the confrontation samples based on the space coordinates to obtain a plurality of confrontation sample clusters;
generating a corresponding convex hull for each challenge sample cluster as the challenge sample space.
5. The method of claim 4, said determining whether the input data falls within the confrontation sample space, comprising:
determining spatial coordinates of the input data;
judging whether the space coordinate falls into any convex envelope or not;
if so, determining that the input data falls into the countermeasure sample space.
6. The method of claim 4, said determining whether the input data falls within the confrontation sample space, comprising:
determining spatial coordinates of the input data;
judging whether the distance between the input data and any confrontation sample cluster is smaller than a distance threshold value or not according to the space coordinate;
if so, determining that the input data falls into the countermeasure sample space.
7. The method of claim 1, wherein the monitoring parameter is a quantity of input data falling into the countermeasure sample space, and the attack condition is that the quantity reaches a quantity threshold.
8. The method of claim 1, wherein the monitoring parameter is a proportion of the input data falling within the challenge sample space, and the attack condition is that the proportion reaches a proportion threshold.
9. The method of claim 1, upon determining that a counter attack is monitored against the target model, the method further comprising:
and sending alarm information.
10. A surveillance device for countering an attack, comprising:
an acquisition unit that acquires a countermeasure sample space of a target model;
the acquisition unit is used for acquiring input data for calling the target model;
a judging unit that judges whether the input data falls into the countermeasure sample space, the countermeasure sample space being obtained based on a countermeasure sample;
and the monitoring unit is used for calculating monitoring parameters of the input data falling into the antagonistic sample space in the monitoring period according to the judgment result, and determining to monitor the antagonistic attack facing the target model when the monitoring parameters meet the preset attack conditions.
11. The apparatus of claim 10, wherein the determination of the confrontation sample space of the target model comprises:
performing attack testing on the target model to obtain at least one countermeasure sample of the target model;
based on the challenge samples, a challenge sample space of the target model is determined.
12. The apparatus of claim 11, the attack test, comprising:
black box testing based on boundary attack; or
White-box testing based on boundary attacks.
13. The apparatus of claim 11, the determining, based on the antagonistic sample, an antagonistic sample space for the target model comprising:
determining spatial coordinates of each challenge sample;
clustering the confrontation samples based on the space coordinates to obtain a plurality of confrontation sample clusters;
generating a corresponding convex hull for each challenge sample cluster as the challenge sample space.
14. The apparatus according to claim 13, said judging unit:
determining spatial coordinates of the input data;
judging whether the space coordinate falls into any convex envelope or not;
if so, determining that the input data falls into the countermeasure sample space.
15. The apparatus according to claim 13, said judging unit:
determining spatial coordinates of the input data;
judging whether the distance between the input data and the confrontation sample cluster is smaller than a distance threshold value or not according to the space coordinate;
if so, determining that the input data falls into the countermeasure sample space.
16. The apparatus of claim 10, wherein the monitoring parameter is a quantity of input data falling into the countermeasure sample space, and the attack condition is that the quantity reaches a quantity threshold.
17. The apparatus of claim 10, wherein the monitoring parameter is a proportion of the input data falling into the challenge sample space, and the attack condition is that the proportion reaches a proportion threshold.
18. The apparatus of claim 10, further comprising:
and the alarm unit is used for sending alarm information.
19. A surveillance device for countering an attack, comprising:
a processor;
a memory for storing machine executable instructions;
wherein, by reading and executing machine-executable instructions stored by the memory that correspond to monitoring logic against an attack, the processor is caused to:
obtaining a countermeasure sample space of the target model, the countermeasure sample space being obtained based on the countermeasure samples;
collecting input data for calling the target model;
determining whether the input data falls within the countermeasure sample space;
and calculating monitoring parameters of the input data falling into the antagonistic sample space in the monitoring period according to the judgment result, and determining to monitor the antagonistic attack facing the target model when the monitoring parameters meet preset attack conditions.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911242921.8A CN111046379B (en) | 2019-12-06 | 2019-12-06 | Anti-attack monitoring method and device |
| TW109116402A TWI743787B (en) | 2019-12-06 | 2020-05-18 | Monitoring method and device for resisting attack |
| PCT/CN2020/118659 WO2021109695A1 (en) | 2019-12-06 | 2020-09-29 | Adversarial attack detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911242921.8A CN111046379B (en) | 2019-12-06 | 2019-12-06 | Anti-attack monitoring method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111046379A CN111046379A (en) | 2020-04-21 |
| CN111046379B true CN111046379B (en) | 2021-06-18 |
Family
ID=70233551
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911242921.8A Active CN111046379B (en) | 2019-12-06 | 2019-12-06 | Anti-attack monitoring method and device |
Country Status (3)
| Country | Link |
|---|---|
| CN (1) | CN111046379B (en) |
| TW (1) | TWI743787B (en) |
| WO (1) | WO2021109695A1 (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111046379B (en) * | 2019-12-06 | 2021-06-18 | 支付宝(杭州)信息技术有限公司 | Anti-attack monitoring method and device |
| CN112200380B (en) * | 2020-10-23 | 2023-07-25 | 支付宝(杭州)信息技术有限公司 | Method and device for optimizing risk detection model |
| CN113313404B (en) * | 2021-06-15 | 2022-12-06 | 支付宝(杭州)信息技术有限公司 | Method and device for generating countermeasure sample |
| CN113505886A (en) * | 2021-07-08 | 2021-10-15 | 深圳市网联安瑞网络科技有限公司 | Countermeasure sample generation method, system, terminal and medium based on fuzzy test |
| CN113760358B (en) * | 2021-08-30 | 2023-08-01 | 河北大学 | Antagonistic sample generation method for source code classification model |
| CN113486875B (en) * | 2021-09-08 | 2021-12-07 | 浙江大学 | Cross-domain face representation attack detection method and system based on word separation and self-adaptation |
| CN114240951B (en) * | 2021-12-13 | 2023-04-07 | 电子科技大学 | Black box attack method of medical image segmentation neural network based on query |
| CN114419346B (en) * | 2021-12-31 | 2022-09-30 | 北京瑞莱智慧科技有限公司 | Model robustness detection method, device, equipment and medium |
| TWI810993B (en) * | 2022-01-06 | 2023-08-01 | 鴻海精密工業股份有限公司 | Model generating apparatus and method |
| CN114639375B (en) * | 2022-05-09 | 2022-08-23 | 杭州海康威视数字技术股份有限公司 | Intelligent voice recognition security defense method and device based on audio slice adjustment |
| CN116071797B (en) * | 2022-12-29 | 2023-09-26 | 北华航天工业学院 | Sparse face comparison countermeasure sample generation method based on self-encoder |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109961444A (en) * | 2019-03-01 | 2019-07-02 | 腾讯科技（深圳）有限公司 | Image processing method, device and electronic equipment |
| CN110175513A (en) * | 2019-04-15 | 2019-08-27 | 浙江工业大学 | A kind of guideboard identification attack defense method based on the optimization of multiple target road |
| CN110213208A (en) * | 2018-05-09 | 2019-09-06 | 腾讯科技（深圳）有限公司 | A kind of method and apparatus and storage medium of processing request |
| CN110321790A (en) * | 2019-05-21 | 2019-10-11 | 华为技术有限公司 | The detection method and electronic equipment of a kind of pair of resisting sample |
| CN110516695A (en) * | 2019-07-11 | 2019-11-29 | 南京航空航天大学 | Confrontation sample generating method and system towards Medical Images Classification |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5568183B2 (en) * | 2011-07-25 | 2014-08-06 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Information identification method, program and system |
| CN104065622B (en) * | 2013-03-20 | 2018-10-19 | 腾讯科技（深圳）有限公司 | The safe early warning method and device of the network equipment |
| US10360378B2 (en) * | 2014-08-22 | 2019-07-23 | Nec Corporation | Analysis device, analysis method and computer-readable recording medium |
| CN106155298B (en) * | 2015-04-21 | 2019-11-08 | 阿里巴巴集团控股有限公司 | The acquisition method and device of man-machine recognition methods and device, behavioural characteristic data |
| US10108850B1 (en) * | 2017-04-24 | 2018-10-23 | Intel Corporation | Recognition, reidentification and security enhancements using autonomous machines |
| CN108615048B (en) * | 2018-04-04 | 2020-06-23 | 浙江工业大学 | Defense method for image classifier adversity attack based on disturbance evolution |
| CN109165671A (en) * | 2018-07-13 | 2019-01-08 | 上海交通大学 | Confrontation sample testing method based on sample to decision boundary distance |
| CN109450946A (en) * | 2018-12-27 | 2019-03-08 | 浙江大学 | A kind of unknown attack scene detection method based on alert correlation analysis |
| CN111046379B (en) * | 2019-12-06 | 2021-06-18 | 支付宝(杭州)信息技术有限公司 | Anti-attack monitoring method and device |
-
2019
- 2019-12-06 CN CN201911242921.8A patent/CN111046379B/en active Active
-
2020
- 2020-05-18 TW TW109116402A patent/TWI743787B/en active
- 2020-09-29 WO PCT/CN2020/118659 patent/WO2021109695A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110213208A (en) * | 2018-05-09 | 2019-09-06 | 腾讯科技（深圳）有限公司 | A kind of method and apparatus and storage medium of processing request |
| CN109961444A (en) * | 2019-03-01 | 2019-07-02 | 腾讯科技（深圳）有限公司 | Image processing method, device and electronic equipment |
| CN110175513A (en) * | 2019-04-15 | 2019-08-27 | 浙江工业大学 | A kind of guideboard identification attack defense method based on the optimization of multiple target road |
| CN110321790A (en) * | 2019-05-21 | 2019-10-11 | 华为技术有限公司 | The detection method and electronic equipment of a kind of pair of resisting sample |
| CN110516695A (en) * | 2019-07-11 | 2019-11-29 | 南京航空航天大学 | Confrontation sample generating method and system towards Medical Images Classification |
Also Published As
| Publication number | Publication date |
|---|---|
| TW202123043A (en) | 2021-06-16 |
| WO2021109695A1 (en) | 2021-06-10 |
| TWI743787B (en) | 2021-10-21 |
| CN111046379A (en) | 2020-04-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111046379B (en) | Anti-attack monitoring method and device | |
| US11856021B2 (en) | Detecting and mitigating poison attacks using data provenance | |
| CN109756458B (en) | Identity authentication method and system | |
| US10841338B1 (en) | Dynamic rule risk score determination in a cybersecurity monitoring system | |
| CN114666162B (en) | Flow detection method, device, equipment and storage medium | |
| CN109600362B (en) | Zombie host recognition method, device and medium based on recognition model | |
| CN114331829A (en) | Countermeasure sample generation method, device, equipment and readable storage medium | |
| CN110889117B (en) | Method and device for defending model attack | |
| CN111046957B (en) | Model embezzlement detection method, model training method and device | |
| CN111882578A (en) | Foreground image acquisition method, foreground image acquisition device and electronic equipment | |
| CN110119621B (en) | Attack defense method, system and defense device for abnormal system call | |
| CN108600162A (en) | User authen method and device, computing device and computer storage media | |
| CN108512651B (en) | Artificial intelligence image identification attack defense method, system and storage medium | |
| CN112528281A (en) | Poisoning attack detection method, device and equipment for federal learning | |
| CN111476668B (en) | Identification method and device of credible relationship, storage medium and computer equipment | |
| CN108921080A (en) | Image-recognizing method, device and electronic equipment | |
| CN113569611A (en) | Image processing method, image processing device, computer equipment and storage medium | |
| CN109587248B (en) | User identification method, device, server and storage medium | |
| CN113222480B (en) | Training method and device for challenge sample generation model | |
| CN113780363A (en) | Countermeasure sample defense method, system, computer and medium | |
| KR20150131846A (en) | Method and System for preventing Login ID theft using captcha | |
| CN113596064A (en) | Analysis control method and system for security platform | |
| CN114139147A (en) | Targeted attack defense method and device | |
| CN108875467B (en) | Living body detection method, living body detection device and computer storage medium | |
| CN115205608B (en) | Adaptive image countermeasure sample detection and defense method based on compressed sensing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40028428 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |