WO2022222143A1 - Security test method and apparatus for artificial intelligence system, and terminal device - Google Patents

Abstract

Disclosed in the present application are a security test method and apparatus for an artificial intelligence system, and a terminal device. The security test method for an artificial intelligence system comprises: acquiring original image data; generating malicious sample data according to the original image data; and performing a security test on the artificial intelligence system according to the malicious sample data, so as to obtain a security test result of the artificial intelligence system. A security attack against an artificial intelligence system in a real environment is simulated, thereby realizing a comprehensive and real security performance test for the artificial intelligence system, and reducing potential security hazards of the artificial intelligence system.

Description

Safety detection method, device and terminal equipment of artificial intelligence system technical field

The present application relates to the technical field of artificial intelligence, and in particular, to a security detection method, device, terminal device and readable storage medium of an artificial intelligence system.

Background technique

In recent years, artificial intelligence technology, as a strategic technology leading a new round of technological revolution and industrial transformation, has become the most critical technology in every country and every technological field.

However, due to the fact that artificial intelligence technology is strongly dependent on training data and lacks interpretability, when the artificial intelligence system is attacked by security, the attacking user can destroy the integrity of the training data by adding attack data to the training data, making artificial The output of the intelligent system is different from the expected correct output, reducing the accuracy of the output of the artificial intelligence system.

The relevant security detection methods based on artificial intelligence systems usually carry out specific security attacks on artificial intelligence systems, but cannot comprehensively and systematically detect and evaluate artificial intelligence systems, and cannot determine the safety of artificial intelligence systems in actual scenarios, resulting in artificial intelligence. The accuracy of the safety performance test results of intelligent technology is unstable and the authenticity is poor.

Application content

The purpose of the embodiments of the present application is to provide a security detection method, device, terminal device and readable storage medium for an artificial intelligence system, including but not limited to solving the problem that the related security detection method based on the artificial intelligence system cannot comprehensively and systematically detect The artificial intelligence system performs detection and evaluation, and the accuracy of the safety performance test results of artificial intelligence technology is unstable and the authenticity is poor.

The technical scheme adopted in the embodiment of the present application is:

In a first aspect, a security detection method for an artificial intelligence system is provided, including:

Get multiple raw image data;

Generate malicious sample data according to the original image data; wherein, the malicious sample data is image data that makes the output result of the artificial intelligence system different from the expected output result;

A security test is performed on the artificial intelligence system according to the malicious sample data, and a security detection result of the artificial intelligence system is obtained.

In a second aspect, a security detection device for an artificial intelligence system is provided, including:

an acquisition module for acquiring multiple original image data;

A generation module is used to generate malicious sample data according to the original image data; wherein, the malicious sample data is the image data that makes the output result of the artificial intelligence system different from the expected output result;

A test module, configured to perform a security test on the artificial intelligence system according to the malicious sample data, and obtain a security detection result of the artificial intelligence system.

In a third aspect, a terminal device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the first method described above when the processor executes the computer program. The security detection method of the artificial intelligence system according to any one of the aspects.

In a fourth aspect, a computer-readable storage medium is provided, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the artificial intelligence system according to any one of the above-mentioned first aspects is implemented security detection method.

A fifth aspect provides a computer program product that, when the computer program product runs on a terminal device, enables the terminal device to execute the security detection method for an artificial intelligence system according to any one of the first aspects above.

The beneficial effect of the security detection method for an artificial intelligence system provided by the embodiment of the present application is that: by acquiring a large amount of original image data, and generating a large amount of corresponding malicious sample data based on the original image data, the artificial intelligence system is analyzed based on the large amount of malicious sample data. Carry out security performance tests to simulate the security attacks of artificial intelligence systems in real environments, realize comprehensive and real security performance tests for artificial intelligence systems, improve the accuracy of artificial intelligence system security detection results, and reduce the safety of artificial intelligence systems. hidden danger.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that are used in the description of the embodiments or exemplary technologies. Obviously, the drawings in the following description are only for the present application. In some embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a schematic flowchart of a security detection method of an artificial intelligence system provided by an embodiment of the present application;

2 is a schematic structural diagram of a high-speed high-definition image acquisition system provided by an embodiment of the present application;

3 is a schematic diagram of an application scenario for extracting local texture information of an image by a local binary pattern algorithm provided by an embodiment of the present application;

4 is a schematic diagram of the positional relationship of a given pixel point pair based on a grayscale co-occurrence matrix provided by an embodiment of the present application;

5 is a schematic diagram of an application scenario for identifying original image data based on the optimized YOLO3 algorithm provided by an embodiment of the present application;

6 is a schematic diagram of an application scenario for generating malicious sample data based on a similar adversarial sample generation method provided by an embodiment of the present application;

7 is a schematic structural diagram of a security detection device of an artificial intelligence system provided by an embodiment of the present application;

8 is another schematic structural diagram of a security detection device for an artificial intelligence system provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present application.

It should be noted that when a component is referred to as being "fixed to" or "disposed on" another component, it can be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly or indirectly connected to the other element. The orientation or positional relationship indicated by the terms "upper", "lower", "left", "right", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of description, rather than indicating or implying the referred device Or the elements must have a specific orientation, be constructed and operated in a specific orientation, so it cannot be construed as a limitation to the present application, and those of ordinary skill in the art can understand the specific meanings of the above terms according to specific situations. The terms "first" and "second" are only used for the purpose of description, and should not be understood as indicating or implying relative importance or implying indicating the number of technical features. "Plurality" means two or more, unless expressly specifically limited otherwise.

In order to illustrate the technical solutions provided in the present application, the following detailed description is given in conjunction with the specific drawings and embodiments.

An artificial intelligence system refers to a neural network model that has all the functions of a general-purpose operating system, and also includes speech recognition, machine vision systems, actuator systems, and cognitive behavioral systems. For example, an autonomous driving network model applied to the field of autonomous driving, or an autonomous control network model applied to autonomous weapons in the military field. Malicious sample data refers to image data that makes the output of the artificial intelligence system different from the expected output. For example, by attacking the autonomous driving network model through malicious sample data, the autonomous driving network model outputs results such as "driving right" or "turning around" when the input data is the traffic sign data of "driving left", which is different from the expected result. The correct output result "driving left" is different.

The security detection method of the artificial intelligence system provided by the embodiment of the present application can be applied to a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and a personal digital assistant (PDA). ) and other terminal devices, the embodiments of the present application do not impose any restrictions on the specific types of the terminal devices.

In recent years, although artificial intelligence technology has gradually become a core and key technology in the field of science and technology, artificial intelligence technology still has certain security risks. The relevant security detection methods based on artificial intelligence systems are usually based on specific attack data to test the security of a specific vulnerability in the algorithm or implementation process of the artificial intelligence system. The mechanism of the security attack cannot be explained theoretically. , it is impossible to comprehensively and systematically conduct safety detection and evaluation of artificial intelligence systems, and it is impossible to determine the safety of artificial intelligence systems in actual scenarios, which leads to the unstable accuracy and poor authenticity of the safety performance test results of artificial intelligence technology to a certain extent. A security detection method for an artificial intelligence system proposed in this application generates a corresponding malicious sample data set by acquiring a large amount of original image data, adding corresponding gradient interference information based on the texture information of each original image data, and through malicious The sample data set is used to test the safety of the artificial intelligence system, and the safety test results are obtained, so as to realize the comprehensive and real safety performance test of the artificial intelligence system, improve the accuracy of the safety test results of the artificial intelligence system, and reduce the security risks of the artificial intelligence system.

FIG. 1 shows a schematic flowchart of the security detection method of the artificial intelligence system provided by the present application. As an example but not limitation, the method can be applied to the above-mentioned notebook computer.

S101. Acquire a plurality of original image data.

In specific applications, artificial intelligence systems are usually subject to security attacks, and the attack data will make the artificial intelligence output results different from the expected correct output results, resulting in a reduction in the accuracy of the output results of the artificial intelligence system, and there are certain security risks. In order to accurately detect the security performance of the artificial intelligence system, it is set that, in the real environment, a large amount of original image data is obtained through a preset collection device, and corresponding malicious sample data is generated based on the texture information of the original image data. Intelligent systems conduct attacks to test the security of artificial intelligence systems. Among them, the original image data refers to the image data collected by the preset collection device in the real environment, or the data set used for training the artificial intelligence system. Artificial intelligence systems specifically refer to vision-based artificial intelligence systems, such as automatic driving neural network models or face recognition systems applied in the field of automatic driving.

It is understandable that when the type of original image data is richer, more levels and richer safety tests can be performed on the artificial intelligence system; for example, taking the autonomous driving network model as an example, the existing traffic sign data sets include: Data sets such as CTSDB, CCTSDB, Tsinghua-Tencent 100K Tutorial, Baidu ApolloScape, etc., but the above traffic sign data sets often have the problem of incomplete data. To this end, it is set to collect a large amount of traffic sign data in a targeted manner through a specific preset collection device in a real environment. Preset capture devices include but are not limited to HD cameras.

In one embodiment, after acquiring the original image data, the method further includes:

Image conversion is performed on each of the original image data by a preset data enhancement method to obtain an original image data set; wherein the preset data enhancement method includes at least one of symmetry processing, rotation processing and scaling processing.

In a specific application, after obtaining a large amount of original image data, in order to expand the original image data set and simulate the original image data with diversity in the real environment, image conversion is performed on each original image data through a preset data enhancement method. , to obtain the corresponding original image data set; the preset data enhancement method includes but is not limited to at least one of symmetry processing, rotation processing and scaling processing.

FIG. 2 exemplarily provides a schematic structural diagram of a high-speed high-definition image acquisition system.

In Figure 2, in the real scene, set the high-speed camera and the high-definition capture card to capture image data in real time, store the image data through the memory, and display the captured image data through the monitor to obtain a plurality of original image data .

S102. Generate malicious sample data according to the original image data; wherein, the malicious sample data is image data that makes the output result of the artificial intelligence system different from the expected output result.

In a specific application, the corresponding malicious sample data is generated by adding interference information to the original image data. The malicious sample data refers to image data that makes the output result of the artificial intelligence system different from the expected output result. The types of malicious sample data include target malicious sample data and non-target malicious sample data; among them, target malicious sample data refers to an attack that makes the artificial intelligence system output a specified wrong result based on the specified input data by attacking the artificial intelligence system. Data; for example, when the input data is the specified "driving left" traffic sign data, the artificial intelligence system is attacked through the target malicious sample data, so that the output result of the artificial intelligence system is the specified "driving right". Or, when the input data is the designated "no entry" traffic sign data, the artificial intelligence system is attacked through the target malicious sample data, so that the output result of the artificial intelligence system is the designated "go straight". Non-target malicious sample data refers to attack data that makes the artificial intelligence system output random results (different from the expected output results) by attacking the artificial intelligence system. For example, when the input data is "driving left" traffic sign data, the artificial intelligence system is attacked through non-target malicious sample data, so that the output of the artificial intelligence system includes "driving right", "going straight" or "turning around", etc. In order to reduce the accuracy of the output results of the artificial intelligence system.

S103. Perform a security test on the artificial intelligence system according to the malicious sample data, and obtain a security detection result of the artificial intelligence system.

In specific applications, according to the different security requirements of the artificial intelligence system, the malicious sample data is adjusted, and the artificial intelligence system is attacked based on the adjusted malicious sample data to obtain the corresponding security detection results.

In one embodiment, the generating malicious sample data according to the original image data includes:

Calculate the texture information of each original image data in the original image data set;

Based on the texture information of each original image data, gradient interference information is added to generate a corresponding malicious sample data set.

In specific applications, the attack data of vision-based artificial intelligence systems is generally image data, and the texture information of image data is an important feature of regular arrangement in visual information, which can describe the local area of image data from a pixel to The local intensity variation of another pixel, reflecting the homogeneity in the image data. Therefore, it is set to add gradient interference information to the texture information of the original image data to generate the corresponding malicious sample data: first, the texture information of each original image data in the original image data set needs to be calculated, and based on each original image data, add Corresponding gradient interference information causes large pixel changes in each original image data to obtain the corresponding malicious sample data set.

In specific applications, texture information is mainly reflected by the grayscale distribution of pixels and their surrounding spaces, which is essentially a statistical feature related to grayscale changes. Methods for calculating texture information include but are not limited to local binary patterns. (Local Binary Patterns, LBP) algorithm, Gray-level Co-occurrence Matrix (GLCM), Discrete Fourier Transform Local Phase Quantization (LPQ), Weber's local feature based on Weber's law ( Weber Local Descriptor, WLD).

Among them, the Local Binary Patterns (LBP) algorithm mainly measures the neighborhood attribute value (grayscale or RGB single channel) of the surrounding window by the central pixel value in a specific window, and only records the size relationship to reflect the local texture information. The expression method is the concatenated code obtained by binarizing the size relationship. The local binary mode algorithm has the advantages of simplicity, strong operability, rotation invariance, grayscale invariance, scale invariance, and robustness to illumination changes.

FIG. 3 exemplarily provides a schematic diagram of an application scene of a local binary pattern algorithm for extracting local texture information of an image;

As shown in Figure 3, the gray value of the center pixel in a window with a size of 3 × 3 is 83, and the gray values of the 8 pixels adjacent to the center pixel are compared with the gray value of the center pixel. If the center pixel is detected The gray value of the adjacent pixel is greater than the gray value of the central pixel, then the gray value of the adjacent pixel is recorded as 1; otherwise, it is recorded as 0. The eight-bit binary number is obtained and converted into a decimal number, and the converted decimal number is used as the local binary mode value of the center pixel in the window, which can be expressed as:

In the formula: i represents the serial number of the adjacent pixels in the serial port except the center pixel; I _i represents the attribute value of the ith adjacent pixel; I _c represents the attribute value of the center pixel; s( ) represents the binarization function :

In a specific application, the gray level co-occurrence matrix is a method for calculating the occurrence probability of a given pixel in image data for different gray values.

FIG. 4 exemplarily shows a schematic diagram of the positional relationship of a given pixel point pair based on a grayscale co-occurrence matrix;

As shown in Figure 4, assuming that two factors, the direction θ and the distance δ in the pixel pair in the image data are given, the correspondence can be determined, any pixel point f(x, y) in the image data and the deviation from the pixel point f The pixel point f(x+dx,y+dy) of (x,y) constitutes a pixel point pair. It is assumed that the gray value of the above-mentioned pixel point pair is expressed as (f ₁ , f ₂ ), and the maximum gray level of the known image data is L. There are L×L types of permutations and combinations of grayscale values (f ₁ , f ₂ ) of pixel pairs. Count the number of occurrences of each gray value ( _{f 1} , _{f 2} ) in the image data and arrange them into a square matrix, and then normalize the probability P(f ₁ , f ₂ ) to get the gray level co-occurrence matrix. It can be seen that the gray level co-occurrence matrix P(f ₁ , f ₂ ) characterizes the gray levels as f ₁ and f ₂ at a given pixel pair f(x, y), f(x+dx, y+dy) The probability of appearing in:

p(i,j,j,δ,θ)={[(x,y),(x+dx,y+dy)]|f(x,y)=f ₁ ,f(x+dx,y+ dy)=f ₂ } (3);

According to the gray level co-occurrence matrix, statistical parameters are extracted, which can be used to describe the texture information of the image. The common characteristic parameters of the gray level co-occurrence matrix are shown in Table 1.

Table 1 Common characteristic parameters of gray level co-occurrence matrix

In specific applications, the process of adding gradient interference information can be expressed as:

In the formula, x represents the input data of the artificial intelligence system; y represents the output data of the artificial intelligence system; x' represents the input data after adding gradient interference information; θ represents the model parameters of the artificial intelligence system; J represents the defined artificial intelligence system. model loss function;

Represents gradient operation; ε represents gradient interference step size; sign variable represents sign function.

In the backpropagation of the network model of the artificial intelligence system, the weights and offsets of the neuron nodes are mainly updated along the gradient direction, so that the network model of the artificial intelligence system converges in the direction of reducing the loss value:

In the formula: W _ij represents the weight of the neuron node in the artificial intelligence system network model; b _i represents the offset of the neuron node in the artificial intelligence system network model; α represents the learning rate.

The corresponding malicious sample data set is generated by adding gradient interference information based on the image texture of the original image data, and the malicious sample data set is input into the artificial intelligence system network model for processing. The input link increases the loss value in the training process of the artificial intelligence system network model, thereby reducing the ability of the artificial intelligence system network model to correctly identify. It can be understood that when the artificial intelligence system network model adopts a linear or approximately linear activation function, the error conduction value will gradually increase.

In one embodiment, after performing image conversion on each of the original image data by using a preset data enhancement method to obtain the original image data set, the method includes:

The content of each original image data in the original image data set is identified, and the label of each original image data is determined.

In specific applications, the optimized YOLO3 algorithm is used to identify the content in each original image data and determine the corresponding label. For example, taking an autonomous driving system as an example, it is necessary to obtain a large amount of traffic sign data as the original image data, identify the content of the traffic sign data through the optimized YOLO3 algorithm, determine the instruction information contained in each traffic sign data, and add the corresponding Label.

Among them, the optimized YOLO3 algorithm refers to the algorithm obtained by optimizing the original YOLO3 algorithm by adjusting the residual structure and anchor. By adjusting the residual structure, the efficiency of identifying and marking the original image data can be improved; Anchor, which is convenient to adapt the YOLO3 algorithm to the size of the original image data (including memory size and aspect ratio information). The original image data is clustered by the clustering Kmeans algorithm to realize the fast training process of the YOLO3 algorithm.

In one embodiment, part of the original sample data that can reflect the performance of the artificial intelligence system network model can be selected from the original image data set as the target data set, and the label of each original image data in the target data set can be identified and determined. In order to reduce the amount of data processing and improve the efficiency of artificial intelligence system security detection.

FIG. 5 exemplarily provides a schematic diagram of an application scenario for identifying original image data based on the optimized YOLO3 algorithm.

In Figure 5, the original image data is specifically traffic sign data. Based on the optimized YOLO3 algorithm, the indication information contained in each traffic sign data is identified and determined, and corresponding labels are added. Parking" traffic sign data, add the "No Parking" tag.

It can be understood that the type of the original image data can be determined according to the label of the original image data, for example, the type of the original image data with the label of "speed limit 40" is "limited"; the original image with the label of "no overtaking" The category of the data is "prohibition"; the category of the original image data with the label "motor vehicle running" is "instruction"; the category of the original image data with the label of "construction ahead" is "warning".

In one embodiment, the adding gradient interference information to the original image data to generate corresponding malicious sample data includes:

Perform clustering processing according to the label of each original image data in the original image data set to obtain image data sets of multiple categories;

The corresponding gradient interference information is added to the texture information of each original image data in each category of image datasets by the method of generating similar adversarial samples, and multiple categories of malicious sample datasets are obtained.

In a specific application, according to the label of each original image data in the original image data set, the original image data set is clustered to obtain multiple original image data of different categories, and the original image data of the same category is processed in parallel. The adversarial sample generation method adds gradient interference information to the original image data of the same category, and obtains the corresponding malicious sample data of the same category.

For example: obtain a large amount of traffic sign data as the original image data set, determine the label of each traffic sign data in the original image data set; perform clustering processing on the original image data set according to the label of each traffic sign data, obtain data including but not limited to A dataset of images in multiple categories such as "ban", "warning", "instruction" and "restriction".

Among them, the method of generating similar adversarial samples refers to the parallel processing of image datasets of the same category, and the malicious sample data generated from the previous original image data in the same category of image datasets is used as the starting value of the next original image data to generate Corresponding methods for malicious sample datasets. That is, based on the similarity of all the original image data in the image dataset of the same category, and based on the parallel processing of the similar adversarial sample generation method, the malicious sample data of the same category is generated, and the number of iterations is reduced.

Based on the similar adversarial sample generation method, the original image data in the same category of image data sets are processed in parallel to generate the corresponding malicious sample data sets, which can reduce the number and time of data reading and improve the generation speed of malicious sample data.

FIG. 6 exemplarily shows a schematic diagram of an application scenario of generating malicious sample data based on a similar adversarial sample generation method.

In one embodiment, the types of malicious sample data include target malicious sample data and non-target malicious sample data.

In one embodiment, performing a security test on the artificial intelligence system according to the malicious sample data to obtain a security detection result of the artificial intelligence system, including:

According to the security requirement level and input data authority of the artificial intelligence system, determine the ratio of target malicious sample data and non-target malicious sample data and the corresponding preset algorithm, perform a security test on the artificial intelligence system, and obtain the first test result;

According to the security performance level of the artificial intelligence system, the artificial intelligence system is subjected to a corresponding preset duration security test through the malicious sample data, and a second test result is obtained;

The safety detection score of the artificial intelligence system is determined according to the first test result and the second test result.

In a specific application, the types of malicious sample data include but are not limited to target malicious sample data and non-target malicious sample data.

In specific applications, the security requirements of artificial intelligence systems in different application fields are different (for example, the security requirement level and security performance level of the face recognition system of customs are higher than those of the face recognition system of shopping malls and residential areas. ; Different input data in the customs' face recognition system also have different levels of authority); therefore, the security attack intensity and attack duration of artificial intelligence systems in different application fields are also different. Correspondingly, set the security requirements for different artificial intelligence systems, and select the corresponding security testing methods, including: based on the security requirements characteristics of the artificial intelligence system with different defense levels, when the security requirements level and/or input of the artificial intelligence system are detected. When the data authority is high, adjust the proportion of target malicious sample data in the malicious sample data set to be greater than the proportion of non-target malicious sample data, and use an attack algorithm with high attack intensity to conduct a security test on the artificial intelligence system, and obtain the corresponding first test. Result; when it is detected that the security requirement level of the artificial intelligence system and/or the input data authority is low, adjust the proportion of target malicious sample data in the malicious sample data set to be less than or equal to the proportion of non-target malicious sample data, and use low attack intensity. The attack algorithm artificial intelligence system conducts security test and obtains the corresponding first test result. The first test result refers to the accuracy rate of the output result of the artificial intelligence system after the artificial intelligence system is subjected to a security attack based on the proportion-adjusted target malicious sample data and non-target malicious sample data through the preset algorithm determined above.

In specific applications, attack algorithms for security detection of artificial intelligence systems include, but are not limited to, Fast Gradient Sign Method (FGSM), Iterative Fast Gradient Sign Method (IFGSM), and C&W attack algorithms. .

The FGSM algorithm refers to accurately modifying the input data by calculating the gradient of the model output to the input to achieve the purpose of attack. IFGSM is an improved algorithm based on the FGSM algorithm. It can generate more accurate malicious sample data than the FGSM algorithm. The attack success rate is higher than that of the FGSM algorithm, and the attack cost increases accordingly. The C&W algorithm can effectively break through a variety of malicious sample defense methods, and is currently recognized as one of the strongest attack methods.

Table 2 Comparison of attack methods with different strengths

In specific applications, based on the characteristics of the security requirements of the artificial intelligence system with different defense durations, when it is detected that the security performance level of the artificial intelligence system is high, the security of the artificial intelligence system is performed for the first preset duration through the malicious sample data set. test to obtain a corresponding second test result; when it is detected that the security performance level of the artificial intelligence system is low, conduct a security test on the artificial intelligence system for a second preset duration through the malicious sample data set, and obtain a corresponding second test result. Wherein, the first preset duration is longer than the second attack duration. The second test result refers to the accuracy rate of the output result of the artificial intelligence system after performing a security attack on the artificial intelligence system with a preset duration based on the target malicious sample data and non-target malicious sample data adjusted by the ratio.

In a specific application, the safety detection score of the artificial intelligence system is calculated and determined according to the first test result and the second test result. Among them, the calculation method and value range of the security detection score can be specifically set according to the actual situation;

For example, safety detection score=first test result*A+second test result*B; where A and B are the weights of the first test result and the second test result, respectively, which can be specifically set according to the actual situation, corresponding to the safety The value range of the sex detection score is [0, 100]. Or security test score=first test result+second test result, and the value range of the corresponding security test score is [0, 1].

In this embodiment, the higher the security detection score of the artificial intelligence system is set, the stronger the security performance of the artificial intelligence system is.

In this embodiment, a large amount of original image data is acquired, a large amount of corresponding malicious sample data is generated based on the original image data, and a safety performance test is performed on the artificial intelligence system based on the large amount of malicious sample data, so as to simulate the performance of the artificial intelligence system in a real environment. Security attack, realize comprehensive and real security performance test for artificial intelligence system, improve the accuracy of artificial intelligence system security detection results, and reduce the security risks of artificial intelligence system.

It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

Corresponding to the security detection method of the artificial intelligence system described in the above embodiment, FIG. 7 shows a structural block diagram of the security detection device of the artificial intelligence system provided by the embodiment of the present application. Parts related to the embodiments of this application.

The present invention also provides another preferred embodiment of the security detection device of the artificial intelligence system. In this embodiment, the security detection device of the artificial intelligence system includes: a processor, wherein the processor is used for executing the storage memory The following program modules: acquisition module, used to acquire multiple raw image data;

A generation module, configured to generate malicious sample data according to the original image data; wherein, the malicious sample data is image data that makes the output result of the artificial intelligence system different from the expected output result;

A test module, configured to perform a security test on the artificial intelligence system according to the malicious sample data, and obtain a security detection result of the artificial intelligence system.

Referring to FIG. 7 , the security detection device 100 of the artificial intelligence system includes:

an acquisition module 101, configured to acquire a plurality of original image data;

A generating module 102, configured to generate malicious sample data according to the original image data; wherein, the malicious sample data is image data that makes the output result of the artificial intelligence system different from the expected output result;

The testing module 103 is configured to perform a security test on the artificial intelligence system according to the malicious sample data, and obtain a security detection result of the artificial intelligence system.

In one embodiment, the security detection device of the artificial intelligence system further includes:

A data processing module 201 is configured to perform image conversion on each of the original image data through a preset data enhancement method to obtain an original image data set; wherein the preset data enhancement method includes symmetry processing, rotation processing and scaling processing. at least one of.

In one embodiment, the apparatus further includes:

The identification module 202 is configured to identify the content of each original image data in the original image data set, and determine the label of each original image data.

In one embodiment, the generating module 102 includes:

a computing unit, for calculating the texture information of each original image data in the original image data set;

A generating unit, configured to add gradient interference information based on the texture information of each original image data to generate a corresponding malicious sample data set.

In one embodiment, the generating unit includes:

a clustering processing subunit, configured to perform clustering processing according to the label of each original image data in the original image data set to obtain image data sets of multiple categories;

The generating sub-unit is used for adding corresponding gradient interference information to the texture information of each original image data in each category of image datasets by using a similar adversarial sample generation method to obtain multiple categories of malicious sample datasets.

In one embodiment, the types of malicious sample data include target malicious sample data and non-target malicious sample data.

In one embodiment, the testing module 103 includes:

The first test unit is used to determine the ratio of target malicious sample data and non-target malicious sample data and the corresponding preset algorithm according to the security requirement level and input data authority of the artificial intelligence system, and to perform security on the artificial intelligence system. sex test, get the first test result;

a second testing unit, configured to perform a corresponding preset duration security test on the artificial intelligence system through the malicious sample data according to the security performance level of the artificial intelligence system, to obtain a second test result;

A determination unit, configured to determine the safety detection score of the artificial intelligence system according to the first test result and the second test result.

FIG. 8 exemplarily provides a schematic structural diagram of another security detection device 100 based on an artificial intelligence system;

As shown in FIG. 8 , the security detection device 100 based on an artificial intelligence system is set to further include a basic hardware layer 104 and a machine learning framework module 105; wherein, the basic hardware layer includes but is not limited to CPU, GPU, FPGA and other artificial intelligence systems. A training/deployment platform that provides hardware base support for the superstructure. Machine learning frameworks: including but not limited to open source machine frameworks such as PyTorch, TensorFlow, and MXNet, which are used to support neural network models of artificial intelligence systems trained under different frameworks.

In this embodiment, a large amount of original image data is acquired, a large amount of corresponding malicious sample data is generated based on the original image data, and a safety performance test is performed on the artificial intelligence system based on the large amount of malicious sample data, so as to simulate the performance of the artificial intelligence system in a real environment. Security attack, realize comprehensive and real security performance test for artificial intelligence system, improve the accuracy of artificial intelligence system security detection results, and reduce the security risks of artificial intelligence system.

It should be noted that the information exchange, execution process and other contents between the above-mentioned devices/units are based on the same concept as the method embodiments of the present application. For specific functions and technical effects, please refer to the method embodiments section. It is not repeated here.

FIG. 9 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As shown in FIG. 9 , the terminal device 9 in this embodiment includes: at least one processor 90 (only one is shown in FIG. 9 ), a memory 91 , and a memory 91 stored in the memory 91 and available in the at least one processor 90 The computer program 92 running on the processor 90, when the processor 90 executes the computer program 92, implements the steps in any of the foregoing embodiments of the security detection method for the artificial intelligence system.

The terminal device 9 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The terminal device may include, but is not limited to, a processor 90 and a memory 91 . Those skilled in the art can understand that FIG. 9 is only an example of the terminal device 9, and does not constitute a limitation on the terminal device 9. It may include more or less components than the one shown, or combine some components, or different components , for example, may also include input and output devices, network access devices, and the like.

The so-called processor 90 may be a central processing unit (Central Processing Unit, CPU), and the processor 90 may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuits) , ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 91 may be an internal storage unit of the terminal device 9 in some embodiments, such as a hard disk or a memory of the terminal device 9 . In other embodiments, the memory 91 may also be an external storage device of the terminal device 9, such as a plug-in hard disk equipped on the terminal device 9, a smart memory card (Smart Media Card, SMC), a secure digital Card (Secure Digital, SD), flash memory card (Flash Card), etc. The memory 91 may also include both an internal storage unit of the terminal device 9 and an external storage device. The memory 91 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as program codes of the computer program. The memory 91 can also be used to temporarily store data that has been output or will be output.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the above-mentioned system, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

The embodiments of the present application provide a computer program product, when the computer program product runs on a mobile terminal, the steps in the foregoing method embodiments can be implemented when the mobile terminal executes the computer program product.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the methods of the above embodiments, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When executed by a processor, the steps of each of the above method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to the photographing device/terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media. For example, U disk, mobile hard disk, disk or CD, etc. In some jurisdictions, under legislation and patent practice, computer readable media may not be electrical carrier signals and telecommunications signals.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed apparatus/network device and method may be implemented in other manners. For example, the apparatus/network device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units. Or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

The above are only optional embodiments of the present application, and are not intended to limit the present application. Various modifications and variations of this application are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims (14)

1. A security detection method for an artificial intelligence system, characterized in that it includes:

   Get multiple raw image data;

   Generate malicious sample data according to the original image data; wherein, the malicious sample data is image data that makes the output result of the artificial intelligence system different from the expected output result;

   A security test is performed on the artificial intelligence system according to the malicious sample data, and a security detection result of the artificial intelligence system is obtained.
2. The method for detecting the safety of an artificial intelligence system according to claim 1, wherein after acquiring the original image data, the method further comprises:

   Image conversion is performed on each of the original image data by a preset data enhancement method to obtain an original image data set; wherein the preset data enhancement method includes at least one of symmetry processing, rotation processing and scaling processing.
3. The method for detecting the safety of an artificial intelligence system according to claim 2, wherein the image conversion is performed on each of the original image data by a preset data enhancement method, and after obtaining the original image data set, the method includes:

   The content of each original image data in the original image data set is identified, and the label of each original image data is determined.
4. The security detection method for an artificial intelligence system according to claim 2, wherein the generating malicious sample data according to the original image data comprises:

   Calculate the texture information of each original image data in the original image data set;

   Based on the texture information of each original image data, gradient interference information is added to generate a corresponding malicious sample data set.
5. The security detection method for an artificial intelligence system as claimed in claim 4, wherein, based on the texture information of each original image data, adding gradient interference information to generate a corresponding malicious sample data set, comprising:

   Perform clustering processing according to the label of each original image data in the original image data set to obtain image data sets of multiple categories;

   The corresponding gradient interference information is added to the texture information of each original image data in each category of image datasets by the method of generating similar adversarial samples, and multiple categories of malicious sample datasets are obtained.
6. The security detection method of an artificial intelligence system according to claim 1, wherein the types of the malicious sample data include target malicious sample data and non-target malicious sample data;

   The said artificial intelligence system is subjected to a security test according to the malicious sample data, and a security detection result of the artificial intelligence system is obtained, including:

   According to the security requirement level and input data authority of the artificial intelligence system, determine the ratio of target malicious sample data and non-target malicious sample data and the corresponding preset algorithm, perform a security test on the artificial intelligence system, and obtain the first test result;

   According to the security performance level of the artificial intelligence system, the artificial intelligence system is subjected to a corresponding preset duration security test through the malicious sample data, and a second test result is obtained;

   The safety detection score of the artificial intelligence system is determined according to the first test result and the second test result.
7. A safety detection device for an artificial intelligence system, characterized in that it includes:

   an acquisition module for acquiring multiple original image data;

   A generation module, configured to generate malicious sample data according to the original image data; wherein, the malicious sample data is image data that makes the output result of the artificial intelligence system different from the expected output result;

   A test module, configured to perform a security test on the artificial intelligence system according to the malicious sample data, and obtain a security detection result of the artificial intelligence system.
8. The safety detection device of an artificial intelligence system according to claim 7, wherein the device further comprises:

   A data processing module, configured to perform image conversion on each of the original image data through a preset data enhancement method to obtain an original image data set; wherein the preset data enhancement method includes symmetry processing, rotation processing and scaling processing. at least one.
9. The safety detection device of an artificial intelligence system according to claim 8, wherein the device further comprises:

   The identification module is used for identifying the content of each original image data in the original image data set, and determining the label of each original image data.
10. The security detection device of an artificial intelligence system according to claim 8, wherein the generation module comprises:

    a computing unit, for calculating the texture information of each original image data in the original image data set;

    A generating unit, configured to add gradient interference information based on the texture information of each original image data to generate a corresponding malicious sample data set.
11. The security detection device of an artificial intelligence system according to claim 10, wherein the generating unit comprises:

    Clustering processing subunit, for carrying out clustering processing according to the label of each original image data in described original image data set, obtains the image data set of multiple categories;

    The generating sub-unit is used for adding corresponding gradient interference information to the texture information of each original image data in each category of image datasets by using a similar adversarial sample generation method to obtain multiple categories of malicious sample datasets.
12. The security detection device of an artificial intelligence system according to claim 7, wherein the types of the malicious sample data include target malicious sample data and non-target malicious sample data;

    The test module includes:

    The first test unit is used to determine the ratio of target malicious sample data and non-target malicious sample data and the corresponding preset algorithm according to the security requirement level and input data authority of the artificial intelligence system, and to perform security on the artificial intelligence system. sex test, get the first test result;

    a second testing unit, configured to perform a corresponding preset duration security test on the artificial intelligence system through the malicious sample data according to the security performance level of the artificial intelligence system, to obtain a second test result;

    A determination unit, configured to determine the safety detection score of the artificial intelligence system according to the first test result and the second test result.
13. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, when the processor executes the computer program, the process according to claim 1 to 6 The method of any one.
14. A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images